Web Crawler for Finding Commentable Blogs

In response to the idea of a web crawler described by BingoBoingo, I threw together a quick script in Python which follows his specification, I think.

The script loads data from a "churn" file. This should be a list of valid websites, one per line. It then goes through the list, looks for any links, which it adds to the todo list, and saves any sites that it finds a comment box into the "targets" file. I added comments to make the whole thing readable, let me know if there is anything confusing.

http://peterl.xyz/wp-content/uploads/crawler.py

Tags:

Leave a Reply