How to crawl a quarter billion webpages in 40 hours

In my experience, one of the hardest parts of writing a web crawler is URL selection: After crawling your list of seed URLs, where do you go next? How do you make sure you don't crawl the same content multiple times because it has a slightly different URL? How to avoid getting stuck on unimportant spam sites with autogenerated content? Because the author only crawled domains from a limited set and only for a short time, he did not need…

Originally I intended to make the crawler code available under an open source license at GitHub. However, as I better understood the cost that crawlers impose on websites, I began to have reservations. My crawler is designed to be polite and impose relatively little burden on any single website, but could (like many crawlers) easily be modified by thoughtless or malicious people to impose a heavy burden on sites. Because of this I’ve decided to postpone (possibly indefinitely) releasing the…

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request patterns, and browser behaviour to try and detect and block…

This was written in 2012, Its even easier these days by using SQS and Cloud Formation. 250 Million is a small number you are better of first going through Common Crawl and then use data from crawls to build a better seed list. Common Crawl now contains repeated crawls conducted every few months and also urls donated by blekko. https://groups.google.com/forum/m/#!msg/common-crawl/zexccXg...