Facebook Crawler Speaks Back
Last week we ran a story about Facebook suing to get a crawled dataset offline. This week we have a bit of a
response written by Pete Warden, the guy who actually did the crawling. He followed robots.txt, and then Facebook’s lawyers went after him. It’s actually a quite interesting little tale and worth your time.
Source: Facebook Crawler Speaks Back