Automated analysis of potentially illegal material using a web spider

Question

I work in information security, and one project I have been considering undertaking would involve using a bot (automated web spider) which crawls Tor hidden services for analysis. I am currently in the process of writing the code for the bot. I am concerned that I will run afoul of laws regarding child pornography or copyright infringement, as some hidden services contain such material. The current implementation of the bot does not make a distinction between legal and illegal sites. It does not query me for permission before performing an action (I never give it the "go ahead").

Some relevant information:

• The bot will be running remotely on a rented server in US jurisdiction.

• No web resources, legal or otherwise, will be saved to persistent storage.

• I will not be viewing any material the bot accesses. I will only view its analysis.

• I cannot keep a site blacklist. The bot will crawl as far as it can.

• The crawling activity will not interfere with the functionality of the website.

• Only public resources that can be accessed by anyone will be crawled.

• The bot may attempt to sign itself up to websites if registration is necessary.

I believe that it is likely, if not certain, that the bot will end up accessing illegal material at least once. When it does so, it will fetch the webpage into memory and analyze it. This analysis will be saved to disk. The analysis will contain information regarding anomalies in image file formats, obfuscated JavaScript, non-standard use of CSS, rare HTML tags and uses, etc. It will also save the URL. After analyzing the page, it will continue crawling as a typical web spider, following links.

Am I responsible for any laws broken by the bot, given that I am not intending for it to access illegal material but am aware that there is a distinct possibility that it will? Is there anything I should be aware of to reduce the risk of the bot taking an illegal action which I would be responsible for? Given that one could argue (correctly, but out of context) that I am writing a program that signs itself up to illegal websites and downloads illegal material, this makes me very weary.

Answer 1

You are generally responsible and liable for what your algorithms and machines do.

You are right, your web crawler will indeed access illegal material given enough external link crawls. Though there is a difference between your computer viewing content and you yourself actually viewing that content.

If your machine were accessing millions of pages each day, I think that your ISP and government officials would likely realize that this is robot automation, and would likely associate it with web crawling activities.

Considering that Google is a web crawler, I think the general public, FBI, NSA and ISPs understand how new web crawlers can be very beneficial to society and so it's unlikely that they would ever want to directly come after you unless they specifically had it out for you.

Answer 2

My answer is on the basis of Michigan MCL 750.540. Although I haven't compared it with the statutes from other jurisdictions, I doubt that other states have much more stringent laws in this regard.

Am I responsible for any laws broken by the bot, given that I am not intending for it to access illegal material but am aware that there is a distinct possibility that it will?

It is not clear whether by hidden services you mean functionality that requires access credentials restricted to particularly authorized people/systems. Other than that, I think you are safe because neither you nor the program incur the conduct prohibited by statutory law. Clearly, the type of specialized analysis you envision is not the type of conduct that the law sanctions.

The prohibited conduct listed in MCL 750.540 is in terms of willful and malicious acts of interruption, access, disconnect, use, etc. Your description of the bot indicates that it is not programmed or geared toward consumption of unlawful material (such as child pornography or organized crime). The lack of naming conventions by which illegal sites would abide renders it impossible to implement the web crawler in a way that it ascertains the legality of that page. A similar reasoning applies to the actual contents any unlawful files which can be downloaded from the crawled websites.

To summarize, the possibility that the bot might crawl and analyze a site that happens to be illegal does not imply malicious conduct on the analyst's/programmer's part.