I work in information security, and one project I have been considering undertaking would involve using a bot (automated web spider) which crawls Tor hidden services for analysis. I am currently in the process of writing the code for the bot. I am concerned that I will run afoul of laws regarding child pornography or copyright infringement, as some hidden services contain such material. The current implementation of the bot does not make a distinction between legal and illegal sites. It does not query me for permission before performing an action (I never give it the "go ahead").

Some relevant information:

  • The bot will be running remotely on a rented server in US jurisdiction.

  • No web resources, legal or otherwise, will be saved to persistent storage.

  • I will not be viewing any material the bot accesses. I will only view its analysis.

  • I cannot keep a site blacklist. The bot will crawl as far as it can.

  • The crawling activity will not interfere with the functionality of the website.

  • Only public resources that can be accessed by anyone will be crawled.

  • The bot may attempt to sign itself up to websites if registration is necessary.

I believe that it is likely, if not certain, that the bot will end up accessing illegal material at least once. When it does so, it will fetch the webpage into memory and analyze it. This analysis will be saved to disk. The analysis will contain information regarding anomalies in image file formats, obfuscated JavaScript, non-standard use of CSS, rare HTML tags and uses, etc. It will also save the URL. After analyzing the page, it will continue crawling as a typical web spider, following links.

Am I responsible for any laws broken by the bot, given that I am not intending for it to access illegal material but am aware that there is a distinct possibility that it will? Is there anything I should be aware of to reduce the risk of the bot taking an illegal action which I would be responsible for? Given that one could argue (correctly, but out of context) that I am writing a program that signs itself up to illegal websites and downloads illegal material, this makes me very weary.

  • 1
    "No decisions it makes or actions it takes are done by me" So you programmed a sentient being?
    – Ron Beyer
    Jul 11, 2018 at 1:01
  • 2
    @RonBeyer No, but it is automated. What I mean is that any decisions it makes are done automatically. It does not ask me for confirmation. "Decision" has a precise meaning in computer science and does not imply sentience or even sapience.
    – forest
    Jul 11, 2018 at 1:09
  • 2
    The problem you face is not specific to bots. Any defaced site might contain illegal content, but you only know this once you navigate to that site.
    – MSalters
    Jul 11, 2018 at 11:35

2 Answers 2


You are generally responsible and liable for what your algorithms and machines do.

You are right, your web crawler will indeed access illegal material given enough external link crawls. Though there is a difference between your computer viewing content and you yourself actually viewing that content.

If your machine were accessing millions of pages each day, I think that your ISP and government officials would likely realize that this is robot automation, and would likely associate it with web crawling activities.

Considering that Google is a web crawler, I think the general public, FBI, NSA and ISPs understand how new web crawlers can be very beneficial to society and so it's unlikely that they would ever want to directly come after you unless they specifically had it out for you.

  • 2
    What if it is only accessing a hundred pages each day, with each page having a much higher probability of hosting illegal materials? There's a big difference between a bot designed to analyze "the underground" and a Google spider that indexes web pages for a search engine.
    – forest
    Nov 11, 2018 at 8:08

My answer is on the basis of Michigan MCL 750.540. Although I haven't compared it with the statutes from other jurisdictions, I doubt that other states have much more stringent laws in this regard.

Am I responsible for any laws broken by the bot, given that I am not intending for it to access illegal material but am aware that there is a distinct possibility that it will?

It is not clear whether by hidden services you mean functionality that requires access credentials restricted to particularly authorized people/systems. Other than that, I think you are safe because neither you nor the program incur the conduct prohibited by statutory law. Clearly, the type of specialized analysis you envision is not the type of conduct that the law sanctions.

The prohibited conduct listed in MCL 750.540 is in terms of willful and malicious acts of interruption, access, disconnect, use, etc. Your description of the bot indicates that it is not programmed or geared toward consumption of unlawful material (such as child pornography or organized crime). The lack of naming conventions by which illegal sites would abide renders it impossible to implement the web crawler in a way that it ascertains the legality of that page. A similar reasoning applies to the actual contents any unlawful files which can be downloaded from the crawled websites.

To summarize, the possibility that the bot might crawl and analyze a site that happens to be illegal does not imply malicious conduct on the analyst's/programmer's part.

  • 2
    Hidden service is the terminology used by the Tor Project for domains only accessible over the Tor network, such as facebookcorewwwi.onion (most sites have random names by design, however).
    – forest
    Jul 12, 2018 at 2:21
  • How is "consumption" defined in this case?
    – forest
    Nov 11, 2018 at 8:10
  • Here by consumption I mean the pursuit or facilitation of the ultimate purpose for which the unlawful material was created, like a pedophile's enjoyment of child pornography, a terrorist's networking, someone's engagement in human trafficking, counterfeiting, and so forth. Since you are only interested in the results of the bot's analyses, you are not storing unlawful material, the bot is not devised to facilitate illegal purposes such as the ones I listed, it does not perform DoS attacks, etc., the bot's crawling and parsing of essentially random sites falls short of being a crime. Nov 11, 2018 at 11:46
  • Regarding hidden services, it is better to use the term “onion service”. And to clarify for everyone else, onion service addresses should not be used as credentials; they should be assumed to be public. Apr 5, 2021 at 15:12
  • @BrianDrake I'm aware that Tor Project changed the name as per the suggestion of their PR team, but in the code and technical specifications, "HS" as an initialism is ubiquitous (e.g. HSDir). Anyway I might have posted this before they made that change.
    – forest
    Apr 25, 2021 at 1:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .