There is a standard through which websites communicate to the web crawlers upon which search engines are based which pages should be indexed and included in the search results. This is called the Robots exclusion standard, is implemented in the robots.txt file that most web sites have, stack exchange's is here.

If one was interested in the whole content of a site, for example one was researching the network structure of web sites, would it be legal to ignore the robots.txt and crawl the whole site (perhaps excluding /search[?/])?

  • There is no universal robots.txt (details differ). News Agencies tried to make new version of standard (with bells and whistles THEY wanted). All major search engines just ignored this it.
    – Tauri
    Commented May 2, 2023 at 15:34

1 Answer 1


The existence of a robots.txt file and the directory and file inclusions/exclusions in a robots.txt file do not constitute a legally binding contract for the use of the website by the visitor; if it exists, the Terms of Service would usually establish the contract for use of the site.

Many TOSs prohibit bots, crawlers and any automated means from crawling or scraping a site, and you are contractually obligated to honor those stipulations simply by visiting the site. Violating a TOS is typically a civil matter, depending on jurisdiction.

Simply crawling a site to "map" it, and crawling a site to "scrape" it and copy all content are two different things. Crawling could violate the TOS. Scraping content can violate the TOS and also be copyright infringement, depending on jurisdiction. See https://law.stackexchange.com/search?q=scraping

I suppose a TOS could state that users are legally bound to the restrictions in a robots.txt, but I've never seen that in a TOS.

  • 1
    Just noting that there would be no reason for a human to visit the sites. Would a crawler be considered able to consent to terms of service?
    – User65535
    Commented Feb 11, 2022 at 0:48
  • Software is not human and can't agree to contracts. The human running the software is the person liable under contract law. Commented Feb 11, 2022 at 1:15
  • 1
    But if no human visits the site or agrees to a contract then does a contract exist? Is google bound by the TOS of every site that they index?
    – User65535
    Commented Feb 12, 2022 at 7:00
  • 3
    @User65535 that's a good point. Does Google (or any other entity that crawls the web) have an army of humans who check crawler targets in advance to search for applicable TOSs and determine which are acceptable to Google to agree to? If the human running the crawler is deemed to agree to the TOS of every site the crawler visits, surely someone would add "visitors must pay me X dollars per visit" to their TOS and dun Google for payment. Since that doesn't seem to be a thing, it seems reasonable to conclude that crawlers don't trigger TOS agreement. Commented Mar 13, 2022 at 16:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .