Say I have a website with content I want to protect.

Say I want people to access that site — but only manually, while sitting at their computer typing the keys — not by writing some computer program that somehow accesses the data remotely or automatically by robots.

Is that even possible to do? Say in the Terms and Conditions agreement maybe I include a clause that says something to the effect:

You agree not to use this site by copying anything automatically or by using any robots; You will only access the site manually.

I seem to recall somewhere on this site somebody mentioned a case that ruled that once someone made the information available, they had no right to control how others would use or access the information? Is that right or am I imagining I read something here that never was?

  • I down voted by mistake and only realised because of your comment.
    – Terry
    Dec 17, 2015 at 9:49
  • I've removed the down vote. It told me I couldn't when I did before. I'll try supply a better answer on my break.
    – Terry
    Dec 17, 2015 at 10:27
  • @Terry: Excellent. Thanks. I'll get to it in the morning. Dec 17, 2015 at 10:37

1 Answer 1


Pragmatically, on your server side, there is nothing about an http request from a computer program that is any different than one typed manually, so if the "robot" doesn't intend to honor your demand, there's little or nothing you can do.

Most sites implement a "robots.txt" file in their root directory that defines the site's policy, and many/most of the web crawlers and spiders respect that policy...but it's an honor system.

You can use sophisticated metadata methods to block programmatic access to your site...refuse new connections less than five seconds apart, for example.

Edit: There's no law that says you have to honor an http GET request. You can allow or refuse them at your most capricious whim.

  • You could also make users sign up for the site and login with a CAPTCHA. That would discourage all but the most determined robots.
    – Patrick87
    Dec 17, 2015 at 15:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .