Say I have a website with content I want to protect.

Say I want people to access that site — but only manually, while sitting at their computer typing the keys — not by writing some computer program that somehow accesses the data remotely or automatically by robots.

Is that even possible to do? Say in the Terms and Conditions agreement maybe I include a clause that says something to the effect:

You agree not to use this site by copying anything automatically or by using any robots; You will only access the site manually.

I seem to recall somewhere on this site somebody mentioned a case that ruled that once someone made the information available, they had no right to control how others would use or access the information? Is that right or am I imagining I read something here that never was?

  • I down voted by mistake and only realised because of your comment. – Terry Dec 17 '15 at 9:49
  • I've removed the down vote. It told me I couldn't when I did before. I'll try supply a better answer on my break. – Terry Dec 17 '15 at 10:27
  • @Terry: Excellent. Thanks. I'll get to it in the morning. – Alexanne Senger Dec 17 '15 at 10:37

Pragmatically, on your server side, there is nothing about an http request from a computer program that is any different than one typed manually, so if the "robot" doesn't intend to honor your demand, there's little or nothing you can do.

Most sites implement a "robots.txt" file in their root directory that defines the site's policy, and many/most of the web crawlers and spiders respect that policy...but it's an honor system.

You can use sophisticated metadata methods to block programmatic access to your site...refuse new connections less than five seconds apart, for example.

Edit: There's no law that says you have to honor an http GET request. You can allow or refuse them at your most capricious whim.

|improve this answer|||||
  • You could also make users sign up for the site and login with a CAPTCHA. That would discourage all but the most determined robots. – Patrick87 Dec 17 '15 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.