Can I ban robots?

Question

Say I have a website with content I want to protect.

Say I want people to access that site — but only manually, while sitting at their computer typing the keys — not by writing some computer program that somehow accesses the data remotely or automatically by robots.

Is that even possible to do? Say in the Terms and Conditions agreement maybe I include a clause that says something to the effect:

You agree not to use this site by copying anything automatically or by using any robots; You will only access the site manually.

I seem to recall somewhere on this site somebody mentioned a case that ruled that once someone made the information available, they had no right to control how others would use or access the information? Is that right or am I imagining I read something here that never was?

Answer 1

Pragmatically, on your server side, there is nothing about an http request from a computer program that is any different than one typed manually, so if the "robot" doesn't intend to honor your demand, there's little or nothing you can do.

Most sites implement a "robots.txt" file in their root directory that defines the site's policy, and many/most of the web crawlers and spiders respect that policy...but it's an honor system.

You can use sophisticated metadata methods to block programmatic access to your site...refuse new connections less than five seconds apart, for example.

Edit: There's no law that says you have to honor an http GET request. You can allow or refuse them at your most capricious whim.