Is web scraping legal without authentification?

Question

I am currently learning how to web scrape the web and I encountered a major problem.

It seems that every website has some part of its terms of use talking about web scraping being strictly prohibited.

My question is:

Let's say that I scrape a website like Glassdoor or Indeed which has all the public information, without logging into an account. By doing that, I technically do not break the terms of use because I never agreed to respect them in the first place, right? Is it legal? Am I right or wrong in that instance?

Thank you.

Answer 1

Its etiquette to abide by a websites robots.txt file, if that file disallows scraping of certain sites, but whether scraping breaches anything other than a boilerplate terms of use is very much undecided in law right now (you don't state a jurisdiction, so I will assume the US) in that various arguments have been tried and both succeeded and failed depending on the individual cases merits.

There is currently a lawsuit winding its way through the US courts specifically around scraping of public information - this particular lawsuit is between a company called HiQ Labs and LinkedIn.

Interestingly enough however, in 2017, HiQ Labs was granted an injunction against LinkedIn to prevent them from blocking scraping by HiQ Labs while the case proceeded.

So, unfortunately, the answer is very much "watch this space".

Answer 2

You aren't supposed to do it if the site says no. Even Google and the big search engines try to abide by the site's rules.

Also, if you cause harm by sending too much traffic to the server you can get in trouble. https://www.forbes.com/sites/ericgoldman/2015/03/24/qvc-cant-stop-web-scraping/#2da04b343ca3