0

I am currently learning how to web scrape the web and I encountered a major problem.

It seems that every website has some part of its terms of use talking about web scraping being strictly prohibited.

My question is:

Let's say that I scrape a website like Glassdoor or Indeed which has all the public information, without logging into an account. By doing that, I technically do not break the terms of use because I never agreed to respect them in the first place, right? Is it legal? Am I right or wrong in that instance?

Thank you.

| improve this question | |
2

Its etiquette to abide by a websites robots.txt file, if that file disallows scraping of certain sites, but whether scraping breaches anything other than a boilerplate terms of use is very much undecided in law right now (you don't state a jurisdiction, so I will assume the US) in that various arguments have been tried and both succeeded and failed depending on the individual cases merits.

There is currently a lawsuit winding its way through the US courts specifically around scraping of public information - this particular lawsuit is between a company called HiQ Labs and LinkedIn.

Interestingly enough however, in 2017, HiQ Labs was granted an injunction against LinkedIn to prevent them from blocking scraping by HiQ Labs while the case proceeded.

So, unfortunately, the answer is very much "watch this space".

| improve this answer | |
-3

You aren't supposed to do it if the site says no. Even Google and the big search engines try to abide by the site's rules.

Also, if you cause harm by sending too much traffic to the server you can get in trouble. https://www.forbes.com/sites/ericgoldman/2015/03/24/qvc-cant-stop-web-scraping/#2da04b343ca3

| improve this answer | |
  • This answer doesn't really answer the question. The first paragraph says "you aren't supposed to" but not if it it is actually illegal to do it anyway. The second paragraph describes a rather extreme case where the scraping actually caused direct damage in form of a denial-of-service. It doesn't really apply in a case where the scraping activity doesn't impair the functionality of the scraped site. – Philipp Jun 5 '19 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.