2

If a website has cloudflare protection or some sort of anti-robot protection, then it's pretty obvious that the website does not want to be scraped. However, the thing with programming is that there are always workarounds regarding protections/restrictions.

Let's take the recent reddit API pricing change for example (https://arstechnica.com/gadgets/2023/06/reddits-new-api-pricing-will-kill-off-apollo-on-june-30/). It created a lot of drama and the high price basically kills 3rd-party apps.

However, while APIs are convenient, it is quite possible to use a program-controlled browser to automate tasks, such as outputting all the titles of the posts on a certain day.

Obviously, reddit loses money if people all do it this way instead of using its APIs. Reddit can put out captchas and anti-program protections, but again, those can be bypassed somehow.

Or, if this action of web scraping is illegal, then reddit would be able to sue whoever's scraping their site for money. So, is it illegal?

2 Answers 2

3

No, it’s not legal

The Computer Fraud and Abuse Act makes it illegal to access a computer beyond authorization. You are not authorized. We know this because you are using technical means to bypass protection. Like someone picking a lock on a door.

8
  • But this isn't like bypassing a password. If you just do things manually, you'll get in no problem, and all the content/services are available to the public anyways. If a website detects that you are using automation (such as spamming), then they can ban you according to their TOS, but you could also manually do the same thing. I'm just not sure how strict this "beyond authorization" is.
    – No Name
    Commented Sep 12, 2023 at 23:06
  • 1
    @OneCuriousPerson if they screen to stop automated access, then automated access is not authorised.
    – Dale M
    Commented Sep 12, 2023 at 23:11
  • You're not making unauthorised access to the content if it's freely available to everyone with an internet connection and a Web browser, though? You're not hacking their servers, you're just browsing their web pages.
    – nick012000
    Commented Sep 13, 2023 at 7:50
  • @nick012000 you are making unauthorised access to my house if you come in through the window instead of the door. Because I allow one doesn’t mean I allow the other.
    – Dale M
    Commented Sep 13, 2023 at 10:50
  • @nick012000: OP's proposal is to avoid becoming legally aware that scraping is unauthorized, but I don't it very convincing. If the court asked why OP didn't use the API, an honest answer (i.e., API access forbids scraping) would probably be enough to fail. Regardless, this isn't sustainable: Reddit could explicitly revoke any purported authorization by sending a C&D notice. The BrandTotal lawsuit suggests that such a notice would be sufficient.
    – Brian
    Commented Sep 13, 2023 at 15:44
1

It doesn't matter much that they don't want their website to be scraped. What matters is that there are effective protections to prevent scraping, so you would be accessing their computer while evading effective protection methods. And that may make it criminal.

The fact that you are clever enough to circumvent these protection methods doesn't make them not effective. Like locking the door of your home; someone who is handy with locksmiths' tools doesn't have any excuse to open your door.

5
  • What is the bar for when it is "effective"? I've seen some web sites that just place a javascript box restriction that can be bypassed (e.g. by disabling JavaScript). So if I turn off JavaScript in my browser and visit the site, now I'm a criminal? That does not seem correct.
    – Brandin
    Commented Sep 12, 2023 at 7:46
  • As far as I know, effective protection mechanisms are only legally relevant if they are things in place that are specifically to prevent copying. Accessing by automated means (aka "scraping") is something that site operators may want to prevent for various reasons, but as far as I know, those reasons have nothing to do with copyright, so I don't see why protection against scraping has legal significance. For example, if I manually copy your site, that is still (probably) copyright infringement, even though I passed the captcha, etc.
    – Brandin
    Commented Sep 12, 2023 at 7:48
  • 1
    @Brandin that's irrelevant: the Computer Fraud and Abuse Act and others make it illegal to access a system that is protected - The effectiveness of the protection is not a factor in that law, the non-allowed access triggers criminality of the action. Effective-protection comes into play where bypassing copy-protection methods comes into play in other countries.
    – Trish
    Commented Sep 12, 2023 at 8:37
  • 1
    "That may make it criminal" seems like a non-answer.
    – bdb484
    Commented Sep 12, 2023 at 10:52
  • @Brandin Can your grandma "scrape" it?
    – gnasher729
    Commented Sep 12, 2023 at 13:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .