What are the limits of website scraping under the Computer Fraud and Abuse Act?

Question

I have read several news in the EFF's website about court appeals and fights related to web scraping.

I am a programmer and have always been worried about what are the limits of scraping.

Normally, I think any public site can be scraped, as 'viewing' a website is not different from that.

That seems to be the EFF interpretation too. And the terms of service seem not to count, as far as I understand.

I would like insight in the following situations:

• What if we scrape using a username/pwd?
• What rule applies when the 2 parties are in different countries?

Answer 1

The U.S. 9th Circuit Court of Appeals ruled on September 9 that the U.S.'s Computer Fraud and Abuse Act (CFAA) is not violated when a company scrapes public websites.

Though, I suspect that scraping identity or access codes when access was not granted to you could be considered theft.

Also, see: Is it legal to share a script that scrapes a website and presents in a different format?

I've always enjoyed programming scrapes...gratifying and relaxing, like sudoku or Tetris. :-D

-shelDev

Answer 2

The CFAA is filled with words and phrases like "intentionally", "knowingly", "with intent", which allows web scrapers that play by the rules

Search engine crawlers are also web-scrapers, with some extra steps. In general, search engines respect what's in the robots.txt. That generally means they do not use username/passwords to scrap.

Most websites want you to scrap them - that's how Google works. What they don't want is you making too many requests per minute, which takes computing power away from humans, or accessing personal user information, which is why scrapers don't use username/passwords regularly.

The CFAA explicitly mentions unauthorized access

(1) having knowingly accessed a computer without authorization or exceeding authorized access, and by means of such conduct having obtained information that has been determined by the United States Government pursuant to an Executive order or statute to require protection against unauthorized disclosure for reasons of national defense or foreign relations, or any restricted data, as defined in paragraph y. of section 11 of the Atomic Energy Act of 1954, with reason to believe that such information so obtained could be used to the injury of the United States, or to the advantage of any foreign nation willfully communicates, delivers, transmits, or causes to be communicated, delivered, or transmitted, or attempts to communicate, deliver, transmit or cause to be communicated, delivered, or transmitted the same to any person not entitled to receive it, or willfully retains the same and fails to deliver it to the officer or employee of the United States entitled to receive it;

So stealing a password is illegal, but the CFAA goes further. Even using a legit password for unauthorized purposes is illegal.

The case of most interest is probably U.S v Aaron Swartz. A Harvard faculty member a with valid JSTOR login created a scraper to download all the JSTOR content. In order to get around scraper protections on JSTOR, he plugged into a data closet directly and did some other technical wizardry. He was charge by the government with felonies and committed suicide.

In this case, he had a legal username/password but wasn't allowed to scrap the site. The government felt he exceeded his access authority and charged him, even though he had authorization to access the site.

Your other question about people in 2 different countries is very easy and completely unrelated. The law that applies is the country where the computer running the scraper is located. That's why it's next to impossible to prosecute the hackers that hijack facebook accounts and push forex scams - they're not in America.