Is scraping real estate data illegal in anyway?

Question

So a relative of mine works in the real estate business and she wants me to write a Python script that scrapes property prices and lot size from a website, calculate price per sqft, and email her the address and property price if the ppsqft is less than 200 dollars.. Rinse and repeat every 24 hours.

You'd think it should be straightforward enough using urllib beautiful soup and re, but the agency's website would lock my script out every single time and I actually had to define a headers dictionary, change the User-Agent key to the one my browser sends, and use it as the headers argument when requesting data from the website using urllib. And no, the agency offers no API for robots.

My question is, is my script legal? I reasoned that since the same data is offered online for anyone using a browser free of charge, should be the same for a program right? I mean anyone can save the page source as a txt file everyday and point the script at it and the wouldn't know a thing. Granted it would be more manual, but still.

Answer 1

Usually the answer to "is it illegal?" to methodically copying data from another website (or copy from any other published work, be it a book, movie, music, article, internet resource, etc.) is clear: of course it is.

It doesn't matter if it is a one time scrape or a continuous crawl and copy of some or all data; it's still copying and there are copyright laws.

Read more:

I have a question about copyright. What should I read before I ask it?

Is it legal to scrape a website and create my own database?

https://law.stackexchange.com/questions/tagged/copyright

Copying something regularly is not Fair Use. See:

http://fairuse.stanford.edu/overview/fair-use/what-is-fair-use/

https://law.stackexchange.com/search?q=fair+use

To be sure, read the TOS for the site you want to scrape. If there is no TOS, or the TOS says you can scrape and copy, go for it. If the TOS says scraping is not allowed, you may still be able to license the content; ask them.

I'm sure the site has safeguards to prevent scraping, and subverting those will also be covered in the TOS. Read it. And the legal penalties for breaking their TOS. The TOS is a legal contract you sign when you access the site.

I reasoned that since the same data is offered online for anyone using a browser free of charge, should be the same for a program right?

No. The TOS will outline the difference in usage between humans and bots and scripts.

Answer 2

Just because everyone using their browser is authorised, that doesn't mean you are authorised. You are in danger of committing the criminal offence of accessing a computer without authorisation.

As you noticed, you had to circumvent measures that were designed to make your unauthorised access impossible. So if you went to court, you wouldn't have any excuse. And then it can be argued that what you are doing is copyright infringement, and interference with someone else's business.

In other words, don't proceed without getting a lawyer. Especially not if the idea is to "help a relative". Tell your relative that you want to get legal advice first, and that she should pay for the lawyer. If she doesn't want to do that, you know how much she cares if you get into trouble.

Answer 3

Yes and no to illegal but will be against TOS. Copyright covers the data on websites and you have a licence to view the data, the reason this might not be a copyright infringement is that copyright does not cover facts (or fake facts) so it depends on if the property price as a fact (it is a fact that they are selling that property for that price) or creative data (it is their opinion that, that property is that price so that property's value is a peace of creative work) I can not find any documentation on which it would be.

now to the TOS bit, just because it is free to access does not mean you can do anything with it. Most websites now have Terms Of Service which will say that you have permission (a licence) to view its content but not change or redistribute it. Most TOSs also have a clause that you cant use automated scripts to view the site. So if it is legal you will have to fight then as they will try to stop your script from working at lest because you are costing them in terms of bandwidth and using at least one of the limited connections on the server and the cost of the electricity the server used to serve you the page.