So a relative of mine works in the real estate business and she wants me to write a Python script that scrapes property prices and lot size from a website, calculate price per sqft, and email her the address and property price if the ppsqft is less than 200 dollars.. Rinse and repeat every 24 hours.
You'd think it should be straightforward enough using urllib beautiful soup and re, but the agency's website would lock my script out every single time and I actually had to define a headers dictionary, change the User-Agent key to the one my browser sends, and use it as the headers argument when requesting data from the website using urllib. And no, the agency offers no API for robots.
My question is, is my script legal? I reasoned that since the same data is offered online for anyone using a browser free of charge, should be the same for a program right? I mean anyone can save the page source as a txt file everyday and point the script at it and the wouldn't know a thing. Granted it would be more manual, but still.