Many sites prohibit scraping of their content, which is understandable. But I do not clearly understand what approaches of data collection are considered to be scraping, or even allowed to be prohibited at all.

For instance, if my app has an embedded browser which allows a user to load up any website of their choice... and then push a button in my app which will extract all html content from that embedded page onto the user's machine, would this be my responsibility or theirs, to adhere to the site's Terms of Use?

Tools like Scrapy seem to be legal software since it's the users responsibility to follow Terms of Use, not Scrapy's. Note: I plan to sell my software eventually but will never have access to any "scraped" data by users, that's on their end.

My point is: My app contains generic algorithms and logic which are applied uniformly to whichever website the user has navigated to. The logic doesn't target any particular site or structure, but simply has built-in logic that can be used on any site. It's my understanding that it would be the end user's responsibility to ensure they don't use my app for a site which prohibits data collection at all. Is this a correct assumption?

I don't get how I can be prohibited to write generic algorithms that can be permitted on many sites, but prohibited on few. I'm open to reading articles, court cases, or any other related material you can throw at me.

  • What can your app and/or the user do with the saved html content? Why would I, as a user, press that "save" button?
    – Bart van Ingen Schenau
    Dec 29 '16 at 19:29
  • You'd press it because the content then goes through our internal algorithms which organizes any available, relevant data in a way which facilitates the end user's job as opposed to having them perform the same sifting and sorting manually. The logic applied to the data is identical regardless of which particular website a user is navigated to. If they load up a site with insufficient data (say Google's homepage) then nothing happens when they click "save". It's essentially filtering the data through static logic flows in order to potentially save the user time. Whose responsibility here?
    – Nova
    Dec 29 '16 at 19:38
  • 1
    Related (because I don't know the answer): blog.icreon.us/advise/web-scraping-legality
    – BrianDHall
    Dec 29 '16 at 19:54
  • 1
    It seems to me that you are no more responsible than the browser is, for supplying a 'Save As...' menu entry.
    – LSerni
    Dec 29 '16 at 20:00
  • Thanks to you both, that was an informative article Brian, and I agree with you LSerni, it does seem similar. Although 'Save As' saves the content essentially how it's displayed in the browser, whereas my software actively applies logic to the data collected and displays it to the user in a different format.
    – Nova
    Dec 29 '16 at 20:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy