I am based in Germany and have a question regarding web scraping and how this is legal for existing sites.

Two examples:

  • google.com they have a search engine and scrape the internet. They do not only display links to the search results but they also provide previews to the content (the two lines below each link in the search result). As my own website also appears in google I know for sure there is not a contract or agreement between google and every site appearing in the results.
  • kayak.com they do have a search engine for flights for example. When I used it, I can see for example a flight from FRA to NYC coming from delta.com . When going to http://delta.com/robots.txt I can see though that the api / endpoint to search for a flight is disallowed. Kayak has for sure used this result to display it on their own website and link back to kayak. Does kayak.com have an explicit contract or agreement with all other pages linked to?

Is the conclusion from this that I am allowed to create a SaaS service with a search engine on top and display the results to the websites without their explicit agreement? Or is this search-engine case always allowed for websites?

My follow-up question is, of course, what when this is commercial use? Take Google Flight for example - they don't earn money directly with their search ( I think ) but still they display ads on the page and with this are making money indirectly with the search results (by deriving the fitting ad from the result coming from another website, e.g. delta.com).

Can someone shed a light on this for me :)?

1 Answer 1


There’s a difference between the two examples

Search Engines

There have been a number of cases decided in the USA about the legality of web scraping, cacheing, thumbnails, etc. for the purposes of search. Other nations tend to follow the American lead when it comes to internet law.

The TL;DR is that the courts have determined it to be fair use within the boundaries of that purpose. Scraping for other purposes has not yet been litigated.

Flight data

The fact that Delta has a flight from FRA to NYC at a given time on a given day is not subject to copyright because facts aren’t artistic or literary expression. Kayak can take note of this fact and reproduce it in their own words because Delta doesn’t own the fact.

  • Also Kayak might have a contractual relationship with an API provider and pay commission on bookings made via the API.
    – Lag
    Feb 21 at 10:15
  • 1
    @Lag that might be true! I heard in the "how I built this" podcast that they did not have that in the beginning. They just scraped and offered search.... Feb 21 at 11:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .