4

Problem: I would like to create a plugin for my website. This Plugin makes it possible to recognise a shared link in a comment field. Afterwards the plugin fetch the link and scrape the meta data of the shared source (link). After successfully scraping the Link will reformat to a nice "Card" with thumbnail and description text (Note 1). A kind of card instead of just a link. Or better the way Google does it.

Question: Is it allowed to "scrape" metadata without prior consent of the website operator?

My thoughts on this: Since I am only scraping the metadata by machine and it is there to be used by search engine bots, social platforms anyway, that one is allowed to do it.

Note 1 : (only pages that are public and explicitly allowed to be crawled by search engines).

Update

Example for better understanding

Below you can see an example of the metadata I read from a page. Below you can see the editing on my page with my own style.

<title>Kanada: Abgeschossenes Flugobjekt hatte zylindrische Form - DER SPIEGEL</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1,user-scalable=no">
<meta name="MSSmartTagsPreventParsing" content="true">
<meta http-equiv="imagetoolbar" content="no">
<meta name="apple-itunes-app" content="app-id=424881832">
<link rel="manifest" href="https://www.spiegel.de/public/spon/json/manifest.json">
<meta name="theme-color" content="#e64415" media="(prefers-color-scheme: light)">
<meta name="theme-color" content="#D03D12" media="(prefers-color-scheme: dark)">
<meta name="google-site-verification" content="lwpSnwoyvMkHS9nHmLBZuYOashAHfooOHSHeA_KS7ek">
<meta name="robots" content="index, follow, noarchive, noodp, max-snippet:-1, max-image-preview:large, max-video-preview:-1">
<meta name="Googlebot-News" content="index, follow">
<meta name="copyright" content="DER SPIEGEL, Hamburg, Germany">
<meta name="email" content="[email protected]">
<meta name="author" content="DER SPIEGEL">
<meta name="date" content="2023-02-12T07:08:39+01:00">
<meta name="last-modified" content="2023-02-12T07:08:39+01:00">
<meta name="locale" content="de_DE">
<meta name="description" content="Es war innerhalb von zwei Tagen das zweite Objekt im Luftraum über Nordamerika, das von US-Kampfjets abgeschossen wurde. Nun gibt es erste Angaben der kanadischen Regierung, wie der Flugkörper aussah.">
<meta name="news_keywords" content="Ausland, USA, Kanada, Justin Trudeau, Volksrepublik China, Spionage, Luftfahrt">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:site" content="@derspiegel">
<meta name="twitter:title" content="Erste Erkenntnisse der kanadischen Regierung: Abgeschossenes Flugobjekt hatte zylindrische Form">
<meta name="twitter:creator" content="@derspiegel">
<meta name="twitter:image" content="https://cdn.prod.www.spiegel.de/images/c431237d-0187-4cc6-95f6-c3bc8c12ede1_w1280_r1.77_fpx59.37_fpy50.jpg">
<meta property="og:title" content="Erste Erkenntnisse der kanadischen Regierung: Abgeschossenes Flugobjekt hatte zylindrische Form">
<meta property="og:type" content="article">
<meta property="og:url" content="https://www.spiegel.de/ausland/kanada-abgeschossenes-flugobjekt-hatte-zylindrische-form-a-8134d1f8-6259-46d3-b1bc-0de4446380fc">
<meta property="og:image" content="https://cdn.prod.www.spiegel.de/images/c431237d-0187-4cc6-95f6-c3bc8c12ede1_w1280_r1.77_fpx59.37_fpy50.jpg">
<meta property="og:description" content="Es war innerhalb von zwei Tagen das zweite Objekt im Luftraum über Nordamerika, das von US-Kampfjets abgeschossen wurde. Nun gibt es erste Angaben der kanadischen Regierung, wie der Flugkörper aussah.">

And now a screenshot of my page where the metadata is displayed. enter image description here

What I store in my database? The content from the metatags: title, description, author, sourcelink, tags and the link to the thumbnail.

1
  • 2
    Not a legal note, but you may be interested in scraping for a Creative Commons or equivalent machine-readable license. wiki.creativecommons.org/wiki/… This could be used to only show previews for those sociable websites who permit their visitors to use their content in a fair way. Feb 11, 2023 at 20:06

4 Answers 4

14

You are asking the wrong question. It should be: When you have downloaded the content and metadata, what are you allowed to do with it and what is forbidden?

Somebody owns the copyright to the text and images in the thumbnail. This could be the operator of the third party website, or that site has licensed the content from yet another party. You haven't licensed it from anybody. So you can watch the content in accordance with the TOS, and your computer can evaluate the metadata to do it, but you cannot display it on your own site.

Details will differ between jurisdictions, of course. You might also be held responsible for illegal content in the thumb you generate.


Follow-up: There seems to be some question of what 'thumbnail' and 'card' mean in this context. This answer assumes a somewhat scaled-down representation of the content of the entire page, not just a collection of actual metadata like content length and expiry.

10
  • Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, on Law Meta, or in Law Chat. Comments continuing discussion may be removed.
    – Pat W.
    Feb 13, 2023 at 14:22
  • @o.m. thx for your comment. I wonder why my example above is a copyright infringement? can you explain that to me please? Feb 13, 2023 at 16:04
  • @MaikLowrey, I would say the meta tag with name=description contains copyrightable text, not just metadata.
    – o.m.
    Feb 13, 2023 at 16:26
  • @o.m. If this were the case, why does the author put copyrighted material in the meta description? That makes no sense to me. Feb 13, 2023 at 16:30
  • @MaikLowrey, the web protocols are a technical category, copyrighted is a legal category.
    – o.m.
    Feb 13, 2023 at 16:42
3

Question: Is it allowed to "scrape" metadata without prior consent of the website operator?

Yes, it is. The whole point of meta data is that you know what this page is about. If the page did not want copy-righted material in it, it would just not put it into the meta fields.

The preview you see when you paste a link into a chat window, for example in Teams or Facebook, is built on the Open Graph Protocol, an optional part of a website that the owner can include if they want their page displayed in such a neat little preview.

This protocol is explicitely made to give a third party (for example Facebook) enough data to present said preview/thumbnail of the page. It is not limited to any one company, Facebook is just an example, any company can use it. This is what is was made for!

I would assume that the owner voluntarily giving any visitor and any third party the information how to view their link as a preview, in a structured way that assumes this will be used by third parties, automatically means they want the third party to show it this way.

A note: only metadata though. Leave the actual non meta data content alone. That is theirs, they gave it to their direct viewers in a way that the direct viewers can see it. The fact that you technically can scrape it does not give you permission to display it anywhere.

4
  • Good Morning nvoigt! Thx for your good answear. Your answer coincided with my assumption. I have already written my programme for this and it does exactly what you say. It only reads the meta data and if this is not enough also the twitter and og metag tags. It is important to mention that I store these on my server so that I don't have to "crawl" the individual links every time I call up my page. Feb 12, 2023 at 6:51
  • 1
    By "store", do you mean you store the finished thumbnail, for caching purposes for links that were used in the past? That should be fine I think. However, the content behind the link may have been changed, so it might not be advisable to do so. I would trust your (or actually, your users, because you should do this client side) caching mechanism for anything they downloaded from the internet, that should have methods to deal with it, send only the minimum data needed. Your program should request fresh data every time and trust the system, instead of reinventing just another cache on top.
    – nvoigt
    Feb 12, 2023 at 7:00
  • I do not save the thumbnails on my server. Only the link to the preview image. You are right, of course, that one could give an update service to the programme. Thank you for this creative idea. Note I have edited my answer to make the example more vivid. Feb 12, 2023 at 7:10
  • I try to write a summary of our entire conversation: law.stackexchange.com/a/89185/49036. What do you think about? Feb 13, 2023 at 18:37
0

I used to scrape the web professionally for a search engine.

The Web is a consensual operation. You send

 GET /page.html HTTP/1.1 
 Host: example.com 

And then their server does whatever the heck it chooses to do. It can give you dead air, error return or content. They can't force terms & conditions onto you merely for giving the above request; that is not a legally binding signature.

But once they give you content unauthenticated, they have no leg to stand on to claim that you "did them wrong". However, there are a few exceptions to that rule.

Don't DOS them

The problem is, when a stone cold novice writes their first web crawler, they usually write something like this:

 foreach url (list.of.URLs) {
    fetch url; 
    new.urls = searchTextForURLs (url); 
    append new.urls to list.of.urls; 
    sleep(5000);     # this line is forgotten
 }

This particular example is spidering, which is much worse. But regardless --

Invariably without fail, a novice's first URL scraper will forget to include that last line. Even if it's scraping a multitude of sites, it ends up doing a lot of consecutive URLs on a single site, and that can hammer a small site to death. That is a DOS (Denial Of Service) and they have a cause of action.

Don't get lost in infinite recursion

It's common for sites to be designed, for completely legitimate reasons, to have an infinite namespace of URLs. You will never stop discovering URLs, just because of the way they structure them. Or they may syndicate content, so their /shop/ space may entail Amazon's entire catalog. Crawling this can lead to DOS, but even if you get that sleep(5000) line in there and pace yourself, doing a page every 5 seconds for eternity still adds up to significant server burden. And again they may have a cause of action.

Respect robots.txt and other directives

<meta name="robots" content="index, follow, noarchive, noodp, max-snippet:-1, max-image-preview:large, max-video-preview:-1">

Your crawler needs to parse robots.txt and similar on-page directives like the above Robots meta-tag, and respect what it is telling you. They don't really have much standing to complain if you are crawling stuff they don't want crawled, yet, they "forgot" to exclude it in Robots.txt or other means.

Of course once they notice your crawler, the first thing they'll do is amend robots.txt to exclude it. So check it periodically. I've been on the receiving end of many a user complaint because we cached robots.txt (so we weren't hammering robots.txt with every query; that would be rude) ... as a result, their change didn't result in the immediate abatement of traffic that they were hoping for, and they had to wait a few hours.

If you want to be polite, it also helps to give your crawler a unique name, so they can call it out specifically. In that case you need to respect robots.txt directives against that particular crawler name.

0

There have already been answers and they have not all definitively answered the question. I have thought about this again and would like to share it here. In my opinion, the best answer was from nvoigt. That's why I'm summarising it again and adding my thoughts.

Basic consideration

It can be assumed that anyone who puts a website on the internet and operates it wants to make the information accessible to the public. If you don't want to do that, you can protect your site with different possibilities: html access, programmatic authentication etc. Those who do not want "hard" protection may only want their site not to be indexed by search engines.

This is where robots.txt comes into play. With this you can exclude your entire page from the search engine index or only parts of it. By default, every page visited by a search engine robot is included in the index. Exceptions are, of course, websites that violate the guidelines of the search engine operator.

Metadata describe and give a short summary about the respective page. The site operator can enrich the meta data with more information through special notations with more information. Page operators make such metatag optimizations in order to be found even better. For example, through appealing short descriptions. To generate a better click through rate. The presence and use of so-called OG tags in the head area are also indicators that show that the site operator is actively spreading his content. This is an conclusive / implied act.

If you also explicitly mention in the robots.txt file that your page may be indexed, you also explicitly want the metadata to be read. The site operator can also explicitly exclude search engines and user agents. The bot operator should also respect this.

Conclusion and answer

If all factors are given, nothing stands in the way of reading out the metadata. And so I would claim that there is no copyright infringement.

But as soon as you earn money with the snippets on your page, another level comes into play. In Germany, it is called the Leistungsschutzgesetz für Verlage (ancillary copyright law for publishers). The Leistungsschutzgesetz is primarily understood as a right to protect investments. Ancillary copyright is also referred to as "neighbouring rights". Sometimes also as "neighbouring rights", in Anglo-American terms "neighbouring rights". Thus, neighbouring rights apply alongside copyrights. In Germany, publishers had complained that Google and Co. were making money from advertising by displaying snippets on their site. The publishers wanted a piece of that. A long and tough process ended with Google, Facebook and others paying the publishers a share of the revenue. So anyone who pursues all the above-mentioned points and non-commercial goals can relax here. But you should still not feel safe. Because as soon as the small and non-commercial site becomes successful (traffic), one or the other company would certainly get in touch with its lawyer.

Final personal comment

The answer of my question is yes and no. This example shows once again that the original idea of the internet. Free knowledge transfer for everyone no longer applies. It's always about stupid money.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .