The questions what you have to do and what you should do are different in this situation.
You have to / have not to
You don't have to comply to an Abmahnung (§ 97a UrhG). But if you don't and its found to be legit you will have to pay a severe fine. The idea behind the Abmahnung is that it allows Urheberrechtsinhaber to inform you of the law infringement and make you stop it without the action requiring a court, thus reducing the load on the court system and reducing the time the law infringement persists.
You don't have to comply to a DMCA notice (This might change with TTIP and other following trade agreements), but depending on your target audience you probably want to. Read on Richtlinie 2001/29/EG for the EU-Equivalent, its legally binding in Germany and implemented with the Abmahnung.
You have to react to a German court order (but its expensive if it comes to this). Usually you will receive an Einstweilige Verfügung (Preliminary injunction) that you have to stop spreading the data. Some weeks after that there will be a formal court hearing.
Most likely you should do the following: Implement an easy way to report Urheberrechtsverletzungen (like a form saying 'This document [link] should not be in public circulation. I own the rights, see here [link or document upload].') Then see how much false-positives you get.
In the scientific community you will most likely not get that much false-positives on this form, so you can hide the content, then test the claim and after that remove it.
This enables people to remove offending content from your site without you having to pay the Abmahnung fee.
Your legal situation
As your content is user generated, the users are liable for their uploaded content, as long as you can produce their identity, usually an IP address and Date+Time will do. See 'Störerhaftung' for details.
You can (and depending on your legal form, you have to) name a German court, but any German court can take responsibility (see the Landgericht Hamburg cases on Urheberrechtsverletzungen on why that's a problem for you).
Form of an Abmahnung
An Abmahnung is form-free, so there are formal requirements (§97a(2) UrhG), but not a strict form in which they have to be presented. An E-Mail is sufficient (to the address published in your whois or Impressum). In this the sender must supply information about the data they wish to see removed and a reason why they want it. They may (and will) charge you a fee for the Abmahnung.
Reaction to an Abmahnyng
You have to react to an Abmahnung (that you wish to comply) by all of:
- Removing the offending content.
- Replying to the Abmahnung by a form-free statement that says you will not continue the offense. (Strafbewehrte Unterlassungserklärung) A professional Abmahnung will contain one with a huge fine if you break it, that you can sign and return. See #1 below on why that's maybe not what you want.
- Paying their fee.
You can react to an Abmahnung (that you don't wish to fully comply) by either of:
- Replying with a statement that reduces the fee and fine of the original Abmahnung (e.G. if they want you to pay 100.000 € for a text snippet and you think that's unjustified). You then have to pay the fee you set (if you set one) and they have to sue you if they want more. This is a "modifizierte Unterlassungserklärung". You have to remove the offending content of course for this to be meaningful.
- Reply with a statement saying that you will not comply. They may sue you then.
- Don't reply at all. They may sue you then.
- You could talk to the other side and try to strike a deal, but be sure to have a record of this conversation and maybe let your lawyer do it for you. If they sign a contract with you, they may not sue you afterwards.
- Send them an Abmahnung to stop sending you additional Abmahnungen. They may sue you over that.
If you expect them to sue you can:
- Do nothing, you should then expect an Einstweilige Verfügung (see above).
- Write a Schutzschrift where you document why the Abmahnung is illegitimate so there won't be an Einstweilige Verfügung. This does not stop a judge to issue one anyway, but she has to provide a written reason why your Schutzschrift is not applicable.
- File a Feststellungsklage so that they can't sue you anymore (because there can be only one case for one offense). This has the advantage that you can select the court, but the disadvantage that you have to pay for it.
In either case you want to talk to a layer and have sufficient reserves, this can get quite expensive. If I remember correctly: you can reduce fine and fee in court if it otherwise would cause your business to come in existential danger, but I don't find a citation for this at the moment.