Does the Geneva Convention apply to the party denying that the war is going on?

Question

Suppose a country R invades another country U without declaring war and insists that the invasion is not a war, but a "special operation", aimed at liberating U's citizens from U's government, or some BS like that. U's position, on the other hand, is that R's invasion of U is an act of war.

The first question: can captured soldiers from R's army claim protection under Geneva Convention? The second question: is R required to respect Geneva Convention rights of the captured U soldiers?

Furthermore, R hired mercenaries from yet another country, S. Those mercenaries, even though acting at R's behest and under R's instructions, are not a part of R's or S's armies. The third question: does the Geneva Convention apply to them too?

Finally, volunteers from other countries are volunteering to help U's resistance, despite not being citizens of U and despite the volunteers' nations refusing to take any part in the conflict. Are those volunteers protected too?

Answer 1

International Humanitarian Law applies to all armed conflicts

The Geneva Conventions are a part of the overarching body of this law. It applies to all armed conflicts, not just declared wars.

An armed invasion by R of U is a conflict to which IHL applies irrespective of if it is a declared war or not. BTW, declared wars outside Africa and South America are virtually unknown since WWII

Answer 2

These questions are addressed explicitly in the text of the third Geneva convention, which is the one that applies to prisoners of war.

Suppose a country R invades another country U without declaring war and insists that the invasion is not a war, but a "special operation"

Article 2:

In addition to the provisions which shall be implemented in peace time, the present Convention shall apply to all cases of declared war or of any other armed conflict which may arise between two or more of the High Contracting Parties, even if the state of war is not recognized by one of them.

The Convention shall also apply to all cases of partial or total occupation of the territory of a High Contracting Party, even if the said occupation meets with no armed resistance.

Although one of the Powers in conflict may not be a party to the present Convention, the Powers who are parties thereto shall remain bound by it in their mutual relations. They shall furthermore be bound by the Convention in relation to the said Power, if the latter accepts and applies the provisions thereof.

(Emphasis added)

Therefore:

can captured soldiers from R's army claim protection under Geneva Convention? The second question: is R required to respect Geneva Convention rights of the captured U soldiers?

Yes to both.

The next question concerns mercenaires, which are the subject of article 47 of the first protocol to the Geneva conventions:

1. A mercenary shall not have the right to be a combatant or a prisoner of war.

2. A mercenary is any person who:

(a) is specially recruited locally or abroad in order to fight in an armed conflict; (b) does, in fact, take a direct part in the hostilities; (c) is motivated to take part in the hostilities essentially by the desire for private gain and, in fact, is promised, by or on behalf of a Party to the conflict, material compensation substantially in excess of that promised or paid to combatants of similar ranks and functions in the armed forces of that Party; (d) is neither a national of a Party to the conflict nor a resident of territory controlled by a Party to the conflict; (e) is not a member of the armed forces of a Party to the conflict; and (f) has not been sent by a State which is not a Party to the conflict on official duty as a member of its armed forces.

These six conditions require an analysis of each individual's circumstances to determine whether that individual is a mercenary.

Therefore:

Furthermore, R hired mercenaries from yet another country, S. Those mercenaries, even though acting at R's behest and under R's instructions, are not a part of R's or S's armies. The third question: does the Geneva Convention apply to them too?

No, because the question as posed assumes that they are mercenaries. However, there are many circumstances under which people hired from S might not be mercenaries. In particular, if they are motivated by reasons other than financial gain or if they are paid on a similar scale to the members of the armed forces, they are not mercenaries under the convention and they are subject to its protection.

Finally, volunteers from other countries are volunteering to help U's resistance, despite not being citizens of U and despite the volunteers' nations refusing to take any part in the conflict. Are those volunteers protected too?

Probably. If the volunteers are inducted into the armed forces, they are certainly not mercenaries (Art. 47(2)(e)). Describing them as "volunteers" suggests that their motivation is ideological rather than material, which would exclude them from being considered mercenaries. Aside from those factors, one can also look at their "material compensation." If they're getting a lot of money (or land, or what have you), and they're not inducted into the U's military, then they are probably mercenaries and therefore not protected.

Answer 3

Assuming that R and U have signed the Conventions (some protections apply to countries that haven't signed the Conventions, but a treaty isn't binding on countries that haven't signed it, although of course the principle that countries can't be held to treaties they haven't signed is itself not binding, so there's nothing practically stopping a country from prosecuting for war crimes by a country that didn't sign, it's just that the country doesn't have the moral standing of "you violated your agreements):

The first question: can captured soldiers from R's army claim protection under Geneva Convention?

Generally speaking, yes. The Conventions (not that it's plural) apply to armed conflict, regardless of what the parties label it as. However, claiming full POW status has several requirements, such as openly displaying marking of the country. So, for instance, if, in this hypothetical of yours, soldiers from R were to enter U while not in R uniforms, and pretending to be local "freedom fighters" native to U, then they would be unlawful combatants, and would have their protections under the Conventions significantly reduced (but not eliminated).

The second question: is R required to respect Geneva Convention rights of the captured U soldiers?

In the sense of the Conventions saying they are required to, yes. In the sense of being practically held to that requirement, not necessarily. You don't get immunity for war crimes by simply claiming it's not a "war".

Furthermore, R hired mercenaries from yet another country, S. Those mercenaries, even though acting at R's behest and under R's instructions, are not a part of R's or S's armies. The third question: does the Geneva Convention apply to them too?

The Conventions give lesser protections to mercenaries. However, there are particular requirements for someone to be considered a "mercenary" for this purpose. Not everyone who might be called a "mercenary" in common parlance falls under this category:

Additional Protocol I defines a mercenary as a person who:
a) is specially recruited locally or abroad in order to fight in an armed conflict;
b) does, in fact, take a direct part in the hostilities;
c) is motivated to take part in the hostilities essentially by the desire for private gain and, in fact, is promised, by or on behalf of a Party to the conflict, material compensation substantially in excess of that promised or paid to combatants of similar ranks and functions in the armed forces of that Party;
d) is neither a national of a Party to the conflict nor a resident of territory controlled by a Party to the conflict;
e) is not a member of the armed forces of a Party to the conflict; and
f) has not been sent by a State which is not a Party to the conflict on official duty as a member of its armed forces.
https://ihl-databases.icrc.org/customary-ihl/eng/docs/v1_rul_rule108

Finally, volunteers from other countries are volunteering to help U's resistance, despite not being citizens of U and despite the volunteers' nations refusing to take any part in the conflict. Are those volunteers protected too?

If they are "volunteers" in the sense of not receiving payment, then they are not mercenaries under the Conventions' definition, and so are fully entitled to their protections, although again being entitled to be protection and being protected are not quite the same thing. Also, R could, hypothetically speaking, set up puppet governments that are, officially speaking, separate sovereign and which haven't signed the Conventions, and claim that these separate entities are killing the volunteers of their own accord.