Must/may the court give that instruction?
No. It would be error for the court to give that instruction.
The only privilege for which an adverse inference instruction is generally authorized in civil litigation is the 5th Amendment privilege against self-incrimination. This is because when you invoke it, you are implicitly asserting that your testimony could be used against you, if you gave it, to show that you were guilty of a crime, and because a relevant question will be about events pertinent to the lawsuit.
An adverse inference is also not allowed for invoking the 5th Amendment when you are a defendant in a criminal case because that would undermine its purpose in the the criminal justice system.
An invocation of a marital privilege, in contrast, merely implies that you are married, which is not something that would normally and naturally suggest that you did something for which there is civil liability.
One could probably imagine a fact pattern in which being married was a disputed issue that could give rise to liability (e.g. under the "family car doctrine"), of course, in which the invocation of the privilege would estop A from asserting a defense on the ground that he isn't married to A's wife (either at the time of the communication if the confidential communication privilege is raised, or at the time of the testimony, if the right to not testify against a spouse privilege is raised, as the case might be). Tricky cases would involve people who were unmarried at the time of the accident but subsequently married. But, outside very unusual facts, people generally don't deny that they are married in a lawsuit and then try to assert the marital privilege in a lawsuit.
Does the answer change depending on whether it is the litigant or the
spouse who invokes the privilege?
No.
