When traveling abroad, how does a nation state know that a couple is married?
Usually by accepting the couple's verbal statement. Some countries include a remark in a married person's passport with the name of the person's spouse, which is obviously more likely to be accepted in case of doubt, but not all countries do this.
The usual formal proof of a foreign marriage is a certified copy of the foreign marriage certificate with an apostille or similar legalization whereby the foreign ministry of the country where the document was issued (or, in some federal contexts, of a constituent entity) attests to the legitimacy of the official who certified the copy and to the authenticity of the certification. This is a fairly burdensome process that is normally associated with proving family relationships in connection with establishing residence in a foreign country. Nobody is going to expect visitors to have such documents. For a casual or temporary purpose such as hospital visitation rights, the certified copy might be helpful even if it is not legalized.
On the other hand, in a country where homosexuality is criminalized, the certificate could be used as evidence against the couple.
What laws are in place regarding hospitalization and next of kin?
This will be governed by national law. Human rights treaties probably have something to say about it, at least indirectly, but if your spouse is in the hospital that's not going to be of much use. Local law and the hospital's policy will control.
What happens to a same-sex couple traveling to a nation that is not amicable to same-sex unions, say India?
This is rather too broad, especially as it will depend on the sympathy of the people involved as well as on local law. The outcome could be anywhere from according full spousal visitation rights to the arrest of both spouses followed by conviction, imprisonment or possibly worse, deportation, and a ban on reentering the country.