Under Article VI of the US Constitution, the federal constitution and valid federal laws are the supreme law of the land, and judges in every state are bound to apply them regardless of anything in the laws or constitution of any state.* If a state legislature passes a law banning same-sex marriage, a state court is required under the federal constitution to instead apply the federal constitution (as interpreted by the US Supreme Court in Obergefell v. Hodges) and rule as though same-sex marriage is legal.
When it comes to a federal court, things are even easier: a state cannot command a federal court to do anything. A federal court’s authority is laid out in Article III of the federal constitution and in federal statute, which (per Article VI) is supreme over anything in the laws or constitution of any state.
There have been times where states attempted to challenge federal supremacy. Normally, this is handled by going to federal court. Decisions in state courts can ultimately be appealed to the US Supreme Court, which can reverse them if they incorrectly applied federal law. Most of the time, that’s the end of matters: when a federal court rules, state officials comply.
Occasionally, that’s not enough. If a state disobeys the orders of a federal court, they can be enforced by federal agents. If even that isn’t enough, the President can deploy the armed forces to uphold federal authority.
* There are situations which are more complicated, like when something is a federal crime but not a state crime. I can do more research on that if you want, but for now I’m going to leave it at “it’s more complicated.”