For example, say the USA and Canada sign a treaty. But then Canada feels that the USA broke the treaty, but the USA maintains that they didn't. Is there some "court of law" equivalent where both sides can argue and have a decision reached as to guilt/consequences/damages that must be paid? Or is it simply up to diplomacy, and the nations just have to reach some new agreement based on threats/leverage?
Depending on the detail agreed within the treaty (or convention) one avenue is the International Court of Justice, because...
...Some treaties or conventions confer jurisdiction on the Court. It has become a common international practice for international agreements - whether bilateral or multilateral - to include provisions, known as jurisdictional clauses, providing that certain categories of disputes shall or may be subject to one or more methods of pacific dispute settlement. Numerous clauses of this kind provide for recourse to conciliation, mediation or arbitration; others provide for recourse to the Court, either immediately or if other means of dispute settlement fail.
The ICJ website (linked above) lists treaties etc which contain clauses relating to the jurisdiction of the Court in contentious proceedings.
Although there does not appear to be one between Canada and the USA, Note that:
The inclusion or omission of a treaty is without prejudice to its possible application by the Court in a particular case.