As said elsewhere both countries have jurisdiction (can prosecute the accused within the limits of their laws).
What would actually happen would depend upon several factors, the most important being not where the crime was committed, but who apprehended the accused. One country might try for extradition, the other may or may not agree to it. Montana has the death penalty, if the crime was committed there by a Canadian citizen, they might want to take custody in order to avoid having their citizen killed by another country.
Double jeopardy may or may not apply depending upon the law of the country claiming jurisdiction (for instance if our hypothetical killer was Iranian, he could serve his time in either the US or Canada and still be re-prosecuted in Iran).
Basically international law exists only as a voluntary amendment/adoption to a countries laws, and so there is no defitive statement that can be made without describing the exact circumstances. For instance if the crime had been drug dealing with the money and drugs being tossed across the border, our Iranian subject wouldn’t have had to worry about re-prosecution as Iran law would have been satisfied with his conviction and punishment by either.
But it’s a nigh universal fact that you can’t imprison someone that is beyond your reach, so...it do nes upon whee you are.