Has there been a case where a Canadian court refused to enforce a court order from the United States?
There are many, but an interesting one that comes to mind is Gulf Oil Corporation v. Gulf Canada Ltd. et al.,  2 SCR 39, 1980 CanLII 192 (SCC). The Supreme Court of Canada got involved in a case where the parent company in the U.S. was requesting production of evidence (through the American courts) from its Canadian subsidiary. As there was legislation preventing the release of this information, the subsidiary applied to the Canadian courts for permission, which was refused.
Canada refuses extradition to the United States for any crime that could have a capitol punishment (death penalty). This isn't malicious as the United States will only sign extradition treaties where the crime has a similar crime and penalty under both legal systems. Since Canada has no death penalty, it meets the treaty terms. There are some work arounds as they will extradite if the state that is trying the crime abolished the death penalty OR the prosecution gives explicit guarantees that it will not seek the death penalty.
Conversely, the United States will not extradite someone who is about to be tried under Canadian Hate Speech Laws, as there is no U.S. equivalent. In addition, defamation is harder to secure an extradition for because U.S. has stricter burdens of proof than most countries in the world