Ultimately the answer (in the US) is the US Constitution. The courts pretty much have the unlimited power to interpret the law, and the limits on power mainly pertain to what the state can do. The length of the leash on the government depends on what kind of rights are at stake. The weakest and default limit is known as "rational basis", and comes down to asking whether a government action is rationally related to a legitimate government interest (such as stopping terrorist attacks) plus whether there was due process and equal protection. There are more rigorous standards (intermediate scrutiny, strict scrutiny) in case a law involves a "suspect classification", or in case a fundamental constitutional right is infringed. In the case of strict scrutiny, the government would have to show that it had a compelling interest in the action, the law would have to be "narrowly tailored" (i.e. does that thing and only that thing), and should be the least restrictive way to achieve that result. Roe v. Wade is probably the best-know example of that kind of review, which held that abortion laws "violate the Due Process Clause of the Fourteenth Amendment, which protects against state action the right to privacy".
A court would not order a party to do an impossible act. A court could order a party to do something which the party had argued was impossible, but you can interpret a court's decision to indicate that it rejected the argument. The defendant would not shoulder the burden of proving that the action was impossible, though they would have to counter the government's argument that the act could be performed if they believe it is not possible. (In the Apple case, the argument would probably be some Apple-internal document that says "Yeah, we can do it, but do we want to?"). There is probably an expense-related limit in that the courts would not order Apple to liquidate all of their assets to comply. But: if a case were to end up at the Supreme Court, the court is stricken with mass insanity and arbitrarily orders a defendant to do the impossible, there is only the option of impeachment and Senate trial to remove the offending justices, and that is just not going to happen. There is no higher authority that overturns SCOTUS.
