This came in the famous case of Marbury v. Madison (1803). At the last moments of his term, President Adams and Congress appointed a bunch of new judges to the courts. One of those was Marbury, but his new commission was not delivered by the outgoing Secretary of State John Marshall. The incoming President Jefferson had the undelivered commissions thrown out and ignored by his own Secretary of State (Madison). So Marbury sued to have his commission instated.
There were some peculiar subtleties to the case. One is that Marbury sued directly in the Supreme Court, rather than an inferior court. A law passed by Congress had granted SCOTUS authority to be the trier of certain cases; said law was repealed before the case went before the court. Another oddity is that the Chief Justice was the very same John Marshall that had originally failed to deliver the commission; he did not recuse himself.
The case was very contentious. Congress and the President were both very combative and eager to claim control of "constitutionality" for themselves. Congress would not much appreciate its laws being thrown out, and Jefferson was of the (combative) mind that it was in fact the President who decided the constitutionality of laws (the constitution can be said to explicitly charge him with enforcement of the laws and protection of the constitution), etc. The court risked getting neutered by both sides with just the slightest misstep. The President was sure to ignore any attempt to make him do anything, and Congress would retaliate if anyone but them threw out their laws. And simply declaring themselves impotent was the same set of problems. That the ruling effectively avoided all such problems makes it one of the great examples in SCOTUS opinions to this day, though not all hold it up in a positive way (it arguably intentionally handled the case backwards, so as to yield a ruling rather than a dismissal; some even argue the case may have been manufactured as a way to formally let SCOTUS claim this power).
The ruling basically said the following:
(1) Does Marbury have a right to this commission? Yes, the commission was validly created and the deliverance of it is just a non-discretionary formality, failure of which is an injury that can be fixed (give him the commission).
(2) Do we have the power to force the Executive branch to do something like this? Yes, for non-discretionary duties that are non-political and owed to a particular person, the courts may order the lower ranks of the Executive branch to do things.
(3) Do we even have the authority to hear this case? No. It is held that the law in question would grant us original jurisdiction over this issue, but that is unconstitutional: our original jurisdiction is completely enumerated by the Constitution, and cannot be expanded by legislation.
(4) So what do we do? Nothing, we just dismiss it. Madison can be ordered to do this thing, but won't actually be so ordered since the law required to let us do so is invalid. And we can invalidate that law, but don't actually do so since Congress has already repealed it.
In this way both Congress and the President were left with no real angles to hold a beef over the court, as neither one of them had effectively had their authority directly neutered or compelled. The ruling did nothing but what was already done, and simply asserted the Judiciary had certain powers it could flex later: it could rule on the constitutionality of laws, and it could order the Executive branch to do things. For what it's worth, the Marshall court never really invoked these powers again, seemingly still mindful of a contentious battle for power between the branches of government that could render the courts impotent, but one way or another the ruling successfully claimed the power to decide the constitutionality of things for the courts.