The answer by @hszmv is correct and I fully agree with it. I would add a few additional points:
Not All People Impeached Are Removed From Office
A ban on holding future public office is imposed in only a minority of impeachments, and can only be litigated if the person seeks to hold such an office and has a non-speculative chance of obtaining one. In some ways this is the provision most ripe for litigation in the courts because it lasts the longest.
SCOTUS doesn't have a monopoly on constitutional law
Every single federal judge has the authority and the obligation to rule on constitutional issues that come before that judge. The U.S. Supreme Court does not have exclusive jurisdiction over constitutional issues, and would almost never have original jurisdiction in any case presented over the legality of impeachment proceedings.
It takes a long time to reach SCOTUS and cases can become moot
A case reaches the U.S. Supreme Court, if at all, by being litigated in a trial court and then an intermediate appellate court (possibly with en banc review in a federal appeal, or an intermediate state appellate court and then a state supreme court) first. This takes time, and many impeachment issues are time sensitive.
For example, any issue about the legality of an impeachment proceeding related to the potential removal from office sanction became moot, and beyond the jurisdiction of the courts, in the case of Donald Trump's second impeachment on January 20, 2021. At that point, only the legality of a bar on holding future public offices, if he had been convicted by the U.S. Senate, and if he had sought a future public office anyway, would have been justiciable. Since it takes a long time for litigation to work its way to the U.S. Supreme Court, impeachment issues will often become moot before it can consider them.
Federal courts are barred from considering hypothetical cases
The ban on the U.S. Supreme Court considering hypothetical cases is considered to be a binding rule of constitutional law that is jurisdictional under the "case or controversy clause" of the U.S. Constitution.
Impeached individuals are uniquely unsympathetic but others lack standing
Few people other than the impeached individual would have standing to litigate issues related to impeachment. So, you are left mostly with people whom two-thirds of the U.S. Senate voted to remove from office going to the courts asking to get their jobs back, and an even smaller subset of people whom two-thirds of the U.S. Senate voted to remove from office and a majority votes to bar from holding future public offices, appealing to the courts for relief in litigated cases. People in those shoes are not sympathetic figures. You have to have messed up pretty badly to each such bipartisan scorn. Unsurprisingly, courts are ill inclined to cut them a break in a close case, which makes such people unlikely to resort to the courts for relief.