Meet Bob. Bob is a supreme court justice and has a hobby of engaging in private, noncommercial, and legally questionable activities. As a result of those activities, Alice is now suing Bob in his capacity as a private citizen. The outcome of the case revolves around a novel constitutional question not covered by existing precedent, and so the case works its way up through the lower courts and is ultimately appealed to the supreme court. Because Bob is the defendent, all the other supreme court justices have a conflict of interest, so what then? Does Bob just recuse himself to argue the case, and the other justices continue business as usual? Or does the court just refuse to hear the case, and let the relevant lower court figure it out? Does it matter whether Alice or Bob is the one to appeal? Does it change things significantly if Alice is an ambassador, so the supreme court has original jurisdiction, and thus can't send it to a lower court?
I know that if every judge had a conflict of interest, they could ignore it, but it seems wrong to apply that rule in this case, since that would result in Bob being both a defendent and judge, which just seems wrong.