An oath of office cannot be legally enforced through the courts, other than to demand that officials take it in order to take office, and to bar people who have taken it and then engaged in treason or sedition from holding public office in any federal, state, or local office, military or civilian, in the United States.
This is something which is required of all federal and state and local public officials under the U.S. Constitution, which states in the third paragraph of Article VI:
The Senators and Representatives before mentioned, and the Members of
the several State Legislatures, and all executive and judicial
Officers, both of the United States and of the several States, shall
be bound by Oath or Affirmation, to support this Constitution; but no
religious Test shall ever be required as a Qualification to any Office
or public Trust under the United States.
The U.S. President's parallel oath of office is found in the last paragraph of Section 1 of Article II of the U.S. Constitution. It states:
Before he enter on the Execution of his Office, he shall take the
following Oath or Affirmation: — "I do solemnly swear (or affirm) that
I will faithfully execute the Office of President of the United
States, and will to the best of my Ability, preserve, protect and
defend the Constitution of the United States."
But, while taking the oath is required, it is also the case that:
An oath is not justiciable. The FBI cannot investigate adherence to
oaths because they are not enforceable as codified law. A prosecutor
cannot establish that an oath has been broken by proving certain legal
elements beyond a reasonable doubt, and a judge cannot adjudicate it.
There is one exception to this rule, however. Section 3 of the 14th Amendment to the U.S. Constitution states:
No person shall be a Senator or Representative in Congress, or elector
of President and Vice-President, or hold any office, civil or
military, under the United States, or under any State, who, having
previously taken an oath, as a member of Congress, or as an officer of
the United States, or as a member of any State legislature, or as an
executive or judicial officer of any State, to support the
Constitution of the United States, shall have engaged in insurrection
or rebellion against the same, or given aid or comfort to the enemies
thereof. But Congress may by a vote of two-thirds of each House,
remove such disability.
So, an oath of office does impose a justiciable duty to not engage in insurrection or rebellion against the United States, or give aid or comfort to the enemies of the United States, upon pain of not being able to hold any state or federal, military or civilian public office without a two-third majority waiver from Congress (in addition to any other consequences that may flow from this conduct without regard to having taken an oath). But, this is a very slight slap on the wrist indeed for committing treason or engaging in sedition (which are criminally punishable by decades in prison or death) after having sworn this oath.
Justice Joseph Story noted in his "A Familiar Exposition of the Constitution of the United States" (1842) that:
A President, who shall dare to violate the obligations of his solemn
oath or affirmation of office, may escape human censure, nay, may even
receive applause from the giddy multitude. But he will be compelled to
learn, that there is a watchful Providence, that cannot be deceived;
and a righteous Being, the searcher of all hearts, who will render
unto all men according to their deserts. Considerations of this sort
will necessarily make a conscientious man more scrupulous in the
discharge of his duty; and will even make a man of looser principles
pause, when he is about to enter upon a deliberate violation of his
official oath.
The purpose of an oath of office is simply to deny public offices to people who are not willing to publicly state that the legal system in which the people taking them will operate is legitimate.
An oath to support the constitution is the modern equivalent of the feudal European ritual of "bending the knee" to one's legitimate feudal superior. This ritual was highlighted, for example, in the Game of Thrones books by R. R. Martin, which is based loosely on the fights over legitimacy in 15th century England's "War of the Roses".
Another legacy of this historical tradition is the duty of someone in military service to salute his or her superior officer.
As a historical note, these seemingly toothless rituals have, historically, been surprisingly effective a screening out hard core extremist leaders trying to bring down a government based upon claims that the whole system is illegitimate when new regimes are established, and following civil wars and insurgencies.
Many regimes, democratic and non-democratic, in Western political history, have imposed similar requirements. When they have done so, this has seriously influenced the political tactics used by factions that deny the legitimacy of the state and its incumbent leaders. Simply taking the oath undermines one's credibility as an insurgent leader, even if one does so in bad faith.
In the United States, oaths of office were also key preconditions to the post-conflict settlements if the Whiskey Rebellion, the Civil War, and some of the lesser known episodes of a century of Indian Wars.
As a more recent example, the requirement of an oath of office has materially influenced the 20th and 21st century political tactics of Sinn Féin, a political movement in Ireland, seeking to make the U.K. political subdivision known as Northern Ireland, which it deems illegitimate, a part of the Republic of Ireland.
Notably, the U.S. Supreme Court in the case Powell v. McCormack, 395 U.S. 486 (1969), distinguished between a Congressional determination that someone has not satisfied the constitutionally established requirements to have an oath of office administered to them after they have been elected, which can be made by majority vote, and a Congressional determination that a member of Congress should be expelled by a two-thirds majority vote of the house of Congress to which the member of Congress has been elected. A Congressional vote to expel a member of Congress is a non-justiciable question that is not tied to the content of the member's oath of office.
Similarly, judicial and executive branch officials in the federal government may be impeached by Congress, and removed from office through that process, only for "high crimes and misdemeanors" and not merely for otherwise failing to live up to their oath of office in a non-criminal manner (although what constitutes "high crimes and misdemeanors" is also a non-justiciable political question).
Some legal authorities, however, have held that in some contexts, the oath of office does reflect an intent to empower executive branch officials to refuse to enforce what the President believes to be unconstitutional legislation, when its constitutionality has not yet been definitively adjudicated yet.
But, courts have also held, for example, that a member of the U.S. military does not have standing to bring a suit claiming that military action in which he is involved was unconstitutionally authorized. This decision was reached on the grounds that the claim that the service member was forced to violate that service member's oath of office does not constitute a justiciable "injury in fact" to that member for standing to sue purposes. Smith v. Obama, No. 16-843, 2016 WL 6839357 (D.D.C. Nov. 21, 2016) at page 10 (as discussed here).