Wikipedia's article on Trial in absentia does a pretty good job of explaining this.
In short, Rule 43 of the Federal Rules of Criminal Procedure says a defendant must be present
- at the arraignment,
- at the time of the plea,
- at every stage of the trial, and
- at the imposition of sentence
But there's a list of exceptions to that:
The defendant can voluntarily leave after the trial commences, waiving their right to be present
A disruptive defendant can be removed by the judge's order (after a warning)
In a misdemeanor case, the defendant can submit a written statement to waive their right to be present
And the defendant need not be present for a conference or argument over a question of law. (A question of law is an argument over what the law actually means or how it applies, which is decided by the judge. The jury decides on a question of fact, i.e. whether not the defendant broke the law. Questions of law are generally extremely technical discussions.)
That first exception is the big one: You have to show up for the start of the trial, and then you can voluntarily leave. It might look bad to the judge or jury, but I believe this happens pretty often with politicians and businessmen, where they'll show up on a few key days and other than that let the lawyers handle everything, or be out for part of the trial because they're busy that day.