Criminal Cases
In criminal cases, federal prosecutors decide which cases to prosecute under federal law, while state prosecutors decide independently which cases to prosecute under state law. Sometimes there is coordination between the two, but there doesn't have to be. Under the dual sovereignty doctrine, the federal government and the state government can both prosecute and punish someone for the same crime without regard to each other's actions.
Federal crimes are always prosecuted in federal courts by federal prosecutors. State crimes are always prosecuted in state courts by state prosecutors (or by private citizens in rare instances).
About 2% of all criminal cases (depending upon how you measure it - by charges brought, by number of criminal defendants, by trial conducted, by time spent incarcerated, etc.) are brought in federal court. Federal court criminal dockets are highly distinct from the overall mix of crimes prosecuted in the U.S.
Federal courts mostly handle immigration offenses (over which federal courts have exclusive jurisdiction since states are forbidden from enacting immigration crimes under the U.S. Constitution), white collar crimes, regulatory offenses (e.g. violations of federal environmental and security and tax crimes), crimes with high mandatory minimum federal sentences (mostly related to drugs and possession of firearms by felons), terrorism, bank robbery, and serious ordinary crimes arising on Indian Reservations.
Less serious crimes committed on Indian reservations involving Indian tribe members (basically misdemeanors and petty offenses and punitive civil violations like traffic tickets) are enforced in the tribal courts of the relevant Indian tribe, rather than in state courts or in federal courts. The answer by Jen to this question discusses this and links to discussions of when tribal courts have jurisdiction which is a complicated subject.
Notably, the death penalty can be imposed for violation of a federal crime by a federal court even if the crime is committed in a state that has no state death penalty.
State courts handle all other crimes. In particular, the state courts handle the overwhelming majority of serious violent crimes committed outside Indian Reservations (much more than 98% of such cases).
A side consequence of this reality is that federal prisons are much less scary places to be incarcerated than state prisons, because they have so many fewer people convicted of serious violent crimes.
One exception to the strong general rule that state courts exclusively handle state criminal cases, is that after all direct appeals are exhausted in a state criminal cases, the convicted defendant can appeal his or her case to the U.S. Supreme Court if the defendant alleges that the U.S. Constitution or a federal law was violated in connection with the conviction, and in habeas corpus review in federal courts to collaterally attack such a conviction (e.g. for reasons not discernible from the trial court record) after all state court appeals and collateral attacks have been exhausted.
The D.C. courts of the District of Columbia, and territorial courts in other U.S. territories including Puerto Rico, function essentially like state courts for most purposes, although there are some subtle differences legally between them and state courts (e.g. the Bill of Rights applies directly and not merely via the 14th Amendment in these courts because they are part of the federal government).
Military Justice and Military Tribunals
As an aside, there is also a whole separate justice system under the Uniform Code of Military Justice that mostly applies to quasi-criminal offenses committed by soldiers (a small number of cases involving civilians on military bases or traveling with military units are also handled in these forums). This system is also a federal one but is not connected except at the U.S. Supreme Court level at the very top, to the civilian federal criminal justice system. It even has its own jails and prisons.
Finally, there is also a U.S. system of anti-terrorism military tribunals created after 9-11 and based at a U.S. military base in Guantanamo Bay, Cuba. This was basically a failed system and only dozens of inmates with cases a couple of decades old continue to be processed in this system with new case filings in this system having ended long ago. No or almost no cases there have been resolved with final trials on the merits convicted defendants of the charges against them.
Civil Cases
In civil cases, the person bringing the lawsuit decides whether to bring it in the federal courts or the state courts in the first instance.
But some cases (e.g. copyright and patent enforcement, bankruptcy, cases between U.S. states, and cases involving diplomats) have to be brought in the federal courts, while other cases (domestic relations, probate, and junk fax cases) must be brought in state courts.
The junk fax law is weird since it is the only federal law of which I am aware that authorizes private lawsuits that can be brought only in state courts.
Federal courts have jurisdiction (usually concurrent with state courts except as noted above) over cases with a U.S. government party, over cases with a party who is a foreign country or diplomat from that country, over cases between U.S. states, over cases in which the claims in the original complaint arise under federal law including treaties to which the U.S. is a party (federal law defenses and federal law counterclaims don't give rise to federal court jurisdiction), over real estate ownership disputes arising under the laws of more than one state. Note that with the few exceptions noted above (e.g. intellectual property cases) cases arising under federal law can be tried in state courts if none of the defendants remove the case to federal court.
Federal courts can also consider cases where there is a question of state law over which state courts don't have exclusive jurisdiction (e.g. probate and domestic relations) if it is related in some way to a claim above over which the federal courts do have jurisdiction (this is called pendent and supplemental jurisdiction).
Federal courts in civil cases also have jurisdictions over "diversity cases" arising under state law where the amount in controversy exceeds $75,000, in large class action lawsuits, and in interpleader lawsuits (i.e. suits between parties from different states with competing claims to own property that the person who possesses the property have asked a court to resolve for them).
Unless a case brought in state court is removed by a defendant to federal court in a timely fashion, it stays in state court.
Federal court civil cases are less skewed relative to the state courts in the mix of civil cases that they handle than in the sphere of criminal cases. But civil lawsuits by federal government plaintiffs enforcing regulatory laws and federal government legal rights (notably actions to collect student loans administered by the U.S. Department of Education), bankruptcy cases, cases against government officials alleging civil rights violations, copyright and patent infringement cases, and large class actions make up a significant part of the civil docket in federal courts.
Another very important subset of federal civil cases (numerically) are habeas corpus petitions and other petitions by prison inmates. The federal courts have jurisdiction over collateral attacks on state court criminal convictions after all state court remedies to attack the convictions are exhausted, and over civil rights lawsuits against prison officials.
The vast majority of federal habeas corpus petitions and prisoner's petitions are dismissed summarily on the recommendations of federal magistrate judges (deputy federal judges appointed by federal judges) to federal district court judges. But the cases that survive are a major relief valve in death penalty cases, other very long sentence cases in state court whose correctness is disputed (especially in states with dubious quality state criminal justice systems). They also regulate the worst of the worst state prison mistreatment situations. Ultimately, close to half of U.S. death sentences are overturned in some phase of appeal or collateral attack on the sentence, often in federal court habeas corpus petitions.