In US Law, banning speech based on its content is called "Content-based" speech regulation (shocking, I know), as opposed to "Content-neutral" speech regulation like requiring all protests to end before a specific time. Content-based speech regulation can be constitutional if it passes strict scrutiny, but in the case of banning swastikas, it would fall under an even narrower subset of content-based speech regulation called "viewpoint regulation." I haven't found a case where the swastika or Nazi flag was banned in particular, but we can find reasoning that appears to safely protect the peaceful display of the Nazi flag and ideology from government restriction in Police Dept. of City of Chicago v. Mosley, 408 U.S. 92 (1972), a case in which Chicago banned picketing within 150 feet of a school except in the case of labor disputes related to the school. The Supreme Court found that this amounted to viewpoint-based discrimination, writing in the majority opinion:
But, above all else, the First Amendment means that government has no
power to restrict expression because of its message, its ideas, its
subject matter, or its content
Necessarily, then, under the Equal Protection Clause, not to mention
the First Amendment itself, government may not grant the use of a
forum to people whose views it finds acceptable, but deny use to those
wishing to express less favored or more controversial views. And it
may not select which issues are worth discussing or debating in public
facilities. There is an "equality of status in the field of ideas,"
and government must afford all points of view an equal opportunity to
be heard. Once a forum is opened up to assembly or speaking by some
groups, government may not prohibit others from assembling or speaking
on the basis of what they intend to say. Selective exclusions from a
public forum may not be based on content alone, and may not be
justified by reference to content alone.
As with all rights restrictions, a particular restriction may be Constitutional if it passes "strict scrutiny," namely, it:
is necessary to a "compelling state interest";
is "narrowly tailored" to achieving this compelling purpose; and
uses the "least restrictive means" to achieve the purpose.
In general, a ban on the peaceful display of Nazi imagery or promotion of Nazi ideas would fail the first test, as the government does not have a compelling interest to suppress ideas which might be distasteful to some or even the majority of people.
I have seen an argument that because the Nazi regime's stated goal included genocide, that promoting that ideology amounts to advocating violence. Speech which advocates violence or criminality may be criminalized, but only under a specific "imminent lawless action" test expressed in Brandenburg v. Ohio:
Freedoms of speech and press do not permit a State to forbid advocacy of the use of force or of law violation except where such advocacy is directed to inciting or producing imminent lawless action and is likely to incite or produce such action.
This case was brought against a KKK member who advocated various violent acts at a rally. The main distinction is that his speech did not call for specific violence, but merely advocated for it in general. This almost exactly mirrors the rationale for banning the Nazi ideology based on its advocacy of violence, and shows that a ban on such grounds would be unconstitutional. "Imminent" was clarified in Hess v. Indiana to mean that the action must be intended to produce actual lawless action at a specific point in the future, not simply advocate for it in general. Hess was a protester who was being forced off a street by police, said "We'll take the fucking street later" and was convicted of disorderly conduct for it, which the court reversed as his statement "amounted to nothing more than advocacy of illegal action at some indefinite future time" and was therefore protected by the First Amendment.