This is an open question.
California's Unruh Act prohibits discrimination in public accommodations on the basis of political affiliation.
This same issue has come up previously, in a case where four neo-Nazis showed up wearing swastika pins at a German restaurant. When they refused to remove the pins, the restaurant called the police to remove them.
The Nazis sued the restaurant under the Unruh Act, which prohibits various forms of discrimination in public accommodations (restaurants, hotels, etc.) Although the Unruh Act does not specifically mention discrimination on the basis of political ideology, the California Supreme Court has interpreted its list of classes as describing, not limiting, the classes eligible for protection, which it has also explicitly said include political affiliation:
Whether the exclusionary policy rests on the alleged undesirable propensities of those of a particular race, nationality, occupation, political affiliation, or age, in this context the Unruh Act protects individuals from such arbitrary discrimination.
Marina Point, Ltd. v. Wolfson, 30 Cal. 3d 721, 726 (1982).
Based on these interpretations, the trial court refused to dismiss the Nazis' case against the restaurant, and the parties eventually settled without going to trial. Read commentary about the case here.
So it seems clear that the Unruh Act prohibits discrimination on the basis of political affiliation. Because wearing a swastika indicates that you affiliate with the National Socialist German Workers' Party or one of its offshoots, the Unruh Act probably prohibits a business from discriminating against customers on the basis of wearing a swastika.
But federal law may pre-empt the Unruh Act.
But the problem doesn't end there, because intepreting the law that was creates potential conflicts with federal public-accommodations law, employment law, and the First Amendment.
For instance, federal law prohibits the creation of a "hostile environment" in terms of both providing public accommodations on equal terms regardless of race, and in terms of equal employment opportunity regardless of race, color, religion, sex or national origin. Proving a hostile environment can be pretty difficult, but if you could demonstrate that allowing swastikas on premises created a hostile environment for customers or employees, you'd then have a strong Supremacy Clause argument that the Unruh Act can't be enforced to require the admission of swastika-wearing customers.
Beyond that, businesses have First Amendment rights on generally the same terms as natural humans. There's a reasonable argument to be made that those businesses, in banning swastikas, are communicating a First Amendment-protected anti-Nazi message, or that they are exercising their right to control who speaks in the forum that they control. If the court were to accept either of those arguments, it would again mean that the Unruh Act probably could not be enforced to benefit those wearing swastikas.