Under US federal law, specifically the The Civil Rights Act of 1964, a restaurant or other place of public accommodation may no refuse service because of sex, race, national origin, or religion. Relatively recently, discrimination on the basis of sexual orientation has been held to be sex discrimination. These are the protected classes.
Under the ADA discrimination on the basis of a disability is unlawful, and reasonable access to disabled people must generally be provided. Some states have passed anti-discrimination laws that add additional protected classes in those states.
In general, a restaurant ore other business is free to refuse service to people on grounds other than membership in a protected class. (Note that things closely associated with a protected class will generally be protected. For example, forbidding crosses or burkas will generally be treated ass discrimination by religion. Forbidding an "afro" hair style might be treated as race discrimination.)
A business could, for example, refuse to serve left-handed red-headed people. or people carrying books of poetry. This would probably be ill-advised, but legal, unless and until these are added to the list of protected classes.
The question says:
I understand that restaurants need not follow the government mandated mask policy for their private place.
Depending on just what the OP means this may be a mistake. Some mandates require a business not to admit an unmasked person (or did, I am not sure if any such are still in effect). Such a mandate, if issued with valid authority, would be binding on a restaurant.
Some states have recently passed laws forbidding businesses from requiring masks. I believe these are still under litigation, but if they hold up a restaurant would be forbidden to refuse service to an unmasked person in those states.
I do not know of any US states that have forbidden businesses to refuse service to a masked person. In the absence of any such law, it would be lawful to refuse service to a person wearing a mask, to the best of my understanding.
If a current law, regulation or order requires bushiness to enforce masking on all customers, then the issue could not really come up, because a business that is on;y willing to serve unmasked people could not operate in compliance with such a regulation.