Boxing doesn't have to be regulated by a government agency, but it generally is regulated in this way (1) because states are concerned about regulating fighting with an intent to do physical harm to the other player (which is strictly speaking against the rules even in contact sports like hockey), which would otherwise violate criminal laws unless there was a statutory exception, and (2) because they want to regulate gambling associated with the events (which, again, they wouldn't have to do, but in practice, all states completely ban gambling subject to specific regulated gambling systems, or de minimus exceptions).
Historically, many boxing "leagues" were corrupt and manipulated both to punish people disliked by organized crime by forcing people to participate to cleanse what would otherwise be illegal gang vigilante activities and to cheat people betting on matches by fixing the results of fights. Dueling was also historically a major social problem among the political and economic elites in parts of the country.
States have the power to ban boxing, football, wrestling, judo, lacrosse, hockey, rugby, fencing, and any other contract sport if they wish to do so, under the U.S. Constitution. States allow contact sports because they choose to do so as a matter of policy, not because the right to engage in contact sports is protected by the First Amendment.
There is a constitutional right to have sex with other consenting adults under the U.S. Supreme Court's Lawrence v. Texas, 539 U.S. 558 (2003) decision (which is not based primarily on the First Amendment right to free speech), but there is no constitutional right for consenting adults to authorize physical harm to each other in an agreed form of combat.
Free speech does not protect contract sports such as boxing because the protected conduct does not primarily exist to communicate ideas, and also because the state has a compelling state interest in preventing people from physically harming each other that goes back to the oldest political theory justifications for the existence of the state in the first place by scholars such as Locke and Hobbes.