From State v. Shelley, 929 P.2d 489, 85 Wn.App. 24 (Wash.App.Div. 1, 1997) (emphasis in mine)
consent may be a defense to assault in athletic competitions
the consent defense is not limited to conduct within the rules of the
games, rather it is to the conduct and harm that are the reasonably
foreseeable hazards of joint participation in an athletic contest
The touchings at issue in consent to hazing or consent to a fight, or a gang member's consent to a beating are... factually distinct from "touchings" occurring in athletic competitions.
Citing the Model Penal Code and adopted by states such as PA, ME, and CO, :
...the conduct and the injury are reasonably foreseeable hazards of
joint participation in a lawful athletic contest or competitive sport
or other concerted activity not forbidden by law.
Quoting the comments to the MPC (this also speaks to public policy):
the social judgment that permits the contest to flourish
necessarily involves the companion judgment that reasonably
foreseeable hazards can be consented to by virtue of participation
Quoting Rollin M. Perkins on Criminal Law:
The test is not necessarily whether the blow exceeds the conduct
allowed by the rules of the game. Certain excesses and inconveniences
are to be expected beyond the formal rules of the game.
There is also State v. Floyd, 466 N.W.2d 919 (Iowa App., 1990)
There court noted there [is] a continuum, or sliding scale, grounded
in the circumstances under which voluntary participants engage in
sport... which governs the type of incidents in which an individual
volunteers (i.e., consents) to participate...
As for my statement that consent is not a defense if it is against public policy see STATE v. HIOTT 987 P.2d 135 (1999), 97 Wash.App. 825
consent is not a valid defense if the activity consented to is against
Tangentially related to public policy is that the Sports Violence Act of 1980 failed to pass. We want our sports violent even outside of the rules.
As for the question of whether these touchings are criminal the answer is yes they are. (Unless a jurisdiction has explicitly excepted specific factors from the definition of the crime.) Participation in a sport contributes to a defense against the criminal charges (as seen in the MPC above).
Here's a recent Economist article discussing consent and prosecutorial discretion and concluding that we generally have not passed laws to address the boundaries of sports violence - Fair game? The article acknowledges that it is hard for courts to determine which types of contact athletes have consented to. The result is inconsistent jurisprudence and questions like this one on Stack Exchange.
A) consent and B) prosecutions would fail because of A so cops and DAs don't waste their time.
As far as the guys who get in a street fight and who are criminally charged they can also use consent as a defense in some jurisdictions. But they probably have some other charges too like trespassing, destruction of property, vandalism, disturbing the peace etc. even in jurisdictions that have consent as a defense they may not accept the defense if application is against public policy.
Speaking of public policy, imagine if our police started making arrests anytime an athlete made a physical move against another athlete in violation of the rules of a game.
So those are my two answers. Consent and public policy. I think these two answers also allow for the well known exceptions.
The comments contain stories of anecdotal bar fights where people are getting arrested. Those fights are very different from fights that happen during sports. Frankly, if two guys are walking down the street arguing and one guy elbows the other in the gut and then the second guy trips the first guy who falls but then the two then keep walking together I'm not sure a cop is going to arrest them. In other words the bar fights leading to arrests are far different from a few punches and the shoving that happen during sports.
Oh yeah - you say punching and biting is clearly outside the context of the game. When someone says something is clear it means whatever is about to follow is not clear. Punching and biting are not outside the context. Granted, with the bite you have a stronger argument but the resulting injury is consistent with combat sports. As for a punch in baseball, football, basketball, hockey... etc - anyone playing at a certain level has experienced this stuff and knows it comes with the territory. You learn that lesson in junior high school and you decide whether to continue to play. IOW you consent.
Regarding your edit:
What are are you going to do boy?
I'm gonna punch you in the face.
Well do it then boy!
We can't arrest people who get their bluff called! Imagine the jury seeing this guy calling someone boy and asking to get punched.