It depends on the jurisdiction. pjc50 has given a link to the law in the UK. In the USA, what you are talking about is known legally as "fighting words."
The US Supreme Court has recognized that some speech is so offensive that it has no First Amendment protection. Such "fighting words" "by their very utterance, inflict injury or tend to incite an immediate breach of the peace."
This doctrine, and successive decisions upholding it, have been criticized because it limits the apparently absolute protection of speech in the First Amendment. Here is an exhaustive discussion of that idea.
In some states, an immediate retaliation, in the form of a punch to the face (the "Fist Amendment"), was once viewed as fair. At least, the police and other authorities recognized that there was little chance of an insulted person who delivered violence in return being convicted by a jury. This was enforced unfairly, of course, according to the prejudices of local law enforcement.
That is no longer the case anywhere in the US. The "fighting words" are disorderly conduct, and a punch in return is a battery. The police will arrest both of you, or if there is no serious physical harm, more likely suggest you call it a night and sober up.
A civil suit by either party is a possibility, and will inevitably result in a countersuit. Neither person can go to jail for losing a civil suit, no lawyer should get involved in such a suit, and a sensible judge will tell the parties that nobody wins, so shake hands.