There isn't a national answer, but every state has laws that set that limit – there is a lot of variation. California defines and criminalizes homicide in the penal code here, and section 195 starts setting forth circumstances where homicide is excusable. 197.2, for example, includes when
committed in defense of habitation, property, or person, against one
who manifestly intends or endeavors, by violence or surprise, to
commit a felony, or against one who manifestly intends and endeavors,
in a violent, riotous or tumultuous manner, to enter the habitation of
another for the purpose of offering violence to any person therein
The jury then has to decide if the facts match the various conditions enumerated throughout the criminal code as justifications, so there are specific instructions that the judge reads, given here. (This is a generic self-defense instruction, not just aimed at homicide). They are told that the accused "may use reasonable force to protect that property from imminent harm", and that reasonable force is "the amount of force that a reasonable person in the same situation would believe is necessary to protect the property from imminent harm". So if the prosecution cannot prove "beyond a reasonable doubt that the defendant used more force than was reasonable to protect property from imminent harm", they are supposed to acquit. Now, suppose that you're sitting at a sidewalk cafe and some passing perp reached for your cell phone on the table. Blowing off his head would be an unreasonable degree of force.
There are additional instructions aimed at specific situations, such as the right to eject a trespasser that allows you to use reasonable force to eject a trespasser who refuses to leave, and again that degree of force is whatever the reasonable man would do in the circumstances that the victim reasonably believed existed (regardless of whether there was an actual threat). Deadly force is specifically allowed in self-defense against a "forcible or atrocious crime", which includes robbery. This does not include deadly force to prevent vandalism.
You will have noticed that it comes down to the jury making a subjective judgment call as to whether the degree of force used was beyond what the hypothetical "reasonable man" would use.
To clarify in response to the comment, the thing that all of the justified force defenses have in common is that the force be reasonable in order to stop the aggressor. If you can (safely) stop the vandal by smacking him once, but it would be satisfying to turn him into paste, you have to remain unsatisfied. Individuals can use force in defense of self or property, but only the law can use force punitively (and as I assume you know, that never includes beating a person up).