I am wondering if there are specific clause in the freedom of speech law in the U.S. that could get you in trouble.
You cannot be successfully sued nor charged with a crime in the US merely because a movie, book, or other work is violent. Consider the rape scenes in the movie Deliverance, or John Hersey's The Wall, a 1950 novel which describes in detail the Warsaw Ghetto and the massacre and torture of its inhabitants by the Nazis during WWII. Or Plevier's Stalingrad, a 1940s novel about the aftermath of the Battle of Stalingrad, also during WWII, and the suffering of both Russian and German soldiers and civilians there. (Both novels were later filmed in less violent versions.) Many other examples could easily be mentioned.
Triggering anxiety or PTSD in a reader or watcher would not be grounds for a suit either -- attempts to bring such suits have been dismissed.
Claims that violent works inspire or encourage crime or anti-social acts have led to attempts to regulate such works, from comic books in the 1950s to video games in the 1990s. None of these works have prevailed. The so-called Comics Code Authority survived for a time because it was a strictly voluntary process -- some publishers never submitted content to the CCA censors. The Motion Picture Production Code was similarly voluntary, as is the current system of movie ratings.
Works that defame someone can be the subject of libel suits. But to win such a suit in the US, the plaintiff must show that statements were made that were false, and not matters of opinion. The plaintiff must also show that such statements were "of and concerning" the plaintiff, that is that they were specifically about the plaintiff. The plaintiff must in most cases further show actual harm to the plaintiff's reputation, unless the false statements were of the limited category known as per se libel, which give rise to a presumption of harm. These include false claims of improper business or professional conduct, false claims that the plaintiff committed a serious crime, or false claims of sexual improprieties. Moreover, in many cases in the US the plaintiff must show actual malice, that is that the defamer made the false statements knowing them to be false, or with reckless disregard for whether or not they were true. This applies when the person bringing suit is a public official or public figure. Mere general violence will not be grounds for a libel suit.
Several comments have asked if violent depictions could constitute incitement and therefore be unlawful. The current rule in the US is that speech (or writing) can only be punished as incitement if it is "directed to inciting or producing imminent lawless action and is likely to incite or produce such action". This language comes from Brandenburg v. Ohio, 395 U.S. 444 (1969), and has thus been stable for over 50 years. Prior to that the US Supreme Court used the "clear and present danger" test, first adopted in Schenck v. United States, 249 U.S. 47 (1919) The actual passage from Schenck was:
The question in every case is whether the words used are used in such circumstances and are of such a nature as to create a clear and present danger that they will bring about the substantive evils that Congress has a right to prevent. It is a question of proximity and degree.
In Brandenberg the Supreme Court held Ohio's criminal syndicalism statute unconstitutional, because that law generally criminalized simple advocacy of violence.
Under neither test would a book or movie that simply depicted violence be subject to legal action simply for that reason. A work would have to specifically advocate violence. Under the current Brandenberg test the work would have to urge people to violent lawbreaking right away, when it is read or watched. and actual lawbreaking would need to occur, or be reasonably likely to occur, for legal action to be constitutional. Actions against incitement under this test have been largely aimed at speeches before crowds or groups, urging them to riot or take other violent and unlawful action right then. While a book, movie, or video could be incitement under this test, it would need to be much more than just violent, it would need to strongly advocate present unlawful violence. That is not what the question asked about.
You cannot be sued for producing a violent movie (or writing a violent book, or creating a violent game) in the US: that is the essence of the First Amendment. You would encounter legal problems if you actually kill people, or potentially animals. There is a voluntary system of video content in the US which tags items for various things that could cause distress, and includes violence as one factor. Since this is voluntary, you don't have to participate (though if you pretend to participate and give your remake of House of 1000 Corpses a G rating implying a rating that was not issued by the association, you could be sued). It was almost illegal in California to sell violent games to those under 18, but that law was found to be unconstitutional, in the case Brown v. Entertainment Merchants Association.
Maybe, if your work causes real harm to someone.
I am not a lawyer, so take this with a grain of salt (I don't know specifics or precedents or anything like that), but in order to win a lawsuit, generally the plaintiff needs to show that the acts they're suing over has caused them real harm. In terms of violent material, there's a few ways that I can think of that might happen:
First, you might defame a real person by saying that they're a violent criminal when they actually aren't, or defame a real group by depicting them as being violent. For instance, if you produced a film about a group of real-life pacifists secretly being serial killers, then they might be able to sue you for defamation.
Second, if the violence in your work itself causes someone real harm. For instance, if your movie has a graphic rape scene, and it causes a rape victim's PTSD to trigger and they kill themselves while quoting your movie in their suicide letter, maybe their family would be able to sue you for the psychological damage your movie caused their relative.
Finally, it's possible that the way you depict the violence might cause real physical harm to someone; for instance, if your graphically violent fight scene if full of strobe lights as a stylistic technique to emphasize each hit, and it causes epileptic seizures among vulnerable people viewing it, and you've failed to warn your viewers about this at the beginning of the movie, then it's possible that they might sue you over any harm that they might suffer as a result of the seizures that your film has induced.
Obviously, the specifics of each of these things would likely vary from state to state, so if it's important to you, I would recommend that you talk to an actual lawyer who's qualified to practice in your state.
Supposedly “real” snuff films have been written and filmed (by real I mean the promotion says they were real, in reality no actors killed, all deaths either faked or news of some sort). There’s no way you could *successfully be sued unless you did something stupid like say it was a specific and well know real person.
If you were to do a supposed documentary or expose in the vein of the Blair Witch Project, but said that your bad guy was a real person and either used or created some somewhat plausible facts to back that up (say an unsolved murder or rape in every city that a band or promotional tour visited, using real unsolved rapes, murders or disappearances where they exist and making them up when they didn’t) you could get sued for defamation. Not because your movie or book was too violent but because people might suppose that your accusations were truthful, damaging the reputation of the person involved.
*sucessfully as in having to pay damages when represented by competent counsel, other negative consequences are certainly possible.