When a defendant counter-sues in a clearly retaliatory way, are there
legal grounds to ask the court to dismiss the suit because it is
retaliatory or is that not a thing?
The procedural facts aren't entirely clear from the question. It isn't clear which court that original lawsuit was filed in, whether the countersuit was a counterclaim in the same lawsuit or a different lawsuit, and if it was a different lawsuit, where that lawsuit was filed.
When a defendant is sued, the defendant is required by law to bring any claims that the defendant has against the plaintiff as a mandatory counterclaim if they are related to the case and is permitted to bring any claims the defendant has against the plaintiff for any reason as a permissive counterclaim.
So, a "retaliatory countersuit" in the same case is entirely ordinary and indeed often required by the rules of civil procedure. There is absolutely nothing improper with filing a countersuit as a "retaliatory" measure unless the countersuit itself (1) is not well grounded in fact (i.e. groundless), (2) is not supported by the law or a good faith argument that the law should be changed (i.e. frivolous), or (3) is vexatious (a term that is not easily defined).
If, instead of bringing counterclaims in the original lawsuit, the defendant brings a separate lawsuit against the plaintiff, the plaintiff could seek to either (1) consolidate the cases if they are filed in the same court system (i.e. a federal case and a federal case, or a New York State case and a New York State case), or (2) move to dismiss the new lawsuit because the claims were required to be brought in the original lawsuit as mandatory counterclaims, or (3) move to stay proceedings in the second lawsuit pending resolution of the first lawsuit, or (4) move to dismiss the claims in the second lawsuit on the merits if it is apparent from the face of the countersuit that it does not state a claim upon which relief can be granted or was filed in the wrong court.
As these myriad options suggest, the procedural steps to take when one party sues a defendant in one court system and the defendant sues the plaintiff in another court system are rather technical and involved (I've dealt with this a few times in the last several years).
New York State has an Anti-SLAPP statute, but it is a quite narrow one:
New York's anti-SLAPP laws, found at N.Y. Civ. Rights Law §§ 70-a,
76-a and N.Y. C.P.L.R. §§ 3211(g), 3212(h), offer protection against
SLAPPs brought by individuals or entities seeking permits or
applications from a government body (like zoning permits) over efforts
of the defendant to report on, comment on, rule on, challenge, or
oppose such application or permission. The statute does not protect
"free speech" in the abstract; it only protects bloggers,
non-traditional journalists, and other online publishers when they
address this narrow class of issues (i.e., the granting or denial of a
public permit or application).
It would be rare that a photographer would fall within the scope of New York's anti-SLAPP statute (SLAPP stands for "strategic lawsuit against public participation").