The tags indicate, and I will assume, that the 3D modeler has a copyright but not a design patent, on a computerized bunch of information on a computer that makes it possible for the computer to display on a screen or through output to a 3D printer, a 3D model of something.
In general, copyright protects expressions of ideas (like a particular 3D model) and other works that are "derivative" of the original expression.
How can a 3D Artist know that his model is being sold some where else? . . . And how can the artist tell that this is his model?
This is a practical but non-legal question. Basically, you hire a private investigator or the moral equivalent of one, to look for infringers, or you wait for people who know that you are the 3D artist of the original, to come to you and tell you, or you surf the web in places you think you might find infringements. People who made 2D works often "watermark" them and use a file format that makes it difficult to extract the watermark from; you could use a 3D equivalent of that approach.
If the real question is how can you know that someone else's 3D model was derived from yours and not created independently, a common way would be to bury some distinctive telltale information in that code that most users would never locate, but which is easy to find if you know what you are looking for (like a sequence of individual pixels in an obscure location that spells your name in Morse code).
And with so much modification that he can tell that its his model but
can't prove it?
If you can't prove it, you can't do anything about it. The burden of proof is on people who claim that someone has taken their intellectual property, not on the alleged infringer.
You could try to bring a lawsuit against that person and then get discovery in the course of the lawsuit (e.g. taking a deposition of the suspected infringer), but without pretty good evidence in hand proving your case when you file it, there is a good chance that you will have your case thrown out and be sanctioned for bringing groundless litigation.
After applying some modifications like Turbo Smooting(Or SubSurf
Modif. In Blender) and then decimates or optimizes the geometry and
then sells it again, then is there a way for the original artist to
fight to protect his work?
At some point, if the modified work is only remotely related to the original 3D Model it is no longer a derivative work for purposes of copyright law and ceases to be protected by copyright.
For example, if I use a photograph of a landscape (which would ordinarily be protected by copyright) and then use that as a basis or reference for drawing a cubist painting with surrealist color schemes, the painting is such a departure from the original copyrighted photograph that it ceases to be a derivative work of the original photograph and the original copyright holder can no longer protect the work. This limitation encourages creativity and prevents the marketplace from being tied up in tenuous copyright cases.