Copyright protection is about certain acts, and not about relationships between products. Copyright law says that the creator of an original work hold the exclusive right to copy and to authorize creation of derivative works. Copyright law does not say that anybody can freely create derivative works as long as they are different to a certain extent. So if you take an original Mario and modify it a teeny bit, that is a violation of copyright; if you take an original Mario and modify it hugely, that is a violation of copyright.
Degree of similarity is relevant on some cases when the factual question arises whether the allegedly-infringing work is based on some protected original. This is most obvious in music cases, where all baroque music has some similarity to all other baroque music, all death metal has some similarity to all other death metal, and so on. There is not a legal quasi-statutory standard for measuring substantial similarity in music. The scientific underpinning of such a standard would be based on (weighted) combinatorics and the idea that there are only so many tunes possible (that would be a huge number, until you get to the "within a genre" condition).
It seems obvious (by your "admission") that the derived works are based on protected works, so Nintendo's permission is required to legally create such works. However, you do or would hold copyright in your unauthorized derived work. Without a trail of evidence such as a SE questionSE question pointing to the connection, the derived images might be hard to connect to the originals. In addition, you may be able to avail yourself of a "fair use" defense, in case you get sued by the original creator. Factors favoring such a defense are the insubstantiality of the copying (a small portion) and the "transformativeness" of your creation.