The relevant question for libel under US law is "would a reasonable person understand this to be a statement of fact about the plaintiff, or to imply a statement of fact about the plaintiff." It doesn't directly matter if the name was changed or not; what matters is if a reasonable person would think the statements in question are talking about an actual person (the plaintiff) and are stating (or implying) actual facts, or if a reasonable person would think the statements in question are pure fiction and don't say anything factual about the plaintiff.
Changing the name tends to make it seem more like fiction, but that's not always enough: suppose I write a long fictional story about Theodore Bau, who is active on the Pile Market series of online Q&A sites, particularly a history one and a board games one, who published an economics book in 2004 and was an econ and history double major, and fills in more details from Tom's SE bios, and in the book talk about how Mr. Bau stole money from clients; I then send thto potential clients of Tom Au. The fact that I changed the names and said "this is a work of fiction and any similarities are coincidental" isn't exactly an automatic get-off-scot-free card.
On the other hand, if I'm telling a story about Tom Au that uses a fair bit of your backstory, with no disclaimer that any similarities are pure coincidence but Mr. Au lives a secret life as a legitimate supervillain, a reasonable person is unlikely to conclude that I'm saying that you actually have a volcanic lair and that you are actually plotting to capture a US and a Russian missile sub to provoke a nuclear war.
In this case, the court determined that a reasonable person familiar with the context could understand the book to be talking about the plaintiff's actual behavior, instead of just talking about a fictional character. The fact that it was fiction and the names were changed suggested that it wasn't talking about the real plaintiff, but the details of the book could make it go the other way.