A credit agency or a lender may use its own formula for deriving a credit score based on various facts about an applicant's credit history. That score need not map closely with other formulae for the purpose devised and used by other score vendors or lenders. A score is, in essence, a statement that "Given X set of facts, we think the likelihood of default is Y". This is not defamatory as long as the facts have been input honestly and the formula has been applied fairly (even if it is ill-judged). One formula might weight home ownership high, and give low scores to renters. Another might rate steady income high, and give low scores to those seasonally employed or self-employed as consultants. Neither would accord with the most common practice, but a lender may use either if it so chooses, and your only recourse is to approach a different lender that uses a different model.
The score is not a fact, the score is an assessment of the significance of the various facts in the credit history. Credit bureaus generally give lenders the full history as well as a score, and lenders can and do modify their decisions based on the details of the history when they think that the score does not accurately reflect the individual history.
Nor is the lender required to tell you how the score is derived from the facts of your credit history. It is (under the US Fair Credit Reporting Act) required to indicate what facts went into the judgement (if you ask), and investigate if you claim that the stated facts are incorrect.
If someone intentionally falsified facts to your detriment, or bypassed the usual formula and simply invented an unfavorable result in your specific case only, that would be improper and you might well have grounds for action if you could prove it. But if you simply disagree with how the lender's model weights the various facts, which you admit are correct, I don't think you have any grounds to claim defamation.