united-states
Can a company lie to a person in the course of a business
relationship?
It is possible for a company lie to a person in a business relationship.
The consequences for the business for doing so, if caught, vary and can be quite fact specific (different kinds of statements in different contexts can be subject to different rules and there is no one general rule) and quite jurisdiction specific (i.e. it can depend upon which state or locality this happens in).
Outside the fact pattern of the question, for example, it would be perfectly legal for a business employee in a business transaction to lie and say that you are good looking, or smart, or funny, or that they enjoy spending time working with you, or that their business was located in Mongolia when you were sitting in their office in Chicago.
I've heard rumors that, if a company finds you suspicious or costly as
a customer, their support agents might deliberately give you the
runaround until you stop trying to do business with them.
This is true.
That does not seem to be illegal in and of itself. But are there
limits on how they are allowed to do this? For example, if you try to
make an online purchase, and the company's system blocks it because
the transaction has too high a chance of being fraudulent (new
account, new payment method, not coming from your home state, etc.),
and then you call in to support to find out what went wrong, they
don't have to explain themselves.
But can they lie to you and claim that some other particular problem
was the cause of your rejected transaction? Or would that be fraud?
It is possible for them to lie to you. This often wouldn't meet the legal definition of fraud, although it might sometimes in particular sets of facts. There could also be other legal consequences for the business for a misstatement like this one.
Civil Fraud
To be actionable in a civil lawsuit for common law fraud it is not enough that the person sued makes a false statement regarding a presently existing fact, as the business in your fact pattern does.
In addition:
The false statement must be about a material fact. Whether the reason that you were denied service is material or not is a close call, when you would be denied service anyway.
The person making the statement must have intended that you believe that the statement is true. This is probably met in your fact pattern.
The person hearing the statement must have actually relied upon the fact being true. This could be hard to establish. What would the customer have done in differently had they not believed that the fact was true?
The person hearing the statement must have been justified in believing what the customer was told. This is probably met in your fact pattern.
The person hearing the statement must have suffered economic harm as a result of their reliance on the false statement. Again, this would be hard to established.
Criminal Fraud Prosecutions
Some jurisdictions also have criminal fraud statutes, enforced in the discretion of a prosecutor, which dispense with the reliance and damages elements of a civil lawsuit for fraud. Even then, however, there is a materiality element which somewhat makes up for the lack of reliance and damages elements.
A Key Fact That Isn't Available Re Materiality and Reliance
In the question's fact pattern, one of the key issues would be whether the false reason for rejecting your transaction (if there was another legally justifiable reason) prevented the customer, for example, from accessing the services from another vendor because the customer might end up thinking it was futile to do so.
Likewise, it would matter if the customer spent money to remedy the false problem you identified (e.g., paying for a software upgrade that wasn't actually the problem).
Deceptive Trade Practices
Some jurisdictions have consumer protection statutes that punish misleading conduct that would not constitute fraud at common law, or that would be harder to prove fraudulent at common law. These offenses are often called "deceptive trade practices" and vary quite a bit from jurisdiction to jurisdiction, and also vary from being quite vague to being very specific (e.g. "not representing that a hearing aid has a rating of more than 10dB than it actually does").
In California, many kinds of deceptive advertising can be enforced with class action lawsuits without the need to show a specific instance of economic harm, because statutory damages can be awarded.
Legal v. Illegal Discrimination
Often the reason that a business doesn't want to do business with someone could arguably be either a legal one (e.g. a poor credit score) or an illegal one (e.g. race). Lying about the true reason for rejecting a transaction would make the business extremely vulnerable to the inference in a discrimination trial that it gave a false reason to cover up an illegal reason, rather than a legal reason, for rejecting a transaction.