does a reference for a candidate employee have liability for what they say about the candidate? To my understanding lying isn't illegal.
Lying is unlawful to the extent that the liar's deliberate intent to mislead other(s) causes or is likely to cause unwarranted harm. This is regardless of whether "the person acting as a reference isn't under obligation not to be a reference". Lies can directly harm the candidate and/or the company, and others indirectly.
The harm to the candidate is known as, or comes in the form of, defamation. Depending on the jurisdiction, an intent to mislead might not even be a prerrequisite for liability. For instance, Michigan statute MCL 600.2911(7) allows suits for libel or slander if "the defamatory falsehood concerns the private individual and was published negligently" (emphasis added). The liar's intent to mislead and his knowledge of the falsehood of his statements can only worsen the harm inflicted and his liability therefor.
Likewise, the liar can be liable to the company for inducing it to hire a candidate the company would not have hired had it known the truth. Liability ensues when the hired candidate makes the company incur losses which would be prevented by relying on a truthful reference.
If the reference is truthful, the chances for liability are significantly narrower. These scenarios typically involve matters of privacy or disclosures that are protected/sanctioned by law.