Frivolity is not really a factor in determining who pays court costs. Under Rule 54, the loser almost always pays, regardless of frivolity.
Sanctions typically come into play for violations of Rule 11, which requires the person filing any document with the court to certify that the document:
(1) it is not being presented for any improper purpose, such as to harass, cause unnecessary delay, or needlessly increase the cost of litigation;
(2) the claims, defenses, and other legal contentions are warranted by existing law or by a nonfrivolous argument for extending, modifying, or reversing existing law or for establishing new law;
(3) the factual contentions have evidentiary support or, if specifically so identified, will likely have evidentiary support after a reasonable opportunity for further investigation or discovery; and
(4) the denials of factual contentions are warranted on the evidence or, if specifically so identified, are reasonably based on belief or a lack of information.
Although subsection (2) is the only one that explicitly addresses frivolousness, all four types of conduct can safely be thought of as falling under the umbrella of frivolity.
If the court determines that the filing was made for any of these improper purposes, it may then move on to decide what sanction is appropriate. Rule 11 gives the court quite a bit of latitude, permitting it to impose any "appropriate sanction." In the vast majority of cases, this will involve an order requiring the violating party to pay for any attorney's fees associated with opposing the frivolous action.
The main rule governing these awards is that the sanction must be designed to compensate the innocent party, not to punish the offending party. For instance, in Goodyear Tire & Rubber Co. v. Haeger, 137 S.Ct. 1178 (2017), Goodyear deliberately concealed data that would have wrecked its defense in a case where its tires exploded and injured the plaintiffs. The court only learned that the defendants had defrauded the court after the case had resolved, and it ordered Goodyear to pay plaintiffs' attorney's fees for everything that had happened since Goodyear first lied about its data. But the Supreme Court threw out that award, saying that sanctions must be "limited to the fees the innocent party incurred solely because of the misconduct."
So if the defendant's entire defense was frivolous from the beginning, he may be required to pay for the entirety of the plaintiff's attorney's fees. But if the defendant merely files a frivolous motion to exclude evidence, he would only have to pay the fees associated with defeating that motion. It would not be uncommon for the court to further order the defendant to pay for costs associated with moving for and securing the sanctions.