For example, if I were to have a scorned friend who is seeking my demise and he/she calls my employer and calls me a rapist or accuses me of sexual harassment, would it be effective to sue? Even if these claims are false, even if my employer does not believe what is going on, let's assume I get fired because the firm does not want to take a risk either way. Additionally, let's say I know that my friend did this but I have no proof (they reached out to my employer pretending to be someone else, or they used a burner phone) and subpoenaing their ISP wouldn't do much good, do I have much of a case in suing them? Does circumstantial evidence count (i.e.: they have engaged in similar conduct with another person in the past)? Also, what kind of damages would I be looking for? Would this be a claim in small claims court? Also, suppose my salary is $100k
If you sue a person for a tort X, one of the things you have to prove is that the defendant did do X. A baseless belief that it must have been so-and-so will do you no good. You do not have to have iron-clad evidence of your allegations, for a civil suit, but you have to show with a preponderance of evidence that the claim is true. A combination of "hates me" and "provably did this a number of times in the past" could well suffice.
As for damages, it depends on what harm you actually suffered. If you get fired and you show that it was because of a false allegation, you would probably have to take this to the big court, since small claims court handles amounts in the $5,000 range (jurisdiction-specific).
It should be noted in this scenario, this would count as Defamation per se, which are defamation that are intrinsically harmful and do not require you to prove damages if the defamation is proven.
In the course of Discovery, you do not need to prove that the evidence exists, just that it is likely to exist, so you may be able to get the former employer's phone records to see when the call was placed. Naturally, if evidence on the ISP exists, that can be sought for discovery.
You could also get evidence through evidence of past history (especially helpful is that she did this before. Look for case info related to that).
As others correctly point out, false accusations of rape are defamatory per se. However, I should add that an award of substantial damages (which are much greater than nominal damages of $1) requires proof that the defendant made the defamatory falsehoods with the mental state of actual malice. Negligence is sufficient only when the defamation is not privileged and causes special damages. Actual malice is consistently defined as "defamer's knowledge that the statements are false or defamer's reckless disregard of the truth or falsity of the defamatory statements".
even if my employer does not believe what is going on, let's assume I get fired because the firm does not want to take a risk either way
That meets the defamatory threshold to the extent that false statements deter(-red) third parties from associating with you. Nothing requires the third party (here the employer) to admit that he believed the statements at issue. What counts is the evidenced effects of the defamatory publications.
let's say I know that my friend did this but I have no proof
Then how do you know that your "friend" did this? If you don't, the safest approach would be to sue "DOE defendants". Henceforth, your goal will be to prove that your "friend" is the defamatory publisher. You can always conduct discovery on non-parties (including your "friend") and, upon obtaining proof of authorship, amend the complaint to name your "friend" as defendant.
Does circumstantial evidence count (i.e.: they have engaged in similar conduct with another person in the past)?
Circumstantial evidence is admissible, although not in the way you outline. What you describe in parenthesis is known as prior act evidence, which isn't exactly circumstantial (or not always is circumstantial ), and is inadmissible as to "the fact that he did it back then proves that he did it this time as well".
In defamation contexts, circumstantial evidence is most often used for proving the defamer's state of mind at the time he/she made the defamatory falsehoods. Circumstantial evidence may also encompass the defamer's acts subsequent to his/her defamatory statements. For instance, a defamer's subsequent refusals to retract his/her disproved falsehoods constitute clear proof of actual malice. See Dalbec v. Gentleman's Companion, Inc., 828 F.2d 921, 927 (1987): "Malice may be proved inferentially because it is a matter of the defendant's subjective mental state, revolves around facts usually within the defendant's knowledge and control, and rarely is admitted".
Prior act evidence is admissible for proving method, system or intent for doing a thing (see Michigan Rule of Evidence 404(b)(1) or its federal counterpart). Thus, prior act evidence may inferentially prove a defamer's state of mind (such as his/her knowingly falsity when making the statements).
For instance, in 2015 I sued my father's second wife for defamation and harassment/tortious interference with business relation (the briefs in Michigan appellate and supreme courts are here). Defense counsel alleged that her client's mental illness and psychiatric commitment at the time of defaming me denies that she acted with actual malice. I proved that her mental illness is irrelevant, given her prior history of making false accusations of crimes. For example, I submitted copy of a police report where she falsely accused my father of domestic violence in the midst of divorce proceedings (which my father bafflingly dismissed). The police report serves as prior act evidence of this defendant's state of actual malice.