I will use Texas as an example, but I'd be interested in any others...
Per this link, a couple is considered married under common law in Texas if they:
1.) Agree to be married, 2.) Live together as a couple, and 3.) Tell others ("hold themselves out") that they are married
It also states that in order to get out of such a marriage:
"Common law marriages in Texas have the same legal status as a ceremonial marriage." And "if the common law marriage doesn’t work out, you’ll have to get a formal divorce to end it.
However, to get a divorce:
... the couple must first prove to the court that they were married. The person that first files papers with the court has to prove that there was a common law marriage.
Recognizing that if there are children involved, or if one partner contests the divorce, things get messy and need to be sorted out legally. However, what if there are no children, and the couple mutually decide to not follow through with formal wedding plans, and instead agree to simply part ways and pretend that nothing happened?
There is no paper trail. No proof that anything ever happened except perhaps jointly signed leases showing that at one point they lived together. No reason for either to assert a claim over the other for reasons of alimony, custody, or asset disputes. Neither has any interest or desire in attempting to prove to the state that they were married under common law. Really their situation is no different from any other co-habitating and/or engaged couple who later split up before completing an intended marriage ceremony.
To title question then, would the state have any interest or legal right to assert that having once met conditions that would allow them to be considered married, that they are in fact still married despite their objections to that legal status and to being characterized as such? Could both later legally claim to be single and never married?
Bonus points for addressing how "just kidding" or "I was lying" could be used to counter the issue of "holding out". Also bonus points for addressing how taking the 5th might assessed in answering questions around status, particularly in the context of applying for a marriage license later to someone else.