Extortion is distinct from bribery, since you can only bribe a public official but you can extort anyone. In California, Penal Code sect. 518 defines extortion:
Extortion is the obtaining of property from another, with his consent,
or the obtaining of an official act of a public officer, induced by a
wrongful use of force or fear, or under color of official right.
The statutes do not clarify what "wrongfully" means, and the Calcrim jury instructions are also no help. I presume that this distinguishes the case where you create fear in the perpetrator and solicit payment for silence, versus the perpetrator becomes fearful on his own and offers money to "compensate" you (which is legal). So then sect. 519, states what can induce fear:
Fear, such as will constitute extortion, may be induced by a threat of
any of the following:
...
2. To accuse the individual threatened, or a relative of his or her, or a member of his or her family, of a crime.
- To expose, or to impute to him, her, or them a deformity, disgrace, or crime.
You would be threatening cause of fear number 2 (and 3). Then the question arises, were you successful? Note that extortion is defined in terms of obtaining property. If you actually obtain the property, that violates section 520:
Every person who extorts any money or other property from another,
under circumstances not amounting to robbery or carjacking, by means
of force, or any threat, such as is mentioned in Section 519, shall be
punished by imprisonment pursuant to subdivision (h) of Section 1170
for two, three or four years.
If you only make the threat, and the threat is written, then per sect. 523, the same punishment results:
Every person who, with intent to extort any money or other property
from another, sends or delivers to any person any letter or other
writing, whether subscribed or not, expressing or implying, or adapted
to imply, any threat such as is specified in Section 519, is
punishable in the same manner as if such money or property were
actually obtained by means of such threat.
But otherwise (if you do not obtain the property, and the threat is not written), per sect 524 there is a lesser penalty:
Every person who attempts, by means of any threat, such as is
specified in Section 519 of this code, to extort money or other
property from another is punishable by imprisonment in the county jail
not longer than one year or in the state prison or by fine not
exceeding ten thousand dollars ($10,000), or by both such fine and
imprisonment.
There is no law prohibiting you from telling a person that they must return the money they scammed from you, and there is no law requiring you to report an attempt to scam. The line that you are not allowed to cross is creating a fear that you will report them to the police if they do not give you some amount of their property.