Is it illegal to ask a company for money in exchange for information on a bug in their software/website?
That in itself is legal. Indeed, the company would incur unjust enrichment if it coerced you to disclose your discovery for free.
Only if you threatened the company to divulge to others your discovery unless the company pays you, it would be illegal and trigger charges such as extortion (likewise, legislations outlaw the unjustified delivery of programs or instructions for hacking a software/network/etc., although this goes beyond your actual question).
Can the company take legal action against me?
That seems doubtful, futile, and it could backfire (please note I have not done any research on legal precedents about this).
Although the terms and conditions of the website or the End User License Agreement (EULA) of software might prohibit you to reverse engineer (RE)/decompile/etc. the application, anti-RE clauses are unenforceable and the remedies therefor are indeterminate because the sole act of conducting reverse engineering does not subject the company (or third parties) to any losses.
The company's decision to take legal action for your discovery could backfire from two standpoints. First, it calls attention to the fact that the software at issue is defective and unsafe. And second, the bug is likely to be detected by someone else anyway, thereby potentially compromising customers' systems.