Imagine that a software user typed an original novel into a word processor (e.g Microsoft Word or Google Docs). Imaging that the software vendor added something to the terms of service (ToS) saying that anything created by Microsoft Word or Google Sheets became the property of Microsoft/Google. Could Microsoft/Google then demand to be paid a cut of each novel sold by the author/software user? Would this be enforceable in front of the courts?
Under your proposal, the author cannot effectively use the software at all, much less sell his creative efforts. The EULA clearly states that copyright is automatically transferred to the vendor when a document is created. The author gets started, writes "It was a dark and stormy night" then has a moment of writer's block, saves and closes, takes a walk, then tries again. The author has transferred copyright in the work to vendor, and needs vendor's permission to make revisions. So you have to tweak the EULA to grant automatic permission to author to create derivative works (revisions), and to distribute. A more effective strategy would be to impose a revenue-sharing formula on any commercially-exploited work created using the software, rather than attempt to directly seize the intellectual property.
The CC "NC" license attribute is a distant version of what you are looking for - it prohibit any money-making use of the software. What you would add to that is a condition that the software can be used commercially, provided a certain compensation be rendered. You would of course have to be very clear in advance about what was actually agreed to.
Such a clause would be unconciounable
Unless such a clause is negotiated for specifically in advance for any particular contract, it would be an unenforceable clause. that means, nobody expects that clause, and without indicating such a clause is in the software before, it can be declared void by courts.
In some jurisdictions, it could even be illegal: each of the copyright transfers to the software seller has no consideration given to the author, which might require full compensation for any work taken by the seller.