You can do it using a US non-profit, but you need a lawyer
We know it can be done in the US using a non-profit because it has been done. For example, the Clay Institute’s Millennium Prize does exactly what you propose: it pays people who solve unsolved math problems. Similarly, the the Everglades Foundation used a prize to get people to come up with new ways to get phosphates out of the water sources.
To do what they did, you need to follow their lead, and set up and run a non-profit (aka, 501(c)(3), named after the relevant provision of the tax code).
Because non-profits are tax exempt under US tax law, you have to apply to the IRS to become a non-profit. Speaking broadly, the IRS imposes two requirements on non-profits: 1) It must serve a charitable purpose; 2) It must spend a certain percentage of its money every year doing charitable work.
To set up a foundation you have to specify the charitable purpose you will serve (“encourage research in mathematics”), and have a plan to do so (“we will award a cash prize to people who solve important unsolved problems”). If the IRS decides your plan is genuine, then they give you a pass on paying taxes.
To make sure you are carrying out your charitable plan, the IRS requires you to spend a certain percentage of your money each year on charitable work. (This is called the "distribution requirement.") Meeting the distribution requirement may be a problem, since you may not award prizes every year.
You may be able to get a general idea about how the IRS deals with such situations by searching on line. At some point, however, you will probably need to talk to a lawyer with experience setting up non-profits. (FWIW, it looks to me as if the Clay Institute meets its yearly spending requirement by supporting lots of other activities that count towards the spending requirement.)
One other problem that it seems you won’t have involves giving prizes directly to individuals. For obvious reasons, the IRS is generally suspicious of prizes given to individuals, so they impose restrictions. For example, for the winner to avoid taxes, the prizes have to be for previous work.
Where to go for more information:
USA.gov has a nice overview of the application process here. The IRS has several publications and websites that you might find helpful: here, here, here, and here. These cover everything from the general rules to nuts-and-bolts details about which forms to use.
You can read more about the IRS treatment of prizes here and here.
Depending on how comfortable you are with financial statements, you may learn about how these non-profits work by reading their 990 forms. The Clay Institute's forms are here.