I would like to start collecting leaked databases, but is it illegal to own them, even if they're public?
You can't own a database; you might, however, own (have) the copyright to a database if you created it or the creator transferred that right to you. You can also possess a copy of a database: the question is whether it is legal. "Leaked" implies that it is taken without permission, so you might be in violation of copyright law by possessing a copy. The only databases that would escape copyright protection would be those US government works, things put in the public domain, and things publically licensed to allow copying. Plus, any database whose content fails to exhibit a modicum of creativity (Feist).
A database might be inherently illegal (at least in your hands), so it would depend on what the content is. The first thing that comes to mind is a database from a child-porn website, which contains numerous illegal images: see section 110 of Title 18.
"Leaked" information might involve violation of 18 USC 1030 (Computer Fraud and Abuse Act), which prohibits unauthorized hacking. It does not directly prohibit being in possession of a hacked database: but you might still be prosecuted as an accessory after the fact. (That is one of those ad libitum areas of the law where there's no way to know for sure what is and is not "okay"). If they do prosecute you, you might rely on Bartnicki v. Vopper, 532 U.S. 514, where because it was a matter of "public interest", propagation of illegally obtained material was held to be protected by the First Amendment. Also there is the case of the Pentagon Papers.
At the federal level, there are no controls over storing credit card information so if you get a copy of the Target or Home Depot hacked database, there's no federal law against that (if we discount "accessory after the fact"), but there are circumstances in Minnesota where retaining such information could be illegal.