Doesn't being recorded in your own home, or someone elses for that matter, require consent?
That's not relevant here. Media outlets are generally not liable for reporting information that was or may have been obtained illegally, provided they are not the ones that did the illegal act, or directly incentivized its commission (e.g. if Gawker had paid someone to get them such a video).
The person who filmed this act may or may not have committed a crime, but this was not relevant to the case at hand. In fact, Bollea (aka "Hulk Hogan") did sue those responsible for making the video; he settled with Bubba, but not with Clem. Bollea later added Gawker to the ongoing suit against Clem when publication of the tape was ruled to not be a copyright infringement.
Isn't there a reasonable expectation of privacy when you're not in public like that?
Also largely not relevant here, as Bollea's claims against Gawker were not about the production and existence of the tape, but of its publication. Bollea did allege invasion of privacy in his suit, but as concerns Gawker this was about publishing the video.
Also keep in mind that Bollea had gone out of his way to convert his private life into a public spectacle by virtue of his reality tv program. Your reasonable expectation of privacy goes down when you have already set the bar so low that most every aspect of your private life is on public display. Some of Bollea's legal motions in his failed attempts to convict Gawker of copyright infringement were denied on the basis that publication of the video tape may actually constitute fair use.
Celebrities in general have lower expectations of privacy than ordinary citizens. It's the well-known price you pay for fabulous amounts of wealth and fame: lots of people are paying attention to you and are interested in what happens to you, and you can't expect a lot of privacy when everyone's staring at you and hanging on your every word and action. Not that they have zero privacy rights, it's just harder for them to establish the "reasonable expectation".
don't I have the right to not have that material distributed even if I were famous and it was "newsworthy"?
Your question here is the embodiment of the controversy: where does freedom of the press end and personal privacy begin? Freedom of the press is constitutionally enshrined, and tends to be zealously protected in America as a prerequisite for true democracy and freedom. Personal privacy is not, and emerges more as the consequence of laws and the judiciary's application of certain Common Law sensibilities. There are some constitutional protections against governmental infringements (unreasonable search and seizure, that sort of thing), but the protections you have against non-governmental infringements only exist within laws.
So the general expectation you can have is that if something is newsworthy, then it's fair game for the press to report on it. The issue falls to what qualifies as "newsworthy", and who gets to make that judgment?
Part of Bollea's legal argument was that what Gawker published was not newsworthy, going to the extent of having the editor concede in court that a depiction of Bollea's genitalia was not newsworthy. And this "not newsworthy" angle seems to be the heart of what their case was getting at: what was reported was not done because it was "news", but because it would get Gawker attention and profits at Bollea's expense (including capitalizing on his famous Hulk Hogan persona).
It's also the heart of why some people consider this a controversial and potentially problematic case. Those who have voiced 1st amendment concerns have done so primarily out of concern that this trial would allow courts and juries to decide what was "newsworthy" and what was not. Meaning that news outlets might now have to sweat bullets every time they published something because maybe a jury would assert it wasn't newsworthy, leading those outlets to be less likely to report certain aspects of news. Presumably these people feel that the only ones who should be deciding what is newsworthy is the press, and the public vis-à-vis their consumptions and demands thereof.
