It's not a crime per se, but you're breaching contract if you're accessing it by normal means, that is, through a Web browser or through the API.
Youtube Terms of Use 5B, emphasis added:
Content is provided to you AS IS. You may access Content for your information and personal use solely as intended through the provided functionality of the Service and as permitted under these Terms of Service. You shall not download any Content unless you see a “download” or similar link displayed by YouTube on the Service for that Content. You shall not copy, reproduce, make available online or electronically transmit, publish, adapt, distribute, transmit, broadcast, display, sell, license, or otherwise exploit any Content for any other purposes without the prior written consent of YouTube or the respective licensors of the Content. YouTube and its licensors reserve all rights not expressly granted in and to the Service and the Content.
It is a breach of the Terms, which you agree to and are contractually bound by, to download Content unless Youtube allows it. Because Users license, the Content to Youtube, Youtube is entitled to recover loss of income from your breach of contract, including loss of advertising revenue, and possibly even punitive damages.
Legally, whether you can download it depends on the jurisdiction. It is generally acceptable to make a copy of media you purchase, however these rights do not generally extend to media purchased online, where you become bound by the terms of the service provider - in this case, Youtube.
The owner of the Content retains ownership rights and licenses derivation, reproduction and distribution rights to Youtube. That is, if the content owner made it available on some other website for viewing, then you would have the rights to download it for the purposes of viewing it. However, you would still not have distribution, adaptation or modification rights unless they were assigned to you by the content owner.
You may have a a Fair Use/Dealing defense for the content, depending on the purpose and nature of your usage. 17 U.S.C. § 107 outlines the considerations in a US Fair Use defense, which essentially boils down to:
- Nature and purpose of the use
- Nature of the copyrighted work
- Amount of the copyrighted work used
- The effect of the use on the value of the copyrighted work
In the absence of case law to the contrary¹, the Terms of Use, copyright law, and fair use law are only legal principles we can rely upon in determining the legality of downloading content from Youtube.
1. It's possible that there is case law, but I've just spent a bit of time searching and haven't found any cases where end users were sued for downloading content.