It is, in most jurisdictions, not a crime to download YouTube videos.
For criminal law, the answer is that it is not illegal. In many jurisdictions, downloading music or video of any kind from the internet is not a crime. Thus, police has no power to punish you for downloading, and even less power to shutdown such "downloader" websites.
YouTube's Terms of Service seem to disallow such downloads so YouTube has the right to terminate the agreement with the downloader. They may prevent you from viewing any more videos, for example. However, I think Google does not even have a technical measure in place to do that.
Google is not interested in preventing you from using YouTube (its servers can handle that) and that is pretty much the only punishment it can use.
Google could use the Terms of Services to say that the downloader-websites are breaking them and thus should not have access to YouTube. The websites could be sued for breaking the Terms of Services (and the court could order them to stop) and Google can block those websites from accessing YouTube by technical measures. It appears it did so in at least one case.
Google could sue you (or the websites) for advertisement revenue loss, but it is unlikely.
In theory, Google loses advertisement revenue from the video playbacks you would have done on YouTube but did not play because you downloaded the video and played it offline (contrary to the Terms of Service). Google might attempt to sue a downloader website on this basis and try to make it give it money. It would be difficult for Google to prove that it deserves such money, though.
I cannot imagine how it would prove that "you would have played the video online, again, with advertisements, if you didn't download it. Note that when you replay a video from browser cache, advertisements do not replay.
Why Google doesn't sue downloader-website more often? It's expensive.
Google could pay a lot of money to shut down a website via court order but if it really wants to do so, they can apply a technical measure (such as IP block) instead. It is much cheaper and has the same effect.
This may change in the near future.
There's a some talk around about European court rulings and directives that may change this. It is quite possible that in the near future, even viewing illegally uploaded music on YouTube will be criminal. I find this doubtful because of the difficulty of proving knowledge ("How was I supposed to know that it wasn't an official clip?").
As for whether downloading to a file (as opposed to downloading to the browser cache) will become criminal, I really doubt that.