Can responsible disclosure be seen as an open invitation to try and
penetrate a website or service
Yes. Hacking is lawful if the person who gains access to the system(s) is authorized by its owner to do so (by owner I mean the proprietor or the systems/network administrator in charge of the system). The lack of authorization would trigger Section 138ab of the Dutch penal code. An open invitation certainly amounts to authorizing anyone to try to hack the system.
Beware that this English translation of the Dutch penal code is flawed and misleading. It portrays Section 138ab as defining the concept of "unlawful entry" purely in terms of the method of gaining access, irrespective of the existence or absence of owner's authorization. The translation inaccuracy lies in automatically attaching to "the act of penetrating" (that is, binnendringen) a connotation of "unlawful".
The key word in the legislative language of Section 138ab is wederrechtelijk, which means "without agreement by the right/title holder" (zonder toestemming van de rechthebbende, see here or here).
As for your follow-up question:
If I start to try and hack a website, but I do not find a
vulnerability to report, I did something illegal. does responsible
disclosure still apply?
An open invitation or individualized authorization is crucial for immunizing yourself against prosecution or liabilities.
Although the language of Section 138ab is strictly in terms of actually gaining access to the system, the mere attempt to hack a system might trigger other sections of the penal code. For instance, a failed attempt that has the effects of a DOS attack could "hinder the access to or use of a computerised device or system by [...] sending data", Section 138b. Similarly with the first item of Section 139c, or item 1 of Section 139d. Hence the importance that the attempt to penetrate a system be authorized beforehand: Wie dus andermans beveiliging wil testen, kan dat maar beter vooraf vragen.