Would this be legal for L to use prisoners to prove his theories about
No (unless the prison gave informed consent to this risk, perhaps in exchange for some favor for his family).
But, suppose that the next of kin of the prisoners sued the Japanese government for wrongfully endangering the decedent who was killed by Death Note. (As it happens, death by Death Note is not itself in this instance any more painful than death by a valid means of execution in Japan. Indeed, it might even involve less suffering and less dishonor for the prisoner.) The appropriate amount of damages for the wrongful death of a murderer with just hours left to live would probably be lower than the appropriate amount of damages for anyone else. They might even be merely nominal damages (e.g. 100 Yen).
It is probably also a crime (e.g. reckless endangerment of someone in the custody of the state), but the prosecutor and law enforcement are likely to look the other way and decline to enforce the criminal law in this situation.
Of course, any legal liability, civil or criminal, requires the court to determine that a reasonable person would believe that Death Note deaths were possible at all, and that this death was actually caused by the involvement of the prisoner in the scheme and was not merely a coincidence. In real life, those would be insurmountable barriers to civil and criminal liability, but this is, by assumption, not a real life situation. The cause that would be revealed by an autopsy, if I recall correctly, would be heart attack or some similar natural cause, in this particular instance.
While not strictly relevant, it is also helpful to know that in Japan (one of the relatively few developed countries that retains the death penalty), unlike most other death penalty jurisdictions, inmates on death row do not have a right to advanced notice of their execution date, and generally do not know what the date will be (sometimes days in the future, sometimes years) until a matter of hours before they are executed. This is a feature of the Japanese death penalty that human rights activists have criticized.
The story is set in Japan, but would this be illegal in the U.S.A.?
Essentially the same analysis applies, although the exact legal authorities cited in support of this conclusion would be different.