For example, in this experiment, in trial 5 the subject (the older brother) actually says "stop", yet the experimenter (the younger one) still keeps doing it. Does that mean the experimenter doesn't respect the subject's right of autonomy? Can the subject sue the experimenter for harassment? Or is there a case when against someone's will still be ok? Is there any resource that I can read more on this?
The experimenter actually respects the subjects right of autonomy, and because this is just pretend, there is no imaginable legal action. There is a (maybe not so) hypothetical scenario where A gives an electric shock to B. Variant 1 is that A sneaks up on B and shocks him: that is assault and is legally actionable. Variant 2 is that B consents to the shock. Because of consent, B cannot sue A unless the shock was misrepresented (purportedly 2 volts and 1 milliamp, actually 120 volts at 10 amp). Variant 3 involves strapping B down (with permission). B can at any time withdraw consent. If B withdraws consent and A continues to shock him, this is probably assault, especially if B is restrained. If B lamely says "cuttitout" but makes no effort to leave (being unrestrained) the jury may find that the appearance of withdrawing consent was not sincere, since he could have just gotten off the electric chair, unless the last shock paralyzed him. This is a fact-intensive analysis. In the video, the actions of the "subject" are not credible cases of withdrawing consent, it is just Youtube theater.
Here, A and B are just two kids horsing around, in the US. Things get more complicated when you do things under the auspices of an organization required to do a review of such experiments (e.g. the high school or university). The theory is that the institution will have explicit standards and will review the experiment, and somewhere in there there is a part where the subject is told that they can leave at any time and if they have problems, they can contact X (the IRB for the institution). There is generally no legal fallout for the experimenter, but there can be institutional fallout (getting fired / banned). Actually physically restraining a subject is, however, legally actionable: but saying "I really need you to continue the experiment" is not legally actionable, though may well result in IRB sanctions. Suppose the university approves an experiment with restrained subjects and increasing shocks. There is a point at which they cannot approve the procedure, and if they do, they are potentially subject to legal consequences for the institution. Investigating nuances of IRB actions would be better carried out on Academia SE ("has anyone ever had funding suspended for sloppy IRB practices").
This is not a medical experiment: if it were, other laws would be relevant (hence you cannot engage in unauthorized, unreviewed medical experiments on your brother).