Is it considered retaliation for an organization to reject an applicant who has sued or filed a complaint against someone in the organization?
If One Files an Official Complaint Against or Sues a University, Is It Retaliation if They Reject Your Application?
Not necessarily. Retaliation has to do with the criterion that prompts an adverse decision, not the decision itself. You are asking about the event of rejection itself, which might be lawful nonetheless.
You mention that you are a good candidate academically. That might not suffice, though, since it is lawful for universities and many others kinds of entities to ponder some criteria in addition to applicants' purely academic profile. Such criteria might include those which realistically subject others to safety risks.
It is unclear from your post whether that university is a public one and the legislation puts it in the category of "arm of the state". If so, the university will certainly resort to the doctrine of sovereign immunity and have your case dismissed without even allowing you to pursue discovery. One effect of that doctrine is that a public university can blatantly indulge in many types of torts and wrongful conduct, and yet the court would not rule against it even if the evidence fully supports your claim(s).
Even if you overcome the issue of sovereign immunity or the doctrine is not applicable, the ensuing discovery proceedings might feel invasive to you. That is because procedural law would entitle the opposing party to conduct discovery on the matter of your mental health (my understanding is that the episode you mention refers to your mental health, but that is not entirely clear from your post). Pursuing this line of discovery is quite predictable insofar as the prior grievance and the alleged retaliation are premised on the acute mental health episode you mention.
Consequently, the university could prevail on grounds that your admission therein would pose a risk to faculty, students, and/or staff to the extent that the university cannot ensure the preemption of further episodes [involving you] that might hurt others in the university.