Brother X isn't an executor until the surrogate's court appoints him to that post. Until then, he is just someone nominated in a Will to be an executor. The only power that a nominee might have is to stabilize estate assets pending an appointment, and not necessarily even that.
Brother Y, as second in line person to be executor, can apply to open a probate estate and be appointed as executor in a petition filed in the surrogates court for the county where the decedent was domiciled at the time of death. This will be granted as a matter of course, without an in person hearing if Brother X agrees in writing to waive the right to serve as executor.
If Brother X does not agree to waive the right to serve as executor in writing, then what happens is that Brother Y, upon filing a probate petition gives notice to Brother X of the hearing in the surrogate's court in which it is to decide if he will be appointed.
If Brother X doesn't make a timely objection to Brother Y's appointment as executor after Brother Y files a probate petition in surrogate's court and gives a timely notice, then the court will appoint Brother Y as Executor at the time appointed for the hearing (which may be vacated).
If Brother X objects in a timely fashion that he has priority over Brother Y, then he can say so, in which case, the estate will be opened and Brother X will be appointed as Executor unless the surrogate's court finds that Brother X is unfit to serve for some reason raised of its own accord or by an interested party in the probate case.
If the surrogate's court concludes that both Brother X and Brother Y are unfit to serve (e.g. they are both serving prison sentences for fraud at the time), the surrogate's court will instead appoint someone else to the position.
If there is no nominee in the will or relative or beneficiary available who the court finds to be fit to serve, the court will appoint a "special administrator" or "public administrator" to serve as executor (i.e. a third-party who is available to the court to be appointed in such cases).