There appears to be no general federal statute addressing the legality of a male entering a female-labeled bathroom, or vice versa, within federal jurisdiction. There are various regulations that touch on bathrooms, for example the OSHA regulations pertaining to sanitation require that "toilet facilities, in toilet rooms separate for each sex, shall be provided in all places of employment in accordance with table J-1 of this section", but "Where toilet facilities will not be used by women, urinals may be provided instead of water closets". These are regulations imposed on employers, and there is no provision for enforcement by employers, for instance no clause saying that the employer must call the police, or in any way intervene or sanction an employee, if a person enters a bathroom of "the wrong sex".
It is apparently not a crime, as reflected anywhere in the US Code, for a male to enter or remain in a female bathroom, or vice versa. Building managers have some discretion regarding the operation of federal facilities, for example courthouses, which could extend to requesting a male to not enter a female bathroom or vice versa, but there isn't a practical way to determine what departmental regulations exist that would allow eviction of a bathroom-cross-user. Federal law does not protect against sex/gender discrimination in public accommodations.
As for Maryland law, the owner of private property has the right to control his property up to the point that state law take that control from him, and therefore in principle he could call the police to remove a person trespassing in a bathroom (this article addresses the trespass angle in North Carolina). Maryland does not appear to have any case law or statutory exception to trespass laws related to labeling of bathrooms, so enforcement actions would be at the discretion of the owner.
At the lower end of the political hierarchy, Baltimore has an ordinance requiring single-use bathrooms to be gender neutral, and a proposal exists to enact a state law with this effect.
In other words, there does not appear to be any direct, enforceable legal requirement regarding bathroom entry and sex. Trespass laws could be used, but are at the discretion of the property owner (who might be civilly sued for his actions, but the police don't decide the merits of a discrimination suit before evicting a trespasser).