In the US, the question of gender and facility-assignment depends on whose prison it is – federal, vs. a specific state. There may be a specific policy adopted by a particular prison system, but there is no specific statute that generally requires segregation of prisoners based on gender. Policies will exist which protect a prisoner's 8th amendment right to be free from cruel and unusual punishment: subjecting a person to repeated violence would be a kind of cruel and unusual punishment.
At the federal level, the Prison Rape Elimination Act was designed to eliminate rape in prisons (which was already illegal). The act mandates a zero-tolerance policy, and enables studies of the problem – it does not further distinguish gender categories and/or parameters of sexual orientation. Specific policies are highly variable between states and the federal system. In theory, a state could mandate separate prisons according to birth gender, current gender and sexual preference (or, wings within a facility). I don't know of any system that specially segregates bisexual inmates from mono-sexual ones (etc.). There federal policy on transgender prisoners is explained here.
The main policy of relevance is that prisoners are assigned based on their biological sex. Prior policy applied to Transgender individuals, defined as
the state of one’s gender identity not matching one’s biological sex.
For the purposes of this policy, a transgender inmate is one who has
met with a Bureau of Prisonspsychologist and signed the form
indicating consent to be identified within the agency as transgender.
This step allows for accommodations to be considered
That version did not say what "biological sex" is. Under current policy, the facility shall
decid[e] the facility assignment for a transgender or intersex inmate,
the TEC [Transgender Executive Council] should make the following
assessments on a case-by-case basis:
followed by, first: use biological sex as the initial determination for designation. Nothing addresses the former Intersex category, and I can find no explicit statements about what rules they use. This article makes some recommendations about this matter (suggesting making an official rule), but that's a desideratum and not the law, and it does not suggest that there is a known policy pursued by the purported TEC (whose actual existence is very hard to verify).