Reservations aren't of or for a single tribe (meaning roughly "ethnic group"), they are for a particular Nation (whether or not that term is used in a specific case -- it could be "Confederation" or "Band"). In Washington state, the Yakama Nation is a confederation of people who are Klikitat, Palouse, Walla Walla, Wanapum, Wenatchee, Wishram, and Yakama; the Tulalip Tribes covers the northern Lushootseed tribes made up of the Duwamish, Snohomish, Snoqualmie, Skagit, Suiattle, Samish, and Stillaguamish. Only 2/3 of the recognized tribes have reservations. The key legal concept is not exactly being "of a certain tribe", it is "being enrolled". Ancestry relates to enrollment (and disenrollment) via individual rules about membership.
Federal Indian law is encoded in Title 25. Let's just say that a question about what that title says is way to broad to ever answer. Generally speaking, there are two types of reservation land, trust land and fee land. The US government holds trust land in trust, which has various restrictions on transfer, and is exempt from taxation: transfers require higher approval (tribe, BIA). Fee land is land that the individual(s) own outright. In the case of fee land, anyone can purchase that land. The land that would be of interest for this question is trust land. The Yakama reservation, for instance, has open and closed areas, the latter being mostly trust land with use restricted to enrolled members.
On fee land located within a reservation, that does not mean that one is completely free of tribal rules. After all, off the reservation, you are still subject to the local government's land use rules (zoning restrictions, for example). The ruling of Brendale v. Confederated Tribes, 492 U.S. 408 is very complex and says "yes and no". Tribes do not have authority to zone fee land on the reservation owned by non-members, but they do have authority to generally zone fee land on closed areas of the reservation. (Very roughly speaking, fee land is subject to general government-planning type restrictions on land use that we are all subject to; but there are limits on how discriminatory those rules can be. Above all, fee land is in fact owned by the individual, and a tribe cannot claw the land back, though they can buy it back).
A Nation can set rules for membership which would be patently illegal in the general context of US law. In Santa Clara Pueblo v. Martinez, 436 U.S. 49, the situation was that the rule of the Santa Clara Pueblo nation said that the children of a Pueblo woman who married outside the tribe (e.g. to a Navaho) may be denied tribal membership, but did not exclude children of a Pueblo man who married outside the tribe. The consequence is that the children of a woman lose the right to reside on that land after the mother dies (also can't vote, etc.) The short version is that SCOTUS held that this is legal. Indian nations enjoy sovereign immunity, so you can't sue them.