The United States as a nation and 49 of its states are common law jurisdictions (Louisiana is a civil law jurisdiction).
In common law jurisdictions the law is made up of: enacted statute law - constitutions, legislation and delegated legislation and unenacted case law - judgments of judges in cases. These are intertwined and indivisible - neither makes sense without the other.
This contracts with civil law systems where statute law is the primary source of law. That said, no jurisdiction is "pure" common law or civil law as there has always been cross-fertilization of ideas and this has only accelerated over the last 100 years.
Where does the court get this power to make case law?
Whether judges make new law or discover laws that already exist is a philisophical rather than a legal question. Their power to do so is definitely a legal issue - courts are a legal creation. Either they are created by statute or the came into existence through the operation of traditional practice so long ago that their origin is more of interest to historians than lawyers.
I'm not going to go through the details of every court and tribunal in each jurisdiction of the United States so I'll just look at where the powers of the US Supreme Court come from. They are set out in Article Three of the Constitution:
The judicial Power of the United States, shall be vested in one supreme Court, and in such inferior Courts as the Congress may from time to time ordain and establish. ...
The judicial Power shall extend to all Cases, in Law and Equity, arising under this Constitution, the Laws of the United States, and Treaties made, or which shall be made, under their Authority;—to all Cases affecting Ambassadors, other public Ministers and Consuls;—to all Cases of admiralty and maritime Jurisdiction;—to Controversies to which the United States shall be a Party;—to Controversies between two or more States;—between a State and Citizens of another State;—between Citizens of different States;—between Citizens of the same State claiming Lands under Grants of different States, and between a State, or the Citizens thereof, and foreign States, Citizens or Subjects.
You can now see how the enacted and unenacted law are intertwined because "judicial Power" isn't defined in the Constitution - its definition comes from the common law as interpreted by the "supreme [sic] Court". So the question of if a particular decision falls within the decision making powers of the Supreme Court is decided by ... the Supreme Court. For example, the Supreme Court has decided that it does not have jurisdiction to decide disputes over the way Congress conducts the business of Congress - that is not something that falls within "judicial Power".
Similarly "Law" and "Equity" are not defined but they were (and are) long-standing creatures of English common law which the Constitution adopted. It must be remembered that the United States did not spring into existence ex nihilo and that the various states (as they were when the constitution was adopted) had functioning common law legal systems.
Does case law have limitations?
Yes. Case law is only binding on lower level courts in the same hierarchy - it can be overturned by the same level or higher courts and ignored by courts in different hierarchies. It can be overturned by statute law - providing that statute is legal. It fills "holes" in the law so it really only happens when new situations come up - which is not that limiting because society is always changing and old laws become obsolete as a result.
Can case law create procedural rules, civil and/or criminal?
Can procedural rules be used to limit a litigant's constitutional guarantees?
I don't know what you mean. If there are procedural rules (by statute or case law) that serve to limit a constitutional right then the court can be asked to consider if those rules are themselves constitutional. Constitutional rights are not 'unfettered' and reasonable limitations on them in law or practice are allowed. For example, your right to free expression does not allow you to play the trombone at 2 in the morning in an apartment block.