Usually yes, possibly sometimes no.
There is an area of the law where Congress does this exact thing.
If a court wants to transfer certain kinds of federally regulated retirements assets titled in the name of one spouse to another spouse in the course of a divorce, this is only effective if the Court follows the exacting requirements of a "qualified domestic relations order" (similar requirements apply to both federal government employee benefits and to private pension plans governed by ERISA which is a federal law with broad pre-emptive effect over private pension law).
But, this is not considered an improper separation of powers or federalism violation, because courts are not required to issue qualified domestic relations orders at all. Issuing a qualified domestic relations order simply gives a divorce court jurisdiction to divide an asset that it would otherwise not have jurisdiction to divide.
Similarly, Congress can set forth by statute certain facts that must be made by a court in order for it to have the authority to invoke that statute, and under rules of court procedure in existence as a background default set of rules, Courts generally require that findings of fact establishing these facts be present in a relevant court order by a judge.
Some other examples of this kind of legislation involve federal welfare regulations which require that state child support awards be supported by a calculation of the child support guidelines that states must adopt in order for their citizens to receive federal welfare benefits.
Similarly, in criminal cases in federal court, courts are statutorily required to do a calculation of a U.S. Sentencing Guidelines recommended sentence prior to imposing a sentence upon a criminal defendant who is convicted of a federal crime.
As another example, federal law requires the judicial branch to keep and publish statistics regarding a variety of matters related to its business in an annual or periodic report, but does not specify every fine detail of how this must be done or what must be included in these reports, and this makes it necessary, in practice, for federal courts to require "cover sheets" classifying all new cases filed in federal court of all litigants commencing new cases.
At the state level, where the same separation of powers reasoning applies, many state legislatures require courts in temporary protective order cases to enter all orders on a statutorily mandated state form that facilitates entry of those orders into a state database that is available to law enforcement. It is unlikely that a similar mandate at the federal level would be found unconstitutional if it has a rational and reasonable justification such as that one.
Comity favors legislation in which the legislative branch does not micromanage the affairs of the judicial branch more than necessary, but this is muddied by the fact that the job of the judicial branch is to accurately implement the laws enacted by the legislative branch, so there aren't clear cut limitations on what Congress can or cannot mandate.
It isn't unthinkable that Congressional mandates on how the judiciary conducts its business, particularly if it impacts the substance of judicial decisions on constitutional issues, could be held unconstitutional on separation of powers grounds. But, for the most part, Congress doesn't push too hard in this area and the judiciary accepts the requirements that Congress does impose as valid.
While there are logical arguments why Congress should not be able to mandate the details of how courts write their orders, for the most part, in the law, experience and examples are stronger argument than those based upon mere logical reasoning from fundamental principles. Law is not physics.