Summary: A transcription of a "faithful" to-the-letter recorded performance of a public domain work is not copyright infringement. However, once there exists original changes to the underlying composition, the transcribers are infringing copyright in all but the most trivial cases and are in for an uphill legal battle.
Caveat 1: I'm mainly confining myself to US law due to the statement that this concerns a US-based company. However, there might be legal troubles in some jurisdictions for both the company and transcribers where a perpetual and inheritable moral right of first publication is recognized, like Germany (UrhG § 12).
Caveat 2: Outside of copyright law, the company could theoretically gain some limited protection by selling the recording under a contract with a "no transcribing" clause. Since a similar scenario has already been discussed on this site, I am confining myself to the copyright law aspects.
1. What did the transcribers copy?
In effect, I believe the transcribers are ultimately copying an underlying arrangement fixed by the recorded performance. That's a bit dense, so let me explain:
As you already note, a performance is an artistic rendition of the underlying composition. This performance has separate copyright and may even include various changes to the underlying composition such as dynamic changes, instrumentation changes, tempo changes and various flourishes. I believe that a transcription would mainly reflect the underlying composition plus any such artistic changes to that composition made in the performance. Effectively, from a music theory perspective an underlying composition with artistic changes is the definition of an arrangement. From a legal perspective, an arrangement is copyrightable as a derivative work. Even though it's the performance itself that's being copied from and you haven't specified that there's a separate written arrangement, one can consider an implicit arrangement created by the performance. This implicit arrangement is then documented (i.e. fixed) by the recorded performance.
Since a transcription is not capturing the sound itself, but rather the compositional elements of that performance, i.e. the arrangement, I think that it's a useful perspective to consider that the transcribers are copying an arrangement. Even if this way of viewing things turns out not to be 100% accurate, I think it's accurate enough and provides a clearer picture into the underlying legal concepts. I do however admit that some may take issue with this simplification, and it's entirely possible that my answer is completely invalid if this simplification turns out to have no merit.
2. Extent of possible copyright infringement
The company which commissioned the orchestra and the recording undoubtedly has copyright on the performance. However, they do not have copyright on the underlying original composition, as that copyright will have long expired in US law. The company can't claim copyright on material which originates in the underlying public domain composition, but can claim copyright for any original contributions made by the arrangement (17 USC 103(b)).
As such, I would expect an infringement lawsuit launched by the company will be limited to the portions of the arrangement that differ from the original composition and that are then copied into the transcription. In order to properly determine what those portions are, we would need to compare the transcription, performance, and manuscript together. Since it's the company that is initiating a lawsuit, I would expect that the judge would force them to disclose the manuscript for the proceedings (and instruct the parties that the information is not to leave the courtroom, which is ironic since an NDA is how this whole mess started). This is an exercise that will likely get bogged down in excruciating note-by-note comparisons, but luckily those are factual questions, and I'm just going to discuss the higher-level legal concepts.
You also mention there might be "mistakes" included. In theory, copyright protection only applies to creative choices made, I don't believe that mistakes would count. In practice though, this may be difficult to prove and I think the company would argue it was an artistic choice, thus rolling it into the overall changes which constitute the arrangement. However, they might alternatively argue the mistakes are a purposeful copyright trap which can have mixed results.
3. Copyrightability of the arrangement
In my opinion, the obvious argument to start with is that the company's arrangement does not qualify for copyright protection. This is a bit of a spectrum. While a full-fledged arrangement is undoubtedly protected as a derivative work, here we are dealing with just portions of an arrangement. Based on the premise of the performance being an accurate representation of a public domain song, I would expect these portions to be minimal. In fact, in the ideal case of the performance being a perfect faithful rendition of the composition, there is no separate underlying arrangement and therefore no copyright to infringe. This would be similar to the case of a photo of public domain art, which gains no copyright due to lack of originality.
Copyright requires a "modicum of creativity" as determined in Feist Publications, Inc., v. Rural Telephone Service Co.. Feist also shows the reason copyright traps don't work for defending an originality argument: while you can use traps to prove that copying occurred, they provide no indication as to the actual originality/creativity of the work. The transcribers can thus try to argue that the given portions of the arrangement do not meet this standard for copyright protection.
As applied to music, one can consider that the basic building blocks of musical composition are not copyrightable, though large enough combinations of these building blocks do qualify. An examination of this can be found in Marcus Gray v. Katy Perry, where a specific 8-note sequence was found not to be eligible for copyright protection (though do note the case is currently being appealed). The court additionally stated that even if the sequence was copyrightable, such a small sequence would only be found to be infringing if the copying was virtually identical. Since the transcribers are presumably attempting an accurate transcription, this latter point is important to us as it shows that it would not take much copying for copyright infringement to occur. On the other hand, I would expect the differences in the arrangement and original composition might be scattered around and perhaps not be a cohesive artistic unit, which would benefit the transcribers' argument. For further information, LegalEagle has a 25-minute YouTube video discussing this case and the involved legal principles.
As the arrangement is a derivative work, the transcribers might be able to advance a separate "triviality" argument. The Seventh Circuit Court of Appeals (also citing the Second Circuit cases) states in Schrock v. Learning Curve International, Inc.
[T]he key inquiry is whether there is sufficient nontrivial expressive variation in the derivative work to make it distinguishable from the underlying work in some meaningful way.
However, this emphasis the court makes on having the derivative be nontrivially distinguishable from the original doesn't arise from statute, but from a practical concern of determining whether the derivative or original was copied from given a hypothetical subsequent derivative. Assuming the company can prove the original hasn't been "leaked" and is unpublished, I would expect this argument to be discarded. Additionally, here is where copyright traps might actually help the company.
Interesting aside: the German law on derivatives diverges here based on an oddly specific clause for public domain music (UrhG § 3):
Translations and other adaptations of a work which are the adapter’s own intellectual creations are protected as independent works without prejudice to the copyright in the adapted work. The insubstantial adaptation of an unprotected musical work is not protected as an independent work.
4. The asserted truths doctrine
This is a bit of a novel argument, since the Ninth Circuit Court of Appeals only just articulated an "asserted truths" doctrine on 2020-09-08 in Corbello v. Valli:
Adopting an “asserted truths” doctrine, the panel held that an author who holds their work out as nonfiction cannot later claim, in litigation, that aspects of the work were actually made up and thus entitled to full copyright protection.
Basically, an author had claimed that the work was a factual biography, but then in the lawsuit claimed it was fiction in order to gain additional copyright protection since facts are not entitled to copyright protection. The court denied copyright protection for such an action.
If the company is asserting that the performance is a 100% faithful rendition of the public domain work, the transcribers might be able to adapt this argument. They might argue that they relied on the asserted truth that they were copying public domain material. The company might counter that since this is a performance, one would expect that there will necessarily be at least some small differences and so the doctrine does not apply. Here, the transcribers may be able to counter that based on the company's representation, they had expected that any such small differences would not be under copyright due to not meeting the "modicum of creativity" standard. I think this argument is a stretch though.
5. Making the transcription public domain & final points
Assuming the arguments above worked for the transcribers, they don't really need to specifically designate the transcription public domain – it already is by its nature of not being copyrightable by those very same arguments. The one exception though would be that they hold copyright in the layout of their transcription so would have to designate it public domain for that purpose.
If the arguments above do not work for the transcribers and the arrangement is found to have copyright protection, I can't think of any other viable arguments for the transcribers. They might argue fair use, but given their intent to disseminate the transcription, I would expect that argument to fail quickly based on both the nature/quantity of the copying and the damage caused to the company's revenue. On the minimal chance that fair use or some other argument works, the transcribers still could not put the transcription in the public domain. They could say that, but it would be of no real effect, since in this scenario it's the company that holds copyright over the arrangement represented by the transcription.
In all, I believe that if the performance is a rigorously faithful rendition with only the odd mistake or flourish here and there, then the transcription is not copyright infringement. Otherwise, it is. Wow, that took me a while to get to this relatively simple conclusion.
