is there a general answer to whether it contravenes DMCA or copyright protection to have a piece of my software output to another company's proprietary file format?
I can't see why it would be illegal. For example, Microsoft was even forced to publish file formats, which would be rather pointless if you couldn't output to that file format.
There was the situation where compressing images to the .gif format had legal problems because the compression algorithm was patented, but someone found a way to output images to the .gif file format without doing any compression, which turned out to be absolutely legal and not covered by that patent.
When you output to a file format, you are not copying any information of the company. In the extreme case that the file format requires you to copy such information (for example if the file format requires that the first line of the file is "This file format may only be used by ABC company. Any use by any other company is copyright infringement. ", and software refuses to read files without that line of text, then this line is required to be there and therefore doesn't fall under copyright. (In the early years of PCs, IBM computers had the letters "IBM" somewhere in the BIOS, and some software checked for these letters and refused to run if they were not there. Result: Copying these three letters into your own BIOS was legal. )
The DMCA is about circumventing copy protection to copy someone else's copyrighted works. (There is another part having to do with protecting ISPs and websites from getting sued; has nothing to do with this at all). You are not circumventing copy protection, and you are not copying someone else's copyrighted works, so the DMCA doesn't come into this.