Briefly: I am wondering whether information gleaned about a proprietary binary format by perturbing the data using an unlicensed copy of the original software (no longer distributed by the copyright holder) could be legally distributed or used in new software, without express permission from the original publisher. This is assuming the data is not particularly obfuscated and the EULA doesn't expressly prohibit reverse engineering (though I don't know if the EULA would make a difference given the software is already not legally obtained). I am mostly interested in factors other than the EULA which would affect how I can use the results of reverse engineering, since the effect of the EULA has been covered in other answers, and because there would likely be no EULA in effect in this question's case. I would just assume for the purposes of this question that the EULA does not contain any terms that would prevent a licensed user from reverse engineering the software.
These are the main issues I am interested in:
It seems to me that the only immediate infringement is in unauthorized acquisition and use of the software, when doing the reversing. Is this true? This would distinguish my question from ones such as Reverse-Engineering an Application without EULA where unauthorized reproduction/derivation of the software is involved.
Would I own copyright on any artifacts (documentation, interop code) resulting from the reverse engineering, despite the infringement involved in producing it?
Such artifacts would not be direct output from the software itself. Does this make it any different from cases (such as unlicensed photoshop) where the purpose of the unauthorized use is to use the output of the program?
What could I be liable for in case I distributed the artifacts commercially? What if I make them available to other people without any personal commercial purpose, but allow commercial use (such as in an free software / open source license)?
I am aware that it's possible for software patents to apply, but a patent violation would not be affected by how I produced any code.
EDIT: Forgot to mention, I am in the US (Illinois), and the original publisher is based in Japan.
I do have a specific case in mind, so more details follow in case they make a difference.
Without going into too much detail, the software is written for Windows and consists of an editor for creating interactive programs, which can package the data with an interpreter for distribution to end users. Due to its age, the program has compatibility issues with newer versions of Windows.
The software was discontinued by the original publisher. It is available from secondhand sellers (in the original packaging), but obtaining the newest version requires a patcher executable that used to be distributed on the publisher's website. The updater has apparently been archived in the Wayback Machine, but my gut tells me that this would not be a legitimate copy.
The format is a highly-structured, unobfuscated binary format. It is relatively clear how changes in the editor affect the data files, so I do not think any analysis would be considered circumvention of copy protection mechanisms.
So far I have only examined the editable data and do not plan to analyze or work with the packaged data if it will take significantly more work (for example if it is compressed or obfuscated).
I found an umbrella EULA applying to many different products on an archive of the publisher's website. I don't know whether a separate EULA with different terms was distributed with the software I am interested in, but the EULA I found does not expressly prohibit reverse engineering. If I can obtain the software legally, existing questions suggest it would at least not be copyright infringement to perform reverse engineering in the way I intend (Can software be legally rewritten if the original source code is not used?).
I plan to compare data files before and after changing them using the original software. I am not planning on disassembling the software or otherwise referencing the original algorithm used to serialize the data.
Intended purposes of reverse engineering
I wish to document the format and make this documentation freely available to other users.
I wish to create and distribute a tool for converting a project in the original format into a common interchange format like JSON, for import into other projects.
Reimplement and extend interpreter
I wish to implement an interpreter that will run on the converted data files. This would be implemented from scratch, without trying to reverse engineer the original interpreter. The aim would be an accurate reproduction of the original behavior, but not necessarily perfect. My intent is to make the interpreter cross-platform and to include new features not originally present. Ideally, I can freely distribute this (for example under a copyleft license).
Implementing an editor is not one of my goals, as I think the original is adequate for existing projects, and there are better tools available for new projects.