Micrsoft's '.docx' file format is closed source. There are many softwares that convert '.docx' file to any other (ex. pdf,rtf,odt,txt) . These software developers must have to reverse engineer '.docx' file format in order to study it and be able to convert it to another. So is it legal to reverse engineer these kind of proprietary file formats?
Yes it is legal and unnecessary because WordProcessingML is open source and standardised as ECMA 376.
Change the extension from .docx to .zip and browse with Windows Explorer and open with Windows Notepad to your heart's content. The schemes for the various versions are here.
Microsoft also offer Visual Studio libraries to allow direct manipulation of the XML.
It is not illegal per se to reverse engineer software, unless the software is patented. Since software patent is a huge can of worms, I assume there is no patent protection for the item in question (MS says that they "may have patents" on the technology, so it is possible that such patents may or may not be valid in the US). Software is subject to copyright protection, and by the nature of how computers work, one has to make many copies of the item which one acquired, in order to actually use it (install on hard drive; copy from hard drive to RAM to execute). While there is US law that permits such automatic copying that is necessary for the functioning of a program, it only allows this limited copying if you legally own the copy of the software. And it turns out that you don't own the copy of the software, you own the disk (if there is one) and the license to use the software.
So, you have to look at the terms of the license to see what is allowed vs. prohibited. MS is uncooperative about making license terms easily available for inspection so I can't give you a version of the EULA that applies to docx, but from Wd2003, the EULA has a clause
You may not reverse engineer, decompile, or disassemble the Software, except and only to the extent that such activity is expressly permitted by applicable law notwithstanding this limitation
This is not exactly crystal-clear. While it is clear that you can't clone the executable program itself using the aforementioned methods (to the extent that anybody knows exactly what 'reverse-engineering' is), this doesn't say that it is forbidden to make surmises about the logical relationship between a document, and how the program treats the document. If the license said explicitly "nor shall thou attempt to use the behavior of Our Product to discern the logical structure of output files and create a format converter", then that would be clearer. Since they had the option of explicitly saying that you can't try to figure out the file structure, it's not unreasonable to assume that you're not being prohibited from doing so, though there are only two ways to know (ask; don't ask but do).