Everything you want to do is completely illegal. Consider the time you have invested so far a learning experience. Very little of your work will be salvageable.
You need to start over pretty much from scratch with an original visual and musical pallet only vaguely inspired by the original. If you don't, you will be sued for many, many different reasons and lose.
Intellectual Property Problems
There are multiple intellectual property-like problems with your plan.
If you use any registered trademarks (or unregistered claimed of trademark designed with the superscript "TM" by the restaurant) you are probably looking for trouble.
If the architecture or anything in the restaurant is protected by copyright or a design patent, you are looking for trouble.
The restaurant can probably claim "trade dress" protections of its "look and feel" to the extent that those elements are a consistent part of the restaurant's image and is distinctive.
If individuals in the game correspond to individuals in real life from whom you have not received permission, you are probably engaged in tortious violation of that person's right of publicity.
If you use music used in the restaurant without permission in the game, this is probably a violation of somebody's copyright unless you obtain a license (or statutory license for covers of music copyrighted by someone else).
If you are a former employee of the restaurant, there may also be contractual limitations in disclosures of elements that are considered "trade secrets".
Tort Law Problems
In addition to intellectual property issues, there is a good chance that if the correspondence between the real restaurant and your game is close enough, and the game displays the restaurant in a negative light (perhaps partially due to video game hyperbole) that you will be sued (quite possibly successfully) for defamation, for "injurious falsehood", or for the tort of "false light" (if that privacy tort has been adopted in a state where the game is sold).
Bottom Line: You Are Screwed If You Proceed
Basically, if you can't obtain permission, it is a really bad idea to proceed according to your original plan and you will be successfully sued to no end.
Moreover, even if you do win a lawsuit in the end, you will go broke trying to defend it and will be hopelessly distracted for years from your main business of selling video games.
Options To Make Your Video Game Legal
You can either parody the original (which is quite tricky), or you can follow the advice in the comment from @schizoid04 and change things to a similar degree to what you commonly see on television or in movies where they clearly don't have the licensing rights to the real thing.
If the game uses the images in connection with culinary or political criticism it could also be protected, but this video game probably would not rise to that level for intellectual property purposes.
Should I simply give the restaurant a different name? Should I feature
the restaurant in a different city than where it really resides?
What if I mirrored all of the architecture to be backwards like how
people "Mirror" copyrighted video on youtube?
None of these measures (or even all of them combined) go nearly far enough to be sufficient to ward off intellectual property or defamation-type lawsuits.
You need to rebuild from scratch touching on some of the general ideas or cliches for this type of business, but none of the distinctive specifics that associate this particular business with your images.