Flying a flag indicates your country.
If you fly any naval ensign or flag at the stern or topmast, you declare "I am part of this nation, my ship is to be treated as territory of this country". Or to quote a law firm:
The flag of the ship represents the nationality of the ship, i.e. the ship is under the control of the registered country. Based on the ship's flag, the ship must comply with international and maritime law of the registered country in the open sea and It can be used in a variety of ocean conflicts.
If you look at a list of all the naval ensigns, you might notice that even flags that are very close normally have distinct and more distinguishable naval ensigns at times, and in fact, there are no ensigns that can be mistaken for another nation easily.
The order isn't an Art. 91/92 state
However, The Order of Malta is not a recognized state for Article 91, and does thus not have the power to grant an ensign. It also does not have a ship registry where you could register your ship to gain the right to fly the flag of its order works. It has no naval ensign at the current time or territory, but the organization does operate and works with about half of the states on the earth. That doesn't make it a state though, and you don't qualify under non-existant rules to an entitlement to fly the Order Works flag under Article 91 and it doesn't give anyone the power to fly its flag under Article 92. The Hospitallers are also not a member of the 175 Members of the United Nations International Maritime Organisation.
Consequences
If you only fly the "Order Works" flag that looks exactly like the Maltese Civilian ensign, for the law of the sea you are treated as a Maltese civilian ship, as you fly a flag that is recognized as the recognizable country flag, which is the Maltese civilian ensign. Under Article 92, you are now treated as a Maltese ship at first sight, because that is what you look like. If it becomes apparent that you are not actually a Maltese ship, you are next treated as a ship without nationality until your nationality of registration is ascertained.
You could fly only the Hospitaller's State Flag, and avoid the confusion that way, but you would violate the law of the sea (Article 92) by not flying a recognized country flag. As such, you are seen as a nationless ship and can be boarded by any country's warship under Article 110 as they suspect you to be without nationality and need to verify which flag you should fly. Of historical note: The Knights did fly the white cross on red for their fleet.
You could fly a proper ensign for your ship's country of registry, and hoist the Maltese Order Flag as an additional ensign of lower rank. Now you are in total compliance with the law of the sea as long as the additional ensign is noticeably not the main country ensign. It would most likely need to be placed similarly in size and position to a yachting club or a shipping company's flag: clearly smaller than the nationality flag, so it can't be mistaken for the actual ensign of the ship's country of registry.
Article 91
Nationality of ships
Every State shall fix the conditions for the grant of its nationality to ships, for the registration of ships in its territory, and for the right to fly its flag. Ships have the nationality of the State whose flag they are entitled to fly. There must exist a genuine link between the State and the ship.
Every State shall issue to ships to which it has granted the right to fly its flag documents to that effect.
Article 92
Status of ships
Ships shall sail under the flag of one State only and, save in exceptional cases expressly provided for in international treaties or in this Convention, shall be subject to its exclusive jurisdiction on the high seas. A ship may not change its flag during a voyage or while in a port of call, save in the case of a real transfer of ownership or change of registry.
A ship which sails under the flags of two or more States, using them according to convenience, may not claim any of the nationalities in question with respect to any other State, and may be assimilated to a ship without nationality.
Article 110
Right of visit
- Except where acts of interference derive from powers conferred by treaty, a warship which encounters on the high seas a foreign ship, other than a ship entitled to complete immunity in accordance with articles 95 and 96, is not justified in boarding it unless there is reasonable ground for suspecting that:
(a) the ship is engaged in piracy;
(b) the ship is engaged in the slave trade;
(c) the ship is engaged in unauthorized broadcasting and the flag State of the warship has jurisdiction under article 109;
(d) the ship is without nationality; or
(e) though flying a foreign flag or refusing to show its flag, the ship is, in reality, of the same nationality as the warship.