As an example, Canada didn't sign the Apostille Convention while the United States did. Practically speaking, what is therefore the difference between the documents issued by these two states? Or in other words, why wouldn't other countries accept documents issued by Canada where they would accept documents issued in the US?
Documents from countries which did not sign (or to be received by non-signatory countries) must be certified by the foreign ministry of the source country (or its equivalent) and then further certified by the foreign ministry of the receiving country, before such documents can be used in legal proceedings.
The Wikipedia article says:
A state that has not signed the Convention must specify how foreign legal documents can be certified for its use. Two countries may have a special convention on the recognition of each other's public documents, but in practice this is infrequent. Otherwise, the document must be certified by the foreign ministry of the country in which the document originated, and then by the foreign ministry of the government of the state in which the document will be used; one of the certifications will often be performed at an embassy or consulate. In practice this means the document must be certified twice before it can have legal effect in the receiving country. For example, as Canada is not a signatory, Canadian documents for use abroad must be certified by the Deputy Minister of Foreign Affairs in Ottawa or by a Canadian consular official abroad, and subsequently by the relevant government office or consulate of the receiving state.
This seems to answer the question rather completely.