Operationally, in the United States, a public stock exchange is a stock exchange which only conducts transactions involving securities in publicly held companies for members of the general public through brokers authorized to participate in the exchange, while a private stock exchange is a network that facilitates private placements of securities (i.e. transfers of securities that fall within a securities law exemption) in companies that are not publicly held, between accredited investors (basically affluent people, certain sophisticated people and institutional investors).
A company is publicly held if it has more than a certain amount of shareholders with a certain amount of market capitalization, if it is listed on a public stock exchange, or if its securities are marketed to the general public. In practice, there is very little middle ground.
Most companies either have just a handful of owners and investment debt holders (i.e. are closely held), or have a great many (typically thousands), while few are near the threshold between being closely held and public held. Typically, a company with enough securities holders to be publicly held has a net asset value that requires the company to comply with the obligations of a publicly held company.
A company that has securities that is publicly held is subject to much more strict disclosure requirements and liability for non-disclosure of material facts than a company that is not publicly held. Publicly held companies are also not allowed to be taxed as pass through entities. This is quite expensive (hundreds of thousands of dollars to single digit millions of dollars per year in a typical case, mostly for lawyers, accountants, printers and investment bankers).
There are many kinds of securities, but the most common are common stock, preferred stock, and bonds.
Almost all of the technical terms used in this answer have highly specific legal and regulatory definitions.