The place of birth on the birth certificate is where the child was actually born. Indeed, usually it will be more specific than city and state or province and will also identify a hospital or residence or other place where the birth happened.
So, for example, if a child is born to U.S. diplomats in Paris, France (in or out of the embassy grounds), the birth certificate will say that the child was born in Paris, France at Charles de Gaulle Hospital.
But, that child will still be a U.S. citizen in all likelihood, because that child's mother, and/or married father or unmarried father who acknowledges paternity, is a U.S. citizen (in all likelihood) pursuant to 8 U.S.C. §§ 1401 and 1409. The child may or may not be a French dual citizen depending upon the citizenship law of France.
In the case of a French diplomat who has a child born physically in Washington D.C. (inside or outside the French embassy) the birth certificate will likewise state that the child was born in Washington D.C.
The French diplomat's child, however, will not be a U.S. citizen since Section 1 of the 14th Amendment to the U.S. Constitution's first sentence (which is also found in 8 U.S.C. § 1401(a)) states:
All persons born or naturalized in the United States, and subject to
the jurisdiction thereof, are citizens of the United States and of
the State wherein they reside.
But a French diplomat's child is not "subject to the jurisdiction" of the United States, so the French diplomat's child does not gain U.S. citizenship at birth (assuming for simplicity's sake that both of the child's parents are French citizens and are not U.S. citizens) despite the fact that the child was born in the United States.