A court can acknowledge that the accused was defending themself, but decide they acted too violently under the circumstances and is guilty of murder by virtue of excessive or unlawful self-defense. This is called "imperfect self-defense".
Imperfect self-defense is a common law doctrine recognized by some jurisdictions whereby a defendant may mitigate punishment or sentencing imposed for a crime involving the use of deadly force by claiming, as a partial affirmative defense, the honest but unreasonable belief that the actions were necessary to counter an attack. Not all jurisdictions accept imperfect self-defense as a basis to reduce a murder charge.
Wikipedia article on Imperfect self defense
Whether "murder" and "self-defense" are mutually exclusive depends on the jurisdiction and how the self-defense argument is mounted.
In Texas, imperfect self-defense can either mitigate the severity of the crime (reducing a murder charge to manslaughter) or it can mitigate the punishment after the defendant has been found guilty of murder (reducing a death penalty to a life sentence, e.g.). So self-defense and murder are not always mutually exclusive.
In the U.K., imperfect self-defense is a mitigating factor in what is still called "murder".
In Maryland, imperfect self-defense and murder are mutually exclusive. (But I'm not sure my one citation is proof enough.)
The Texas statutes under Title 5 (Offenses Against the Person), Chapter 19 (Criminal Homocide) set out the definitions for "Murder", "Capital Murder", "Manslaughter", and "Negligent Homocide". We'll just look at "Murder" and "Manslaughter" as being relevant here.
Sec. 19.02. MURDER.
(b) A person commits an offense if he:
⇒(1) intentionally or knowingly causes the death of an individual;
⇒(2) intends to cause serious bodily injury and commits an act clearly dangerous to human life that causes the death of an individual
Sec. 19.04. MANSLAUGHTER.
(a) A person commits an offense if he recklessly causes the death of an individual.
So in their basic definitions, it seems that deliberately shooting someone would qualify as "murder" regardless of self-defense being applied, while "manslaughter" would apply if the shooter was just aiming wildly in the dark trying to scare off the shootee or something similar.
The Texas statutes under Title 2 (General Principles Of Criminal Responsibility), Chapter 9 (Justification Excluding Criminal Responsibility), Subchapter C (Protection Of Persons) provides defense-related definitions. Sections 9.31, 9.32, and 9.33 describe "Self Defense" (non lethal), "Deadly Force In Defense Of Person", and "Defense Of Third Person" (lethal or non-lethal), respectively.
Under those definitions, "self defense" explicitly refers to using non-lethal force against someone, while murder requires the use of deadly force. This means "self defense" (in this very narrow definition) and "murder" are mutually exclusive. However, it's clear from reading excerpts from pretty much any murder case that the term "self defense" is also used for lethal force cases in practice, so this logic doesn't hold for real-world cases.
Of note is that neither the homocide statutes nor the justification statutes mention anything about imperfect self-defense. Either you were defending yourself legally, or it was murder. However, there is common law related to the subject.
Appellant therefore was not entitled to have the self-defense issue submitted to the jury at the guilt-or-innocence phase of his trial. Nevertheless, his imperfect self-defense claim warranted submission of the provocation issue at the penalty phase.
Charles Evans v. State of Texas at the Court of Criminal Appeals of Texas (1980) via Justia
Under the common law doctrine of imperfect self-defense, if a defendant merely intended to assault the victim and ended up killing him because of escalating violence that he did not intend, the actor would be acquitted of murder and convicted of another offense, depending on his culpability.
Sergio Huerta v. State of Texas at the Criminal District Court No. 2 (2008) via Justia
As explained below, imperfect self-defense … [is a] defensive issue that may be relevant in evaluating a claim of self-defense in certain cases.
Erick Santos-Valdez v. State of Texas at the Fourteenth Court of Appeals (2014) via Justia
In the 1980 case, imperfect self-defense was only useful in determining the sentence (specifically, death penalty vs. life sentence in this case), and not in determining the charges. In this case he was convicted of capital murder, but imperfect self-defense was at play in overturning his death penalty. So murder and imperfect self-defense were not mutually exclusive. (Appellant robbed a store and claims he only shot the manager because the manager shot first. Clearly, the appellant had no right to self-defense so the murder conviction was upheld, but because he was shooting defensively, it was a mitigating factor in the murder sentence.)
In the 2008 case, imperfect self-defense might have been used to reduce the charge from murder to manslaughter. So in that application, murder and self-defense were mutually exclusive. (Appellant shot in the air during a vehicle robbery to scare the victim. Victim shot appellant. Appellant claims he only walked to the vehicle and unloaded a magazine into the victim because he was surprised by being shot. Appeals court disagreed.)
In the 2014 case, imperfect self-defense is irrelevant to the case, but it's worth noting that the concept is still valid as of five years ago. (Appellant shot a man who was grappling him through the window of his car, after the appellant's friend tried to rob the man. The appeal summary isn't very specific on why this was murder, since it's only dealing with technicalities.)
The UK still considers it murder, but allows mitigating the minimum term of the prison sentence, if self-defense was somewhat involved:
As murder is such a serious crime, the approach to sentencing for this offence is set out in law.
Mitigating factors are things that may reduce the minimum term. These include:
• the fact that the offender acted to any extent in self-defence or in fear of violence
Sentencing Council for England and Wales
The state of Maryland explicitly considers "murder" to include malice, while a legal defense of "imperfect self defense" explicitly requires proof there wasn't malice. As such, "self defense" and "murder" are mutually exclusive under legal definitions, while "self defense" and "manslaughter" are not.
Initially, we note that the difference between murder and manslaughter is the presence or absence of malice.
Imperfect self defense, by contrast, is not a complete defense. Its chief characteristic is that it operates to negate malice, an element the State must prove to establish murder. As a result, the successful invocation of this doctrine does not completely exonerate the defendant, but mitigates murder to voluntary manslaughter.
State of Maryland v. Melvin Faulkner at the Maryland Court of Appeals (1984) via Google Scholar