The grounds for setting aside a conviction because you have a bad lawyer is called "ineffective assistance of counsel". This can (usually) only be raised when your direct appeals have been exhausted in a collateral attack on a conviction which was historically called a habeas corpus petition, although some state systems give it a different name.
In a nutshell, the lawyer's work must be very, very bad to the point where a responsible criminal defense lawyer would feel an urge to throw up upon hearing about it.
The legal standard in making his ineffective assistance claim is set forth in Strickland v. Washington, 466 U.S. 668 (1984): One must show that one's attorney's performance was deficient and (2) that deficient performance caused actual prejudice to the defendant’s defense. Id., 466 U.S. at 687.
To prove deficient performance under Strickland, a state prisoner must
“demonstrate that counsel’s representation fell below an objective
standard of reasonableness” under the then “prevailing professional
norms.” Id., 466 U.S. at 688. The Supreme Court has recognized that
the American Bar Association Standards for Criminal Justice is the
barometer for measuring “what is reasonable.” See: Rompilla v. Beard,
545 U.S. 374, 387 (2005). While the Fifth Circuit applies a strong
presumption that counsel performed adequately and insulates his
informed tactical decisions from ineffectiveness attack unless they
are so egregious as to render the entire trial unfair, the appeals
court has recognized the distinction between strategic judgment
decisions and omissions that amount to no strategic decision at all.
See: Virgil v. Dretke, 446 F.3d 598, 608 (5th Cir. 2006); Moore v.
Johnson, 194 F.3d 586, 604 (5th Cir. 1999).
Deficient performance alone is not enough to secure ineffective
assistance relief. The state prisoner must establish prejudice by
showing that “there is a reasonable probability that, but for
counsel’s unprofessional errors, the result of the proceedings would
have been different. A reasonable probability is a probability
sufficient to under confidence in the outcome.” Id., 466 U.S. at 694.
The Fifth Circuit has interpreted this prejudice requisite to mean
that there is a harmful constitutional trial error only if there is
“more than a reasonable probability that it contributed to the
verdict.” See: Mayabb v. Johnson, 168 F.3d 863, 868 (5th Cir. 1999).
Federal relief after it has been denies at the state level in a state collateral attack, is even more limited by Antiterrorism and Effective Death Penalty Act of 1996 (“AEDPA”), especially 28 U.S.C. § 2254(d) (from the same linked source):
“The AEDPA specifies that federal habeas relief ‘shall not be granted
with respect to any claim that was adjudicated on the merits in State
court proceedings unless the adjudication of the claim resulted in a
decision that was contrary to, or involved an unreasonable application
of, clearly established Federal law, as determined by the Supreme
Court of the United States; or resulted in a decision that was based
on an unreasonable determination of the facts in light of the evidence
presented in the State court proceeding.’
“The Supreme Court has held that a state court’s decision that
correctly identifies the governing legal rule but unreasonably applies
it to the facts of a particular prisoner’s case is sufficient for a
federal habeas court to grant the writ. For a federal court to find a
state court’s application of Supreme Court precedent ‘unreasonable,’
however, the state court’s decision must have been more than simply
incorrect or erroneous; its application of federal law must have been
‘objectively unreasonable.’ Moreover, the state court’s findings of
fact are presumed to be correct, and the federal court only reviews
the facts for clear and convincing error.” Id., at 385.
For example, most of the instances described below have been held in particular cases to be insufficient to overturn a conviction in a death penalty case:
In many cases, the appointed attorneys are overworked, underpaid, or
lacking the trial experience required for death penalty cases. There
have even been instances in which lawyers appointed to a death case
were so inexperienced that they were completely unprepared for the
sentencing phase of the trial. Other appointed attorneys have slept
through parts of the trial, or arrived at the court under the
influence of alcohol.
While not precisely on point, the threshold for reversing a conviction based upon ineffective assistance of counsel is generally lower than the standard to be subject to discipline or to be disbarred as an attorney:
Justice Ruth Bader Ginsburg was more direct in an Associated Press
account: “People who are well represented at trial do not get the
death penalty … I have yet to see a death case among the dozens coming
to the Supreme Court on eve-of-executions stay applications in which
the defendant was well represented at trial.”
Before the 2001 public criticisms offered by Justices O’Connor and
Ginsburg, three major newspapers had conducted investigations that
offered compelling evidence about the deplorable legal representation
provided in capital cases. The Chicago Tribune reported on November
15, 1999 that 12% of those condemned to death from 1976 to 1999 were
represented by “an attorney who had been, or was later, disbarred or
suspended—disciplinary sanctions reserved for conduct so incompetent,
unethical or even criminal that the state believes an attorney’s
license should be taken away.” The newspaper said that an additional
9.5% had “received a new trial or sentencing because their attorney’s competence rendered the verdict or sentence unfair, court records
show.” (Ken Armstrong and Steve Mills, “Inept Defenses Cloud
Verdict”).
Less than a year later (September 9, 2000) the Charlotte Observer
reported that at least 16 condemned inmates in North Carolina,
including 3 who had been executed, were represented by attorneys who
have been disbarred or disciplined for unethical or criminal conduct.
The following day the Dallas Morning News reported that it had
examined 461 capital cases in Texas and found that one in four of the
condemned inmates had been represented at trial or on appeal by
court-appointed attorneys who had been disciplined for professional
misconduct at some point in their careers. (“Quality of Justice,”
09/10/2000).
The most common issue that is successful is a complete or nearly complete failure to present meaningful mitigating evidence in the death penalty sentencing phase of a death penalty case.
What Explains These Patterns?
In Death Penalty Cases
But, this pattern of granting relief for ineffective assistance of counsel is partially a function of jurisprudence politics.
Many judges are opposed to the death penalty, but find on the record before them that it is likely to that defendant was not innocent of murder. If at least two appellate judges on a panel of three appellate judges believe this, a ruling that assistance of counsel was inadequate for this reason vacated the death penalty sentence, but not the conviction, usually leaving the defendant in prison for life without the possibility of parole.
Also, due to "death qualification of jurors" and the particularly poor public defender systems for death penalty case defendants in the states that produce the most death penalties, the risk of wrongful convictions in close cases, and of ineffective assistance of counsel, is particularly high. Most states with quality public defender systems for first degree murder defendants have either abolished the death penalty, or have very few death penalty convictions.
In Other Cases
The difficulty of an ineffective assistance of counsel defense from a "legal realist" perspective is that lots of guilty criminal defendants received ineffective assistance of counsel. If the bar is not set very high, a colorable claim of ineffective assistance of counsel can be raised in a very large share of all felony convictions, even though they are unlikely to prevail on the merits, either because the defendant's guilt was clear, or because the lawyers error was immaterial to the result (two points which overlap).
Keep in mind that there is no right to counsel on a collateral attack of a criminal conviction, that very few convicted criminals serving long terms in prison who had incompetent counsel are affluent, and that charitable free legal representation in collateral attacks on criminal convictions is mostly limited to death penalty cases and rape cases where DNA evidence is available.
More than 95% of convicted felons serving long terms have never set foot on a college campus as a student. So, in the usual collateral attack on a conviction based upon ineffective assistance of counsel, you have a high school drop out, or high school graduate who never went to college and is marginally functionally literate trying to convince a judge who graduated from law school that his lawyer who graduated from law school didn't know as much about the law and how to defend a criminal prosecution as the defendant does.
The likelihood that a defendant who received a marginal quality defense from their lawyer is guilty of a serious crime is particularly high for this subset of cases. Because it takes so long to adjudicate ineffective assistance of counsel claims (it isn't uncommon for it to take two or three years just to rule on a direct appeal from a conviction and another year or three to get a ruling on a collateral attack on a conviction that is denied before an appellate judge so it can make reported case law), almost all of the convictions appealed on this ground that produce case law precedent are very serious offenses like aggravated rape and murder. Otherwise, the issue would have become moot when the sentence of incarceration was served.
For most kinds of offenses, the vast majority of strong cases for the prosecutor result in guilty pleas before trial, producing a substantial sentence discount, so only the hard cases go to trial. But, if the death penalty is on the table (and the prosecutor won't budge, perhaps for political and publicity reasons), or an extremely long sentence for a violent felony is on the table, the sentence one can bargain for in a plea bargain, and the sentence that would be dispensed if one is found guilty at trial, are much more similar. So, the incentive for a defendant to "roll the dice" by going to trial is greater, even if the defendant is actually guilty, so a larger proportion of cases with a high probability of conviction at trial go to trial anyway.
Also, it is hard to provide "competent" acquittal creating criminal defense cases when the client is clearly guilty. Sometimes the defense was weak because the lawyer was incompetent, but sometimes the lawyer presented a dubious defense because that was the best available option under the circumstances. (This doesn't mean that there is nothing for a competent criminal defense lawyer to do; but most of the work for a competent criminal defense lawyer in these cases comes in the sentencing phase and in the plea bargaining stage.)
Judges don't want to order the release of people convicted of serious, usually violent, felonies if there is a good chance that they are guilty and a good chance that a retrial might not result in a conviction simply because the evidence has grown stale. This is particularly true in this subclass of cases (when they don't involve the death penalty which presents some unique issues related to the "death qualification" of the jurors), with a higher than average percentage of actually guilty defendants.
While judges don't want innocent people to rot in prison, due to the barrage of dubious collateral attacks on convictions filed in weak cases, without lawyers, by convicted criminals serving long sentences, judges tend to become jaded about the merit of these collateral attacks. About one in seven cases in federal district courts (the trial court level of the federal courts) is a prisoner's petition. One in five appeals in the federal appellate courts is a prisoner's petition. But only about 1% of those petitions are granted in non-death penalty cases. This is because the defendants serving very long sentences have little to lose and everything to gain by filing even a marginal collateral attack with a low chance of success.
New precedent of Shinn vs. Ramirez
In the case of Shinn v. Ramirez, decided May 23, 2022, the U.S. Supreme Court further restricted the ability of a state prisoner from petitioning a federal court that state convictions should be set aside due to ineffective assistance of counsel. By a 5-4 vote, the U.S. Supreme Court held that "Under [28 U.S.C.] §2254(e)(2), a federal habeas court may not conduct an evidentiary hearing or otherwise consider evidence beyond the state court record based on the ineffective assistance of state post-conviction counsel."