Suppose I, an amateur radio operator, want to buy a car from another amateur radio operator. I send him a contract over ham radio, and he agrees to it. We have violated amateur radio regulations by using it for commercial purposes, but does this nullify the contract?
Contract law involves a number of parts, the most relevant here being the formation of a contract, and the enforcement of its terms. There are various rules about formation, such as that you cannot hold a gun to a person’s head to force them to say yes (the courts would then say there was no contract), and then under other conditions a term in a contract might be illegal, for instance the courts will not order a person to commit a crime as one of the conditions of a contract.
In the scenario that you propose, no term of the contract involves an illegal action, the only presumed illegality is in the circumstances surrounding formation of the contract. Duress is illegal, because there is no voluntary mutual assent. That’s not the case here. So there is no established legal impediment to finding that a contract was formed. There are ample opportunities to test the theory that an element of illegality in the formation of a contract makes the contract void, for example if a physical instrument used in creating or transmitting the contract was used illegally (the paper was stolen, the paper was used without permission of the owner); the assent was made while trespassing; the contract language infringes copyright. Given the court’s very strong commitment to recognizing and enforcing contracts, it is highly unlikely that the courts would reject a contract over a technicality of this type
§97.113 of the FCC rules states the prohibited transmissions, the two relevant clauses being that the prohibited list includes
(2) Communications for hire or for material compensation, direct or indirect, paid or promised, except as otherwise provided in these rules;
(3) Communications in which the station licensee or control operator has a pecuniary interest, including communications on behalf of an employer, with the following exceptions…
(2) is not clearly applicable, since the communication is not for material compensation (e.g. A is not paying B to make a transmission). (3) is more likely applicable, since the parties each have a pecuniary interest in the communication. The exceptions involve being compensated for making a communication, or one can on an irregular basis offer amateur radio apparatus for sale or trade. Since the FCC regulations only provide bare languages and no explanatory texts, and they decline to provide any examples (this seems to be a policy thing), and there is substantial unclarity as to what the restrictions mean, see this analysis.
A finding of letter-of-the-law violation in this case cannot be made by the court in which breech of contract would be litigated. Instead, the FCC would have to first make a finding of law-breaking, then a party would have to prove in a separate cause of action that because of illegality in the circumstances surrounding formation of the contract, there never was an agreement.
As to being "illegal", (which is a strong word most often associated with criminal activity) if the FCC determined your data burst violated the terms and conditions of your radio station license agreement they might revoke your license, but they have no jurisdiction over car sales.
Now to the question of "binding" or not, there is no automatic voiding of the contract due to the medium used to communicate. (handshake, verbal, email, etc.) If they decide to go through with the sale nobody can stop them.
However, if either James or John wanted to get out of the agreement, and the other wanted to enforce it, one could challenge the other. The plaintiff may choose to make "commercial use" of amateur radio a pillar of their case that the contract is invalid, but it would be up to a judge to decide if the method of transmission of an intent to buy was germane.
We have violated amateur radio regulations by using it for commercial purposes, but does this nullify the contract?
Most likely it does, since the use you describe hinders at least one fundamental purpose of an amateur radio service:
Recognition and enhancement of the value of the amateur service to the public as a voluntary noncommercial communication service, particularly with respect to providing emergency communications.
The purpose of this contract seemingly fails to outweigh public policy or public interest in enforcing the prohibition. See Restatement (Second) of Contracts at §181.
Furthermore, the existence of legislation in this regard is indicative of the strength of that policy. See Restatement at §178(3)(b). Other criteria listed therein go in the same direction and are self-explanatory.
Is a contract agreed to in an illegal way binding?
It depends on the specific circumstances, including each party's mens rea, during the formation of the contract. The aforementioned sections of the Restatement ought to determine the validity.
There are other obvious conditions that render a contract null and void, such as where it is entered under duress or coercion. However, that goes beyond the scope of your question.