IANAL, but arguably it could fall under the following prohibition from Schedule 2 of the Consumer Rights Act 2015:
A term which has the object or effect of limiting the trader's obligation to respect commitments undertaken by the trader's agents or making the trader's commitments subject to compliance with a particular formality.
In practice, unless you can find case law that found such by-phone-only terms invalid under that "particular formality" provision (or some other)... it's probably going to be an uphill battle unless you're prepared to lawyer up and go to court.
There is however further Government (OFT) commentary (on the older law, but this provision is unchanged):
It is often administratively convenient if the consumer complies with
formalities – for example, procedures involving paperwork – and may even
be sensible from the consumer's own point of view. But that does not
justify a business opting out of important obligations where the consumer
fails to comply with a minor or procedural requirement and commits a trivial
breach.
Unless the need to observe a formality is obvious and important, or is
prominently drawn to the attention of consumers, they may overlook or
forget it. That is particularly so if it has to be complied with some time in
the future without any reminder. Terms imposing severe penalties for trivial
breaches committed inadvertently are open to strong objection.
Obviously where compliance with a formality involves disproportionate
costs or inconvenience, the potential for unfairness is even greater. An
example would be a requirement to use registered post for written
notification when notification by ordinary post would be perfectly adequate.
So that seems to clearly apply if the phone line (to cancel) is not toll-free for instance, and they put you on hold. It's less clear if courts would read that as prohibiting the mere specification of a (non-costly) method of communication, like a toll-free line. I suppose the inconvenience factor can be raised, if there is substantial wait time.
Otherwise it's less clear if neither cost nor inconvenience applies if the customer can easily reject the trader's choice of communication venue. I suppose the fact that snail mail is generally acceptable for contractual matters can be invoked since given that cancellation (restriction) "has to be complied with some time in the future without any reminder", i.e. you can pretend you forgot that weird clause and used regular correspondence.
Finally, since this is pre-drafted contract by a big business, you can complain to the CMA, formerly OFT. If the law hasn't changed in this regard (since OFT's time), if the complaint (of unfair terms) is judged valid by the government agency, they will sue the company typically obtaining an injunction that explicitly prohibits that clause/practice for all consumers.