Yes, but ...
Unless the builder agrees to end the contract by agreement, you will be required to pay damages which may be more or less than the deposit.
Contract terms
Not all terms of a contract are created equal. The law devices contract terms into conditions, warranties and intermediate terms. A condition is a term that is essential to the execution of the contract, that is, it is "of the essence". A warranty is a requirement that a party is obliged to perform but, if they don't the contract can still reach completion. An intermediate term could be a condition or a warranty depending on how egregiously it is breached and you don't know which until it goes to court.
Any term can be expressly made a condition (or a warranty) by the contract - a common formula being time is of the essence. This is its original meaning and still has its traditional meaning in law although the phrase has entered the vernacular as a synonym for "quickly" or "urgent". In many ways, legal terms are fossils of ancient language use.
So, for your example, the construction work by the builder and the payment by you are conditions: these are essential features of the contract. The quality of the workmanship is a warranty: if the paintwork is a little dodgy or the windows are not quite plumb, that does not stop you from enjoying (most of) what you contracted for. Time, both for the builder to build and for you to pay, are intermediate terms.
Breach of Contract
Now, if one party breaches any term of the contract, the other party is entitled to compensation in the form of damages. Damages for breach is the amount of money required to restore to the innocent party the benefit of the contract.
So, for example, if the paintwork were "dodgy", the appropriate damages are the cost of employing a professional painter to rectify the work. Both parties have an obligation to mitigate loss so, in the first instance, you would have to give your builder a reasonable opportunity to rectify any defects, including dodgy paint, at their expense and, only if they failed to do so could you seek the costs of doing so - whether you actually used the money to fix the paint or blew it all on a holiday in Majorca is irrelevant.
In addition, if the breached term were a condition, including an intermediate term that was egregiously breached, the innocent party has a choice to make in addition to seeking damages. They can either affirm the contract and keep it alive, or they can terminate it and bring it to an end.
So, did your builder breach the contract?
Probably not.
It is unlikely that vague indications of 6-10 weeks to start and 3-4 weeks to finish amount to a term of the contract although they do inform the implicit term that a contract will be completed within a reasonable time. As such, a reasonable time for this contract is 3-6 months. If the work had been completed anywhere near this time the builder would not have been in breach.
However, it's irrelevant.
Your agreement is a valid waiver - in return for the builder's promise to do the job in "winter", you forgave any claim for breach of contract for the delay.
Repudiation
Unless the builder agrees to end the contract on whatever terms you can negotiate regarding further payments or refunds, your decision not to proceed is a repudiation of the contract - ending it without lawful cause - and therefore a breach of a condition. The builder is entitled to damages and can either affirm the contract (i.e. insist that you proceed) or accept your repudiation and terminate the contract for cause.
Practically, affirmation is not possible in this situation as a court will only force you to comply where damages are not adequate compensation. Examples of this would include a contract for the sale of something unique, like art, a vintage car or real property - no amount of compensation will allow the buyer to obtain the benefit of that unique item so a court may force the sale. All the builder has lost is money and money can adequately compensate for money.
The damages the builder is entitled to are those which would put them in the same position they would have been in if the contract had been completed. That is: £20,000 less whatever it would have cost the builder to complete the contract. This may be more or less than the £10,000 deposit.