It seems to me that the company would have a pretty strong case if they decided to sue you in small claims court to recover their expenses. The quote that they sent you and your acceptance of same (either via e-mail or verbally) would probably be viewed as sufficient to constitute a contract: they offered to make the object in exchange for a certain amount of money, you agreed to the terms, and you were both on the same page about what was being discussed.
The fact that there was a delay in the performance of the contract (i.e., they took 4 weeks to complete the job when the quote said 2 weeks) might weigh in your favor, but it doesn't sound like the contract (such as it was) addressed what would happen if the job wasn't completed on time. If there wasn't any specific clause in the contract concerning delays, the standard is whether the delays were "reasonable". They could probably make an argument that a two-week delay is reasonable and that they proceeded with the job in good faith, and that it was incumbent upon you to cancel the job if you wanted to back out. Your case would have been stronger if you had attempted to contact them to cancel the job.
If they did successfully sue you, they would have to show damages. In this case, the damages would be any expenses they incurred in the process of creating the object for you. Since these expenses would probably include materials & labor—exactly the things you agreed to pay them for in the first place—the damages would probably be at least as much as the amount you agreed to in the contract.
It is conceivable that if you take the "screw you, I'm not paying" route, they may decide that it's not worth the aggravation to take you to court over such a small sum. The likelihood of this scenario might depend on how easy it is to file a small claims case in your jurisdiction, and how vindictive the company's management wants to be. Pursuing this route does mean, of course, that you shouldn't expect this company to ever want to do business with you in the future.