Yes, phone numbers would generally qualify as Personal Data under GDPR. It would be so irrespective of whether you have also stored other information along with the phone numbers or not, since also information that indirectly could identify a natural person is Personal Data (provided that there are, somewhere else, public or not public, a register of who holds the specific phone number). See Article 4(1) GDPR.
(One could possibly argue that the phone numbers would not be considered personal data if there is no actual register of who owns a specific phone number with any other party. Or if such register is in practice not available for anyone. It might be so in some cases, although I would not rely on it.)
Whether you have the right to process the phone numbers must be assessed based on its lawfulness (see Article 6 GDPR). It could be based on consent, performance of contract, legitimate interest or any other ground set out therein.