Under Art 25 and Art 32 GDPR, a data controller is required to implement appropriate security measures. Encryption is explicitly suggested as a potential measure. What is appropriate depends among other factors on a risk-based analysis.
Here, I can't see any threat model that would reasonably require encryption of a locally stored literature database. I therefore doubt very strongly that this encryption would be required or appropriate under the GDPR.
It seems more likely that the encryption was implemented to limit informational self-determination of the data subjects, thus going against the GDPR's intent. There might also be concerns about whether this processing is even necessary or suitable to achieve the stated security purposes, considering the data minimisation principle and the purpose limitation principle in Art 5.
GDPR also contains a right to data portability, which requires the data controller to provide the personal data in a commonly used, machine readable format. However, this right only applies under specific circumstances.
The EU's ePrivacy directive regulates among other things how service providers can access information on your device. Such access is only permissible as far as it is necessary to provide a service explicitly requested by the user. Access beyond that would require consent, and consent must be informed, specific, and freely given. However, it is not clear to me which data was affected by the encryption, so there might be an argument that it was explicitly requested by updating.