Can the government of the country of birth legally access this person's personal information while they are living in the EU? Examples are health care records, internet activity and location data.
It depends on the location of the information and on the circumstances under which the country of birth seeks access to the information. Anything located in that country falls under the country's own laws, so EU law is not likely to be relevant.
Anything located in the EU would nominally be subject to EU law, but governments are generally held to different standards than private actors. For example, one of the lawful bases for processing in the GDPR is "compliance with a legal obligation." That includes obligations that arise under foreign law, such as the law of the country of birth.
So, depending on the circumstances, the answer to the question could be "yes" or "no," even for personal information that is and always has been present only within the European Union.
Two notes: First, the GDPR does not operate based on citizenship. It applies equally to a dual citizen residing in the EU and to a third-country national residing in the EU. Second, one commonly cited principle of dual citizenship is that the dual citizen generally has all the rights and obligations of a single citizen under the laws of each country.