You can ask for the data, but the data controller would probably refuse on the ground that it might not have been you carrying the key at the time, and he cannot disclose data possibly relating to another person.
You can ask for the data to be destroyed, but the same argument might apply - that the retained data does not relate to you, but relates to another person.
The situation may be different depending on whether a set of keys are issued to an apartment and the apartment occupiers use whichever key is convenient, or a specific key is issed to an individual for his personal use (such as an employee pass).
The building owner should have a retention policy so that key data is destroyed after a certain period. It should not be kept indefinately.