Is deterministic anonymization of test data ok?

Question

Deterministic anonymization of test data?

When anonymizing multiple data sources and preserving data consistency we usually end up with a mapping table for "real to anonymized" values. Typically files or database.

Deterministic approach

If one instead uses an algorithm to determine the anonymized value, you would not need a mapping-table and thus processing would be much faster. Furthermore you would get repeatable results which makes it easier to unit test.

Example 1. Real SSN --> (algorithm) --> Anonymized SSN 2. Anonymized SSN --> (algorithm) --> Name, Addresses, Phone, etc

For (2) when masking I cannot see any issues, but for (1) I can see the practical limitation for a non-reversable algorithm when the anonymized values needs uniqueness, one-to-one.

What is good enough?

Creating synthetic data from scratch is an unrealistic dream for most businesses. Anonymized copy-of-production is often an enourmous job by itself and you preserve realism and "dirty data" necessary for all-round realistic tests.

Anonymizing test data is by itself not enough if one was to publicize, since it is almost always possible to re-engineer portions of it. However, data that is used internally for developing software at a company will never be publicized and barely shown at demos.

Is it "good enough" to use deterministic approach ?

It is not a perfect world, anonymization in itself is just a "good enough" approach for test-data used internally at a company. Deterministic one-to-one value generation may technically be possible to re-engineer, but it is "good enough" for use in a testing and development department.

Answer 1

You are not describing anonymization, you are describing pseudonymization.

Recital 26:
(26) The principles of data protection should apply to any information concerning an identified or identifiable natural person. Personal data which have undergone pseudonymisation, which could be attributed to a natural person by the use of additional information should be considered to be information on an identifiable natural person. To determine whether a natural person is identifiable, account should be taken of all the means reasonably likely to be used, such as singling out, either by the controller or by another person to identify the natural person directly or indirectly. To ascertain whether means are reasonably likely to be used to identify the natural person, account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments. The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an identified or identifiable natural person or to personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable. This Regulation does not therefore concern the processing of such anonymous information, including for statistical or research purposes.

Article 4, definitions:
(5) ‘pseudonymisation’ means the processing of personal data in such a manner that the personal data can no longer be attributed to a specific data subject without the use of additional information, provided that such additional information is kept separately and is subject to technical and organisational measures to ensure that the personal data are not attributed to an identified or identifiable natural person;

The problem of a deterministic algorithm is that it can be reversed if you know the code, or it might be reverse-engineered. In other words, suppose an attacker hacks your server. They will find pseudonymized data, so you might think that a data breach has been prevented. However, if the attacker can also read the source code, or if the algorithm is trivial to reverse engineer anyway, the personal data will be able to be recovered. Also, the developers will always be able to recover the personal data, because they will both have access to the pseudonymous data and know the algorithm. So who are you actually protecting your pseudonymized data from? You need to make that clear, and then try to find a solution that fits your goals. A simple algorithm might or might not be a good idea, depending on your goals.