This is addressed in OpenStreetMap's Legal FAQ, and, as usual, it depends.
3d. If I use your data together with someone else's data, do I have to
apply your license to their data too?
If the two datasets are independent, no, you don't; this is a
Collective Database.
If you adapt them to work together (for example, by taking footpaths
from the OSM data, roads from the third-party data, and connecting
them for routing), this is a Derivative Database and so you must (as
per 3b). However, if the two datasets are matched "trivially" by, for
example, automated matching using a simple criterion such as
name/locality, this is not "substantial" and remains a Collective
Database. There is a Community Guideline on what constitutes a trivial
transformation.
OpenStreetMap Legal FAQ, question 3.4 3d
So it depends on whether your data and the OSM data are "independent", and whether your changes to the OSM data itself qualify as a "trivial transformation". This distinction is a bit vague, and is based a similar notion in the EU Database Directive. I don't think you'll find much more specific general guidance - if you need more assurance, you'll have to work with a lawyer.
In your case, if your metadata is stored separately from the OSM data (ideally, technically separated, as in a different DB table or different file), I would think it would be considered "independent". The only link between OSM data and your data would be the reference to the OSM object in the metadata (probably by coordinate or OSM ID) - that should fall under _"trivial matching" as explained above.
Thus you create a "Collective Database" and not a "Derivative Database", and the OSM license probably does not apply to your data.