A partial answer (for a manufacturer) is "look it up" – that may tell you if a determination has been made by the FDA. If so, it is not new and
may require a less extensive review the next time it is included in a
new drug product. For example, if a particular inactive ingredient has
been approved in a certain dosage form at a certain potency, a sponsor
could consider it safe for use in a similar manner for a similar type
of product
(the FDA says). The legal part doesn't explain how it is scientifically determine that an inactive ingredient raises safety concerns but to take on example, wheat gluten is in the Inactive Ingredient Guide. This article gives a bit of analysis of the factors that sponsors have to consider in shouldering their burden of proof. If you want to manufacture a new vaccine, you have to show (the FDA) that it is safe. If you dilute the vaccine with water, that ingredient is not a "safety factor". Benzalkonium chloride could be (you would have to show that it isn't), and if it is, that regulation requires you to list it. What's not specified clearly is how many adverse reactions per million doses constitute being a safety factor.