Child marriage is completely illegal only in Delaware and New Jersey, and is also redefined (upwards) in Nebraska and Mississippi. In a little over 1/3 of the states, there is no lower age limit. In Alaska, for example, those betweeen 16 and 18 can marry with parental consent, and those between 14 and 16 can marry with a court order. Arkansas does not impose any lower age on court-approved marriage.
So to some extent, most states do judge each marriage license application on a case by case basis, in particular for those who are below some statutorily set age – 18 most commonly, 19 in Nebraska, 21 in Mississippi. No state requires case-by-case judgment of those over 21. If there were no statutory limits, every marriage would be subject to case-by-case judicial approval. Such a move could have immense consequences for the concept of "rights". In the US, there are various fundamental rights, such as the right to contract, to drink, to marry, to vote, which cannot be freely abridged by the government. A law which curtails a right is subject to "strict scrutiny" meaning that it must addresses a compelling government interest, must be narrowly tailored to address that interest, and must be the least restrictive means of doing so. Subjecting all marriages to case-by-case judicial scrutiny would not survive strict scrutiny, nor would a law freeing all persons from contractual responsibility in lieu of a court order.
The typical age of marriage 18 aligns with the broader judgment that that is the age when individuals are capable of fully exercising rights and taking responsibility. There are numerous "case-by-case" exceptions that go both ways (e.g. a person over 18 may be declared incompetent; a person under 18 may be declared competent). The premise that there are no case-by-case exceptions is wrong. More narrowly, one would ask why in Alaska the lowest legal age for court-ordered marriage is not set at 13 years and 6 months, or 13 years and 4 months. Arkansas answers that question by saying "we let the judge decide".
This only addresses the question of age of marriage, as asked in the OP. Laws regarding statutory rape are even more nuanced (though they do not involve judicial pre-approval). Whether or not a foreign marriage is recognized in a state depends on that state's law. Washington state generally recornizes marriages that are valid in another jurisdiction unless the marriage is specifically prohibited, where bigamy and incest-related restrictions exhaust the legal limits.