My understanding is that if the second husband adopts the two
children, he will be on the hook for child support. But suppose he
doesn't.
Is the second husband then liable for child support?
No. The second husband is only liable for child support if he has adopted the children. (Also, many, but not all, states do not allow for stepparent adoption at all, unless the parental rights of a birth parent have been terminated. The question implies that there has not been a termination of parental rights in this case.)
In theory, if he sought and been granted significant legally confirmed parenting time of the children (parenting time can be awarded to non-parents who have significant relationships with a child in many states), he could be awarded child support from his ex-wife and their father, but he wouldn't have to pay child support, even then.
It is conceivable that the fact that his ex-wife is caring for minor children limits her earning capacity. Her earning capacity could be one factor among many used to determine if alimony will be awarded, and if so, how much, for how long. But, this impact on an alimony award does not mean that it is child support and does not differ in its relevance to an alimony award for many other factors that could impact her earning capacity (e.g. her education, her work experience, any disabilities she has, her age, etc.).
It is also possible that temporary family support during the pendency of the divorce case could take into account the ex-wife's need for funds to support her children, even though they are not his children, but again, this is an alimony type decision, which in the temporary while a divorce case is pending category, is driven by actual need in the short term of each spouse while they are reworking their relative finances (much as it might be influenced by the cost an ex-wife incurs to maintain a horse or dog), rather than by the needs of the children in their own right. She might, for example, own the house but have only a small income, but adjusting the situation so that both spouses have sustainable finances after the divorce is what these court cases are all about.