Can an American company state in their terms of service that if an entity is from a designated country they are not allowed to use the service?
In other words would that be a legal and enforceable provision in the United States?
Yes, and this is very common - Cuba, North Korea and Iran are often excluded.
Mind you, this is from a US perspective. The banned country may have other ideas about the legality of the ban under its laws. But if you weren't planning to do business with that country anyway, that's not exactly going to hurt you.