Austin's view: the command theory of law—rules of a superior sovereign backed by force
The idea that law exists only if the rules are those of a superior sovereign and backed by force is an outdated conception whose most prominant proponent was John Austin. This is known as the "command" theory of law. This clearly cannot explain obligation at international law. But nor can it explain many of the rules in domestic legal systems that we have no problem conceiving of as law (e.g. the law-making process itself, the system of adjudication, rules that confer power, etc.).
Modern views: internal sense of obligation
The command theory of law has largely been supplanted by other theories of obligation such as that of H.L.A. Hart. In this view, law does not require rules to be backed by threats of force, or to be imposed by a superior sovereign. Law can exist and sustain itself purely through a group's internal point of view: do the participants (states, in the case of international law) "use the rules as standards for the appraisal of their own and others' behaviour" and do they see them as imposing obligations?
This focus on the internal viewpoint is actually part of the test for recognizing a norm as customary international law. For something to be customary international law, it must be accomanied by opinio juris. This is a "'psychological' element'" (John H. Currie, Public International Law, p. 195). States need to behave as if a rule has legal obligation. International judicial bodies will look to evidence about how states explain their motives, including how they explain deviations from the rule.
H.L.A. Hart explicitly noted the common question, "How can international law be binding?" (The Concept of Law, "Chapter X: International Law"). His answer:
there is something very confusing in this favourite form of question; and before we can deal with it we must face a prior question to which the answer is by no means clear. This prior question is: what is meant by saying of a whole system of law that it is 'binding'? The statement that a particular rule of a system is binding on a particular person is one familiar to lawyers and tolerably clear in meaning. We may paraphrase it by the assertion that the rule in question is a valid rule, and under it the person in question has some obligation or duty.
He recognizes that "one source of doubt on this point is simply the absense from the system of centrally organized sanctions." But for all the reasons that this is not the criteria of what makes law law or creates the internal view of obligation, even in domestic systems, this fact does not undercut international law's status as law.
He describes nation states' expressed attitude towards the rules of international law:
Yet what these rules require is thought and spoken of as obligatory; there is general pressure for conformity to the rules; claims and admissions are based on them and their breach is held to justify not only insistent demands for compensation, but reprisals and counter-measures. When the rules are disregarded, it is not on the footing that they are not binding; instead efforts are made to conceal the facts.
In what sense is international law binding?
When one makes the assertion that a rule is binding at international law, they are simply describing or predicting that the participants view the rule as generating obligation and duty, likely due to the rule's basis in treaty or customary international law.