More generally, the university gets to set whatever rules it wants, unless there is a law limiting what it can do. For example, in the US a government university (qua arm of government) cannot require you to have a particular religion or profess a particular creed, but a private one can. An employer could require you to be in your office from 9 to 5, even though it is legal to not be in an office from 9 to 5. The general principle is that the institution can set its own rules for operation, unless those rules violate the law.
Generally, the owner of property has the right to say how it can be used. It is quite common for universities to have variable policies regarding access to their books (e.g. "anybody can walk in and do things with the volumes, which are out in the open, except they have to have borrower's permission to take it out of the building" to "you cannot enter at all, and must have special permission to inspect the book -- nobody can take the books out of the building"). My experience is that UK libraries have a tighter rein on their holdings than do US universities (the sampling problem here is non-trivial). In some cases, the absolute control of the property owner is somewhat overridden by law, especially if the institution is a state-run university (not in the UK). Another possibility is that access to books is a contractual right (some kind of "terms of service" that you and they are bound to). Even in the case of US state universities, I cannot imagine a government having a law declaring the right to copy books to be such a fundamental right that a university cannot deny you that right.
The most likely scenario for enforcement is as a contractual matter. A complete analysis of the relationship between the individual and the institution would be way too broad for here, but here is a sample of the questions that could arise. What is the source of your right to access books? Perhaps you are employed as a faculty person: do you have a contract? Can you be fired for breaking rules: are there any limits on what acts can lead to firing? What procedures if any are specified for termination, and what avenue of internal appeal exist? The government would not overrule the institution's decision unless they didn't follow the contractually-governed procedures for termination.
Apart from the legal question, a perfectly coherent reason to prohibit scanning books whose copyright has expired is that the act of scanning them may damage the book.