Section 3 of the Party Wall etc. Act 1996 specifies that:
a building owner shall serve on any adjoining owner a notice
when carrying out any of the repairs listed in section 2 of the Act.
This implies that any work that is not in the list in section 2 does not require any notice to be served.
This is echoed on the Government website, which gives a summary of work that requires notice, and provides examples of minor work which does not.
The official explanatory booklet on the Act goes into a little more detail, explaining that (on p11):
the key point is whether your planned work might have any possible consequences for the structural strength and support functions of the party wall as a whole, or cause damage to the Adjoining Owner's side of the wall. If you are in doubt about whether your planned work requires a notice you might wish to seek advice from a qualified building professional.
However, as the OP points out in a comment below, this is not quite what the Act says. For example, the Government website mentions that "drilling to put up shelves" is OK, but section 2(2)(f) states that notice is required "to cut into a party structure for any purpose".
Bearing in mind that I have no legal training, I would suggest that the answer is in section 4, which covers counter notices. An adjoining owner may require the building owner to carry out certain additional works "as may reasonably be required for the convenience of the adjoining owner" - but not if it were to "cause unnecessary inconvenience to [the building owner]".
It could be argued that having to carry out any of the works listed in section 4(1)(a), just to drill some holes which will have no impact on the structure at all, would not be reasonable, and would cause unnecessary inconvenience.
Ultimately, it would be for a surveyor to resolve any dispute (or, on appeal, a judge). My interpretation, therefore, is that the official advice is making a reasoned assessment as to what a surveyor might consider "minor" in this context.
(A more detailed discussion of the workings of the Act can be found in this paper. One thing that it points out is that the section of the Act relating to party walls is copied almost verbatim from the London Building Acts (Amendment) Act 1939, and hence that the procedures relating to party walls are well established.)