The law on trespass to land is not completely straightforward. Trespass to land can include the unlawful presence of a person, which is what you are interested in, along with things that you may not think of as trespass, such as leaving objects on your property.
You do not need to prove any damages in order to bring a claim: trespass is actionable per se in English law. However, if there is no actual harm, the damages awarded may be nominal. The courts may also grant an injunction to prevent the trespasser from continuing to trespass or returning.
The trespasser may have several defences in law, such as a licence to enter (in other words, permission), necessity, or exercising a legal right. An obvious example of licence to enter would be a shop: the owner has given (implied) permission for you to enter for the purposes of shopping, but may withdraw that licence at any point and ask you to leave.
Dale M has elaborated on some of the legal rights that are a defence to trespass, such as open access land and public rights of way.
There are several remedies to trespass to land in addition to claiming damages or an injunction. In Halsbury's Laws of England, vol. 97 (2015), para 588, it is stated that:
If a trespasser peaceably enters or is on land, the person who is in, or entitled to, possession may request him to leave, and if he refuses to leave may remove him from the land, using no more force than is reasonably necessary.
You might think of a nightclub bouncer ejecting an unwanted and unruly customer as an example of this: the bouncers act as agents for the person in possession and may use reasonable force.
In certain circumstances trespass may also be a criminal offence. For example, trespassing in a residential property may be in violation of the Criminal Law Act 1977, s. 7(1):
(1) Subject to the following provisions of this section and to section 12A(9) below, any person who is on any premises as a trespasser after having entered as such is guilty of an offence if he fails to leave those premises on being required to do so by or on behalf of—
(a) a displaced residential occupier of the premises; or
(b) an individual who is a protected intending occupier of the premises.
Trespass to land therefore has both elements in criminal law, and in tort, and in some cases trespass is a crime. Dale M also rightly points out burglary, and in response to your question about the garden, I do not believe that this would be considered a building. The CPS reference B v Leathley, where the test of a building is stated as "a structure of considerable size and intended to be permanent or at least to endure for a considerable period".