The costs order is in the court’s discretion and will depend on all of the circumstances. Rule 44.3 of the Civil Procedure Rules relevantly provides:
(4) In deciding what order (if any) to make about costs, the court must have regard to all the circumstances, including …
(b) whether a party has succeeded on part of his case, even if he has not been wholly successful; and
(c) any payment into court or admissible offer to settle made by a party which is drawn to the court’s attention …
(5) … (b) whether it was reasonable for a party to raise, pursue or contest a particular allegation or issue;
(c) the manner in which a party has pursued or defended his case or a particular allegation or issue; and
(d) whether a claimant who has succeeded in his claim, in whole or in part, exaggerated his claim.
(6) The orders which the court may make under this rule include an order that a party must pay—
(a) a proportion of another party’s costs …
(f) costs relating only to a distinct part of the proceedings …
In the circumstances you describe, the court may well order that Bob pays the costs of Alice’s claim, and Alice pays the costs of Bob’s counter-claim. Whether any net amount is payable would depend on which part of the case occupied the most court time. But such an order might be inappropriate if the case was run in such a way that it was difficult to allocate costs to one side or another.
As noted in CPR 44.3(4)(c), settlement offers are also relevant. In your second example, suppose that Bob agreed before the hearing that he was liable to pay the net amount of £40, but Alice demanded full payment. This factor would count strongly in favour of ordering Alice to pay Bob’s costs even though the final outcome was a judgment against him.