The short answer to your question is yes: the usual rule is that the losing side “pays the costs” of the winning side. In legal jargon, “costs follow the event.”
However, costs law is its own complex, specialised area of Australian law, often taught as part of a civil procedure course using Dal Pont’s textbook. There are three important caveats to the general rule.
First, costs are in the court’s discretion. The court may decline to order costs in favour of the winning party, or even order costs against the winning party, in appropriate cases. This 2017 article in Precedent summarises some cases in New South Wales where the usual rule was not applied.
Second, the general rule as to costs is part of general civil procedure – the default rules of court that apply to civil litigation in the Supreme and Federal Courts. While similar rules apply in the High Court, general civil procedure does not apply in criminal law, family law, or litigation in small claims tribunals. For example, section 117 of the Family Law Act provides that, subject to exceptions, “each party to proceedings under this Act shall bear his or her own costs.” The NSW Civil and Administrative Tribunal can order costs “only if it is satisfied that there are special circumstances warranting an award of costs.” The average person involved in a minor legal case is unlikely to have the benefit of the rule that costs follow the event.
Third, the usual order as to costs does not normally result in the successful party getting their costs fully reimbursed. As pointed out in the comments, there is a distinction between disbursements (fixed costs such as court fees, which are usually recoverable in full) and professional fees. The general rule is that professional fees are “taxed on a party-party basis,” meaning that the losing party is ordered to pay a standard amount assessed by the court, rather than the full amount actually paid by the winning party to their lawyers. In some cases the shortfall is more than 50%. In exceptional cases, the full amount may be ordered (“costs on an indemnity basis”). The details are highly court and fact specific. And ultimately, a costs judgment must be enforced like any other civil judgment – you won’t get paid if the debtor has no assets.