I was swimming in a college owned pool with my girlfriend and We were the only ones around. The rules of the pool do not prohibit physical contact and the only thing my girlfriend and I where doing was holding hands and then the lifeguard told us that we cannot touch each other. We are both over 18 and I feel that the lifeguard does not have the authority to force us to not touch each other. Vermont state law is wonderfully ambiguous on the subject of what authority lifeguards have, so I am curious. What legal authority does a lifeguard have over adults in a public pool?
I was swimming in a college owned pool...
That's one of the important details; you're either college students or visitors on a campus, and you are bound by the student conduct code or other safety laws set in place by the college that dictate behavior on the college campus. Check with the college about the rules that involved the pool. Campus laws and regulations can differ greatly, depending on if the college is private or a public institution.
The other detail is that the lifeguard has been given authority to control behavior in the pool as an agent of the owner of the facility (the college) with all authority that the college has to regulate the pool. Be using the pool, you have entered into an agreement with the owners of the pool (the college) and that the lifeguard has authority in that area. Look around for a sign near the pool or in the locker rooms that outlines the rules and regulations of using the pool.
They don't have much legal authority as lifeguards. However, the pool does have legal authority to refuse anyone access to their facility. They probably have a list of rules posted somewhere (entrance?) and they are fully within their rights to kick you out if they feel you break them. The lifeguard can make that decision to kick you out.
However, this doesn't mean that the lifeguard is a tyrant and you must dance to his tune. From your description, the lifeguard sounds like he was being overzealous, especially since nobody else was around. Firstly, the lifeguard may be wrong on his understanding of the pool policies. You can (politely) inform him that you disagree with the decision and would like to speak with a supervisor. If no supervisor is available, you will probably have to leave and then submit your complaint afterwards, which won't provide much restitution to you but at least the lifeguard may get a talking to about hassling visitors. Or perhaps the pool will back him up.
Even if the lifeguard is actually right about enforcing a rule, these organizations are fairly sensitive to public opinion. You can publicly complain about this on social media or to other parts of the college who have authority over the pool. Obviously one complaint from you won't do much, but maybe others have also complained and yours tips the balance. In any case, if enough people complain, the pool may very well change its policy.