First, the relationship between the controller and processor is described in great details in GDPR Chapter 4. It is recommended reading.
TL;DR: The controller is the one who calls the shots (i.e. what personal data to process, what means to use to process it, the purpose of processing, and the legal basis for processing). This is usually the owner of information system. The processor is the the entity that actually do the processing. If their relationship is not described in a legal document known as a DPA, both the controller and processor risks fines.
I think you have the right idea about you having different roles in different contexts, and also that you work for clients, you are processor. This also means that the client is the controller.
How to regulate this relationship is described in GDPR Article 28 (3). Its start out by saying:
Processing by a processor shall be governed by a contract or other legal act under Union or Member State law, that is binding on the processor with regard to the controller and that sets out the subject-matter and duration of the processing, the nature and purpose of the processing, the type of personal data and categories of data subjects and the obligations and rights of the controller.
It then goes on to describe what stipulations the contract must contain.
This contract is usually called a DPA. The initials may stand for "Data Processing Agreement" or "Data Protection Addendum".
Basically, if you personal process data on behalf of clients, you need to have this place. Both the controller and processor will be fined by the supervisory authority finds out that you are processing personal data without having such a contract in place.