There are multiple cases in the court system challenging the legality of training AI models on copyrighted work. A recent one seems particularly damming if true, with OpenAI accused of using copyright infringing datasets such as Library Genesis in their training.
The model weights encode the training data. This is a large number of numbers that represents the weight assigned to each node in the model and encodes the training process. As I understand it is is considered to be data rather than creative expression. Note Llama (not really open source) and OpenLLaMA (Apache 2.0) only differ by these weights, and therefore the training that went into generating them. These model weights are uncopyrightable and unpatentable, according to this answer and supported by the federal register regarding patents and the Copyright Office regarding copyright.
Suppose once all these court cases have gone through it is determined that the only breach of copyright was the copying that occurred during the training process. The companies that performed this training would clearly be in breach of copyright. What about a third party, for example a developer who used the open source code/weights to create a non-infringing product? What about an individual user who used that product? What effect does a copyright breach in the creation of data have no users of that data?