The first way to look at the question is whether the output of the program is “substantially similar” to that of a protected work. A copyright infringement claim does not have to show video of a person copying the protected work, the plaintiff simply has to prove that they own the right to the original work and that the infringing work is similar to it (there are various adjectives used such as ‘probative’, ‘striking’, ‘substantial’, and their definitions and application differ according to jurisdiction). The proof requirement is not that you have to prove that the infringing work is exactly verbatim identical to the original, there is just a level of “close enough”, since the prohibited act is copying, not exact copying. If the output looks like it was copied, the defendant is likely to be found to have infringed, unless you can somehow prove that the AI output only coincidentally recreated the original. You claim that “The fundamental behavior of a generative LLM is that it produces a token at a time, rather than copying from somewhere else”, which I don’t dispute, but this is not a self-evident truth, it is only a part of a defense that has to be proven. This is a substantial risk (just saying it over and over again will not persuade a juror, when they are also being told the contrary).
As I suspect you know, you can pair a text with a word-frequency table and write a mathematical algorithm that exactly reduces any original text to a “model”, which can then exactly reproduce an original text not by copying from the clear text, but via the mathematical model. If that is what the program does, there is no question that this is illegal creation of a derivative work (in the same way that it is infringement to read a novel and start translating it into French, without the author’s permission). This is another legal risk, one that is harder to disprove, that you created an unauthoriized derivative work.
There are word-frequency tables ‘out there’ which are created by copying (infringing) protected works and reducing individual words to counts, e.g. “the” occurs 129,465,324 times. Nobody holds copyright on a single word, or two words, or three words… there is a line there, the law has not said what the bright line is (in terms of numbers of consecutive words). But one may benefit from a fair use argument, which is why at least so far nobody has been sued for creating a word frequency table, even though the act of creating the table requires copying of protected text (unless one is doing a table of frequencies for 19th century English, or 20th century laws passed by Congress). LLM works not just in terms of individual words in isolation, it encodes context, which if long enough could be very much like just the approach of substituting frequency-rank for the actual word.
A possible mitigating factor would be the “there are only so many notes” defense that sometimes figures into music infringement. There are (non-trivial) natural limits on musical compositions (fewer atoms that can be combined), because of which one may independently create a work that strongly resembles another protected work. For actual literary texts, the coincidence argument is easy to disprove, but computer code is much closer to musical compositions in that for example there are only so many ways that one can reasonably implement a bubble sort.
In theory, the creators of the language model could be found liable for copyright infringement. The main risk, in my opinion, is that the law (in the US) prohibits creation of substantially-similar text, and a user might well end up in court because they relied on a tool that can produce extremely similar works. The user does not have access to the guts of Open AI or other such LLM software, and to mount a successful defense, they will need non-trivial assistance from the creators of the model to prove that this isn’t just plain old copying.
