Copyright And The Challenge of Large Language Models (Part 1)

by Lee Gesmer | Jul 1, 2024 | Copyright

“AI models are what’s known in computer science as black boxes: You can see what goes in and what comes out; what happens in between is a mystery.”

Trust but Verify: Peeking Inside the “Black Box” of Machine Learning

In December 2023, The New York Times filed a landmark lawsuit against OpenAI and Microsoft, alleging copyright infringement. This case, along with a number of similar cases filed against AI companies, brings to the forefront a fundamental challenge in applying traditional copyright law to a revolutionary technology: Large Language Models (LLMs). Perhaps more than any copyright case that precedes them, these cases grapple with a form of alleged infringement that defies conventional legal analysis.

This article is the first in a three-part series that will examine the copyright implications of the AI development process.

Disclaimer: I’m not a computer or AI scientist. However, neither are the judges and juries that will be asked to apply copyright law to this technology, or the legislators that may enact laws regulating it. It’s unlikely that they will go much beyond the level of detail I’ve used here.

What are Large Language Models (LLMs)?

Large Language Models, or LLMs, are gargantuan AI systems that use a vast corpus of training data and billions to trillions of parameters. They are designed to understand, generate, and manipulate human language. They learn patterns from the data, allowing them to perform a wide range of language tasks with remarkable fluency. Their inner workings are fundamentally different from any previous technology that has been the subject of copyright litigation, including traditional computer software.

LLMs typically use transformer-based neural networks: interconnected nodes organized into layers that can perform computations. The strengths of these connections—the influences that nodes have on another—are what is learned during training. These are called the model parameters or weights, and they are represented as numbers.

Here’s a simplified explanation of what happens when you use an AI like a large language model:

1. You input a prompt (your question or request).
2. The computer breaks down your prompt into smaller pieces called tokens. These can be words, parts of words, or even individual characters.
3. The AI processes these tokens through its neural network – imagine this like a complex web of connections. Each part of this network analyzes the tokens and figures out how they relate to each other.
4. As it processes, the AI predicts the probability distribution for the next token based on what it learned during its training.
5. The LLM selects tokens based on these probabilities and combines them to create a coherent response or output for you, the user.

The “large” in Large Language Models primarily refers to the enormous number of parameters these models contain – sometimes in the trillions. These parameters represent the model’s learned patterns and relationships, fine-tuned through exposure to massive amounts of text data. While larger and more diverse high-quality datasets can lead to better AI models, other factors such as model architecture, training techniques, and fine-tuning also play important roles in model performance.

How Do AI Companies Obtain Their Training Data?

AI companies employ various methods to acquire this data – 

– Web scraping and crawling. One of the primary methods of data acquisition is web scraping – the automated process of extracting data from websites. AI companies deploy sophisticated crawlers that systematically browse the internet, copying text from millions of web pages. This method allows for the collection of diverse, up-to-date information but raises questions about the use of copyrighted material without explicit permission.

– Partnerships and licensing agreements. Some companies enter into partnerships or licensing agreements to access high-quality, curated datasets. For instance, OpenAI has partnered with organizations like the Associated Press to use its news archives for training purposes.

– Public datasets and academic corpuses. Many LLMs are trained, at least in part, on publicly available datasets and academic text collections. These might include Project Gutenberg’s collection of public domain books, scientific paper repositories, or curated datasets like the Common Crawl corpus.

– User-generated content. Platforms that interact directly with users, such as ChatGPT, can potentially use the conversations and inputs from users to further train and refine their models. This practice raises privacy concerns and questions about the ownership of user-contributed data.

In the context of the New York Times lawsuit, it’s worth noting that OpenAI, like many AI companies, has not publicly disclosed the full extent of its training data sources. However, it’s widely believed that the company uses a combination of publicly available web content, licensed datasets, and partnerships to build its training corpus. The lawsuit alleges that this corpus includes copyrighted New York Times articles, obtained without permission or compensation.

The Training Process: How Machines “Learn” From Data

Once acquired, the raw data undergoes several processing steps before it can be used to train an LLM – 

– Data preprocessing and cleaning. The first step involves cleaning the raw data. This includes removing irrelevant information, correcting errors, and standardizing the format. This may involve stripping away HTML tags, removing advertisements, or filtering out low-quality content.

– Tokenization and encoding. Next, the text is broken down into smaller units called tokens. These might be words, parts of words, or even individual characters. Each token is then converted into a numerical representation that the AI can process. This step is crucial as it determines how the model will interpret and generate language.

During training, the LLM is exposed to this preprocessed data, learning to predict patterns and relationships between tokens. This is an iterative process where the model makes predictions, compares them to the actual data, and adjusts its internal parameters to improve accuracy. This process, known as “backpropagation,” is repeated billions of times across the entire dataset. In a large LLM this can take months, operating 24/7 on a massive system of graphics processing chips.

The Transformation From Text to Numbers

For purposes of copyright law, here’s the crux of the matter: the AI industry asserts that after this process, the original text no longer exists in any recognizable form within the LLM. The model becomes a vast sea of numbers, with no direct correspondence to the original text. If true, this transformation creates a fundamental challenge for copyright law – 

– No Side-by-Side Comparison: In traditional copyright cases, courts rely heavily on comparing the original work side-by-side with the allegedly infringing material. With LLMs, this is impossible. You can’t “read” an LLM or print it out for comparison.

– Black Box Nature: The internal workings of LLMs are often referred to as a “black box.” Even the developers may not fully understand how the model arrives at its outputs.

– Dynamic Generation: The AI industry claims that LLMs don’t store and retrieve text in a conventional database format; they generate it dynamically based on learned patterns. This means that any similarity to copyrighted material in the output is a result of statistical prediction, not direct copying.

– Distributed Information: The AI industry claims that Information from any single source is distributed across countless parameters in the model, making it impossible to isolate the influence of any particular work.

However, copyright owners do not concede that completed AI models (as distinct from the training data) are only abstracted statistical patterns of the training data. Rightsholders assert that LLMs do indeed retain the expressions of the original works on which they have been trained. There are studies showing the LLM models are able to regurgitate their training materials, and the New York Times lawsuit against OpenAI and Microsoft shows 100 examples of this. See also Concord Music Group v. Anthropic (alleging that song lyrics can be accessed verbatim or near-verbatim from Claude). Rightsholders argue that this could only occur if the models encode the expressive content of these works.

Copyright Implications

Assuming the AI developers’ explanation to be correct (if its not the infringement case against them is strong), AI technology creates unprecedented challenges for copyright law – 

– Proving Infringement: How can a plaintiff prove infringement when the allegedly infringing material can’t be directly observed or compared?

– Fair Use Analysis: Traditional fair use factors, such as the amount and substantiality of the portion used, become difficult to apply when the “portion used” is transformed beyond recognition.

– Substantial Similarity: The legal test of “substantial similarity” between works becomes almost meaningless in the context of LLMs.

– Expert Testimony: Courts will likely have to rely heavily on expert testimony to understand the technology, but even experts may struggle to definitively prove or disprove infringement.

For all of these reasons, to prove copyright infringement plaintiffs such as the New York Times may be limited to claiming copyright infringement based on the “intermediate” copies that are used in the training process and user-prompted output, rather than the LLM models themselves. 

Conclusion

The NYT v. OpenAI case and others raising the same issue highlight a fundamental mismatch between traditional copyright law and the reality of LLM technology and the AI industries’ fair use defense. The outcome of this case could reshape our understanding of copyright in the digital age, potentially requiring new legal tests and standards that can account for the invisible, transformed nature of information within AI systems.

Part 2 in this series will focus on the legal issues around the “input problem” of using copyrighted material for training. Part 3 will look at the “output problem” of AI-generated content that may copy or resemble copyrighted works, including what the AI industry calls “memorization.” As we’ll see, each of these issues presents its own unique challenges in the context of a technology that defies traditional legal analysis.

Continue reading Part 2 and Part 3 of this series.

Secondary Liability and Sony v. Cox

by Lee Gesmer | May 26, 2024 | Copyright

Copyright secondary liability can be difficult to wrap your head around. This judge-made copyright doctrine allows copyright owners to seek damages from organizations that do not themselves engage in copyright infringement, but rather facilitate the infringing behavior of others. Often the target of these cases are internet service providers, or “ISPs.”

Secondary liability has three separate prongs, “contributory” and “vicarious” infringement, and “inducement.” The third prong – inducement – is important but seen infrequently. For the elements of this doctrine see my article here. 

Here’s how I outlined the elements of contributory and vicarious liability when I was teaching CopyrightX:

These copyright rules were the key issue in the Fourth Circuit’s recent blockbuster decision in Sony v. Cox Communications (4th Cir. Feb. 20, 2024).

In a highly anticipated ruling the court reversed a $1 billion jury verdict against Cox for vicarious liability but affirmed the finding of contributory infringement. The decision is a significant development in the evolving landscape of ISP liability for copyright infringement.

Case Background

Cox Communications is a large telecommunications conglomerate based in Atlanta. In addition to providing cable television and phone services it acts as an internet service provider – an “ISP” – to millions of subscribers. 

The case began when Sony and a coalition of record labels and music publishers sued Cox, arguing that the ISP should be held secondarily liable for the infringing activities of its subscribers. The plaintiffs alleged that Cox users employed peer-to-peer file-sharing platforms to illegally download and share a vast trove of copyrighted music, and that Cox fell short in its efforts to control this rampant infringement.

A jury found Cox liable under both contributory and vicarious infringement theories, levying a jaw-dropping $1 billion in statutory damages – $99,830.29 for each of the 10,017 infringed works. Cox challenged the verdict on multiple fronts, contesting the sufficiency of the evidence and the reasonableness of the damages award.

The Fourth Circuit Opinion

On appeal, the Fourth Circuit dissected the two theories of secondary liability, arriving at divergent conclusions. The court sided with Cox on the issue of vicarious liability, finding that the plaintiffs failed to establish that Cox reaped a direct financial benefit from its subscribers’ infringing conduct. Central to this determination was Cox’s flat-fee pricing model, which remained constant irrespective of whether subscribers engaged in infringing or non-infringing activities. The mere fact that Cox opted not to terminate certain repeat infringers, ostensibly to maintain subscription revenue, was deemed insufficient to prove Cox directly profited from the infringement itself.

However, the court took a different stance on contributory infringement, upholding the jury’s finding that Cox materially contributed to known infringement on its network. The court was unconvinced by Cox’s assertions that general awareness of infringement was inadequate, or that a level of intent tantamount to aiding and abetting was necessary for liability to attach. Instead, the court articulated that supplying a service with the knowledge that the recipient is highly likely to exploit it for infringing purposes meets the threshold for contributory liability.

Given the lack of differentiation between the two liability theories in the jury’s damages award, coupled with the potential influence of the now-overturned vicarious liability finding on the damages calculation, the court vacated the entire award. The case now returns to the lower court for a new trial, solely to determine the appropriate measure of statutory damages for contributory infringement.

Relationship to the DMCA

This article’s header graphic illustrates the relationship between the secondary liability doctrines and the protection of the Digital Millennium Copyright Act (DMCA), Section 512(c) of the Copyright Act. As the graphic reflects, all three theories of secondary liability lie outside the DMCA’s safe harbor protection for third-party copyright infringement. The DMCA requires that a defendant satisfy multiple safe harbor conditions (See my 2017 article – Mavrix v. LiveJournal: The Incredible Shrinking DMC for more on this). If a plaintiff can establish the elements of any one of the three theories of secondary liability the defendant will violate one or more safe harbor conditions and lose DMCA protection.

Implications

The court’s decision signals a notable shift in the contours of vicarious liability for ISPs in the context of copyright infringement. By requiring a causal nexus between the defendant’s financial gain and the infringing acts themselves, the court has raised the bar for plaintiffs seeking to prevail on this theory.

The ruling underscores that simply profiting from a service that may be used for both infringing and non-infringing ends is insufficient; instead, plaintiffs must demonstrate a more direct and meaningful link between the ISP’s revenue and the specific acts of infringement. This might entail evidence of premium fees for access to infringing content or a discernible correlation between the volume of infringement and subscriber growth or retention.

While Cox may take solace in the reversal of the $1 billion vicarious liability verdict, the specter of substantial contributory infringement damages looms large as the case heads back for a retrial.

For ISPs, the ruling serves as a warning to reevaluate and fortify their repeat infringer policies, ensuring they go beyond cosmetic compliance with the DMCA’s safe harbor provisions. Proactive monitoring, prompt responsiveness to specific infringement notices, and decisive action against recalcitrant offenders will be key to mitigating liability risks.

On the other side of the equation, copyright holders may need to recalibrate their enforcement strategies, recognizing the heightened evidentiary burden for establishing vicarious liability. While the contributory infringement pathway remains viable, particularly against ISPs that display willful blindness or tacit encouragement of infringement, the Sony v. Cox decision underscores the importance of marshaling compelling evidence of direct financial benefit to support vicarious liability claims.

As this case enters its next phase, the copyright and technology communities will be focused on the outcome of the damages retrial. Regardless of the ultimate figure, the Fourth Circuit’s decision has already left a mark on the evolving landscape of online copyright enforcement.

Header image is published under the Creative Commons Attribution 4.0 License.

Supreme Court Allows Copyright Damages Beyond 3 Years – But Leaves Key Question Open

by Lee Gesmer | May 17, 2024 | Copyright

Many aspects of copyright law are obscure and surprising, even to lawyers familiar with copyright’s peculiarities. An example of this is copyright law’s three-year statute of limitations. 

The Copyright Act states that “no civil action shall be maintained under the provisions of this title unless it is commenced within three years after the claim accrued.” 17 U. S. C. §507(b). In the world of copyright practitioners this is understood to mean that so long as a copyright remains in effect and infringements continue, an owner’s rights are not barred by the statute of limitations. However, they may be limited to damages that accrued in the three years before the owner files suit. This is described variously as a “three-year look-back,” a “rolling limitations period” or the “separate-accrual rule.”

This is what allowed Randy Wolfe’s estate to sue Led Zeppelin in 2014 for an alleged infringement that began in 1971.

However, there is a nuance to this doctrine – what if the copyright owner isn’t aware of the infringement? Is the owner still limited to damages accrued in the three years before he files suit? 

That is the scenario the Supreme Court addressed in Warner Chappell Music, Inc. v. Nealy (May 9, 2024). 

Background Facts

Songwriter Sherman Nealy sued Warner Chappell in 2018 for infringing his music copyrights going back to 2008. Warner responded that under the “three year look-back” rule Nealy’s damages were limited to three years before he filed suit. Nealy argued that his damages period should extend back to 2008, since his claims were timely under the “discovery rule” – he was in prison during much of this period and only learned of the infringements in 2016.

Nealy lost on this issue in the district court, which limited his damages to the infringer’s profits during the 3 years before he filed suit. The 11th Circuit reversed, holding that Nealy could recover damages beyond 3 years if his claims were timely – meaning that the case was filed within three years of when Nealy discovered the infringement. 

The Supreme Court Decision

The Supreme Court affirmed the 11th Circuit and resolved a circuit split, holding: 

1 – The Copyright Act’s 3-year statute of limitations governs when a claim must be filed, not how far back damages can go.

2 – If a claim is timely, the plaintiff can recover damages for all infringements, even those occurring more than 3 years before suit. The Copyright Act places no separate time limit on damages.

However, lurking within this ruling is another copyright law doctrine that the Court did not address that could render its ruling in Nealy moot – that is the proper application of the “discovery rule” under the Copyright Act. Under the discovery rule a claim accrues when “the plaintiff discovers, or with due diligence should have discovered” the infringement. (Nealy, Slip Op. p. 2). Competing with this is the less liberal “occurrence” rule, which holds that, in the absence of fraud or concealment, the clock starts running when the infringement occurs. Under the discovery rule Nealy would be able to recover damages back to 2008. Under the occurrence rule his damages would be limited to the three years before he filed suit, since he does not allege fraud or concealment. 

However, the question of which rule applies under the Copyright Act has never been addressed by the Supreme Court, and is itself the subject of a circuit split. The Court assumed, without deciding and solely for purposes of deciding the issue before it, that the discovery rule does apply to copyright claims. If the discovery rule applies Nealy has a claim to retroactive damages beyond three years. If it does not, Nealy’s damages would be limited to the three years before he filed suit.

Justice Gorsuch, joined by Justices Thomas and Alito, focused on this in his dissent, arguing the Court should not have decided the issue when the “discovery vs. occurrence” issue has not been addressed:

The Court discusses how a discovery rule of accrual should operate under the Copyright Act. But in doing so it sidesteps the logically antecedent question whether the Act has room for such a rule. Rather than address that question, the Court takes care to emphasize that its resolution must await a future case. The trouble is, the Act almost certainly does not tolerate a discovery rule. And that fact promises soon enough to make anything we might say today about the rule’s operational details a dead letter.

Clearly, in the view of at least three justices, if and when the discovery vs. occurrence rule issue comes before the Court it could decide against the discovery rule in copyright cases, rendering its decision on damages in the Nealy case, and cases like it, moot.

State of the Law Today

What does this all boil down to? Here are the rules as they exist today –

– A copyright owner has been aware of an infringing musical work for 20 years. She finally sues the infringer. Her damages are limited by the three year damages bar. They may be limited even further based on the laches doctrine.

– A copyright owner has been meditating alone in a cave in Tibet for 20 years. She’s had no access to information from the outside world. Upon her return she discovers that someone has been infringing her literary work for the last 20 years. Depending on whether the federal circuit applies the discovery or the occurrence rule, she may recover damages for the entire 20 period, or just the preceding three years. Her lawyers should do some careful forum shopping.

– A copyright owner discovers someone has secretly been infringing her copyright in computer source code for 20 years. The source code was non-public, and therefore the infringement was concealed. She may recover damages for the full 20 year period.

Implications

The decision is a win for copyright plaintiffs, allowing them to reach back and get damages beyond 3 years – assuming their claims are timely and they are in a circuit that apples the discovery rule. But the Court left the door open to decide the more important question of whether the discovery rule applies to the Copyright Act’s statute of limitations at all. If not, the window for both filing claims and recovering damages will shrink. When this issue will reach the Supreme Court is uncertain. However, the Court has the opportunity to take it up as soon as next term. See Hearst Newspapers, LLC v. Martinelli, No. 23-474 (U.S. petition for cert. filed Nov. 2, 2023). In the meantime, the outer boundary of damages is limited only by the discovery rule (if it apples), not any separate damages bar. Plaintiffs with older claims should take note, as should potential defendants doing due diligence on liability exposure.

Update: On May 20, 2024, the Supreme Court of the United States denied the petition for certiorari in Hearst Newspapers, L.L.C. v. Martinelli, thereby declining to decide whether the discovery rule applies to copyright infringement claims and leaving the rule intact. 

Header image attribution: Resource by Nick Youngson CC BY-SA 3.0 Pix4free