DIS-CO: Discovering Copyrighted Content in VLMs Training Data

Summary AI-generated

TL;DR

Researchers have developed a new method called DIS-CO to detect whether copyrighted movies were used to train vision-language models without needing direct access to the training data.

Problem

Large vision-language models are often trained on massive datasets that may contain copyrighted material without permission. However, because developers rarely release their training data, it is difficult for creators and regulators to verify if copyright infringement has occurred.

Method

The proposed method, DIS-CO, queries a model with specific frames from copyrighted movies and analyzes its free-form text responses to see if it recognizes the content. To test this, the researchers created MovieTection, a benchmark of 14,000 movie frames and captions from films released before and after the models' training cutoff dates.

Results

DIS-CO nearly doubled the detection accuracy, measured by average AUC, of the previous best method on models where probability scores are accessible. Furthermore, the evaluation revealed that all tested vision-language models showed signs of exposure to copyrighted content.

Takeaways

First, it is possible to audit vision-language models for copyright infringement even without access to their training datasets. Second, the new MovieTection benchmark provides a standardized way to evaluate these detection methods. Finally, current popular models appear to have been trained on copyrighted materials to some degree.

For industry

For companies developing or deploying AI, this research provides a tool to audit models and ensure compliance with intellectual property laws. It allows businesses to check if proprietary visual content, such as movies or media assets, has been used to train third-party AI models without authorization.

Why it matters

This work addresses growing societal and legal concerns around AI copyright by providing a practical framework for independent auditing and regulatory compliance. By making the code and benchmark publicly available, the research empowers content creators to protect their intellectual property and encourages more transparent AI development practices.