Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews

Summary AI-generated

TL;DR

Researchers have developed a new method called Sem-Detect that identifies AI-generated peer reviews by analyzing the uniqueness of the ideas and claims they express, rather than just looking at their writing style.

Problem

As AI writing tools become more common, it is increasingly difficult to tell if scientific peer reviews are written by humans or generated by AI. This threatens the integrity of the scientific review process, especially since simple style-based detectors can easily misclassify human reviews that were merely polished using AI.

Method

The proposed method, Sem-Detect, analyzes both the writing style and the actual claims made in a review. It compares a target review against several AI-generated reviews of the same paper, exploiting the fact that different AI models tend to make the same predictable points, whereas human reviewers offer more diverse and unique insights.

Results

Tested on over 20,000 reviews from major AI conferences, Sem-Detect improved detection accuracy over the strongest existing baseline by 25.5% at a very low false positive rate. Additionally, the method successfully distinguished fully AI-generated reviews from human-written ones that were simply polished by AI, misclassifying fewer than 3.5% of these polished reviews.

Takeaways

First, AI models tend to converge on the same predictable arguments when reviewing a paper, while humans provide unique perspectives. Second, analyzing the semantic meaning of claims is far more effective for detection than looking at writing style alone. Finally, human reviews polished by AI still retain distinct human semantic signals, allowing them to be correctly identified as human-authored.

For industry

For organizations managing quality control or content moderation, this research offers a robust way to detect automated content by looking at the substance of the arguments rather than just the vocabulary. It provides a reliable tool to verify human effort and expertise in professional evaluation processes, even when contributors use AI assistants to refine their writing.

Why it matters

This work helps safeguard the credibility of scientific publishing and peer-review systems against automated spam and low-effort AI generation. Beyond academia, the underlying approach of semantic claim analysis can be applied to verify authenticity in other critical domains, such as legal analysis, financial auditing, and journalism.