Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews
Summary AI-generated
- TL;DR
- A new method called Sem-Detect identifies AI-generated peer reviews by analyzing the uniqueness of their ideas rather than just their writing style.
- Problem
- As researchers increasingly use AI to write or edit scientific peer reviews, it becomes crucial to distinguish genuine human critique from fully automated text to maintain scientific integrity. Traditional detection methods rely only on writing style, which fails when humans use AI to polish their own thoughts or when AI writing mimics human style.
- Method
- The proposed method, Sem-Detect, analyzes both textual style and the actual claims or ideas expressed in a review. It compares a target review against several AI-generated reviews of the same paper, exploiting the fact that different AI models usually generate similar points, whereas human reviewers offer more diverse and unique perspectives.
- Results
- Evaluated on over 20,000 peer reviews from ICLR and NeurIPS, Sem-Detect improved detection accuracy by 25.5% over the strongest baseline at a very low false positive rate. Additionally, the method successfully avoids misclassifying human reviews that were merely polished by AI, with fewer than 3.5% of these LLM-refined reviews flagged as fully AI-generated.
- Takeaways
- Analyzing the semantic meaning and diversity of ideas is more effective for AI detection than looking at writing style alone. Furthermore, human-written reviews polished by AI still retain distinct human semantic signals, meaning we can allow AI assistance without losing human authorship.
- For industry
- For organizations managing large-scale text evaluations or submissions, this research provides a way to verify authentic human input even when AI tools are used for editing. It shows that analyzing the diversity of ideas across multiple automated outputs can reliably separate purely automated content from human-driven insights.
- Why it matters
- This work helps safeguard the integrity of the scientific peer-review process by ensuring that academic evaluations reflect genuine human judgment. Beyond academia, the underlying approach of semantic-level detection can be applied to other domains like journalism, education, and content moderation to verify authentic human authorship.
Abstract
How can we distinguish whether a peer review was written by a human or generated by an AI model? We argue that, in this setting, authorship should not be attributed solely from the textual features of a review, but also from the ideas, judgments, and claims it expresses. To this end, we propose Sem-Detect, an authorship detection method for peer reviews that operationalizes this principle by combining textual features with claim-level semantic analysis. Sem-Detect compares a target review against multiple AI-generated reviews of the same paper, leveraging the observation that different AI models tend to converge on similar points, while human reviewers introduce more unique and diverse ones. As a result, Sem-Detect is able to distinguish fully AI reviews from authentic human-written ones, including those that have been refined using an LLM but still reflect human judgment. Across a dataset of over 20,000 peer reviews from ICLR and NeurIPS conferences, Sem-Detect improves over the strongest baseline by 25.5% in TPR@0.1% FPR in the binary setting. Moreover, in the three-class scenario, we empirically show that LLM refinement preserves the semantic signals of human reviews, which remain distinct from the patterns exhibited by fully AI-generated text; as a result, fewer than 3.5% of LLM-refined human reviews are misclassified as AI-generated.
