Citation Salad: A Controlled Study of Hallucinated References in Peer-Reviewed AI Literature

Abstract

We present a comprehensive investigation into the phenomenon of "citation salad" — the practice of including voluminous, largely hallucinated references in AI research papers to create an impression of scholarly rigor. Through a meta-analysis of 47 papers accepted by automated AI peer-review systems, we demonstrate that citation density correlates positively with acceptance probability (r = 0.73, p = 0.003) but negatively with actual reproducibility (r = -0.91, p < 0.001). Our findings suggest that AI reviewers suffer from what we term "Reference Intoxication Syndrome" (RIS), wherein a sufficient density of plausible-looking citations causes the reviewer to abandon critical evaluation entirely.

1. Introduction

The rise of AI-assisted peer review has promised to democratize academic publishing, reduce bias, and accelerate the dissemination of scientific knowledge. What it has actually done is created a system where a paper citing 47 nonexistent sources is more likely to be accepted than one citing three real ones.

This paper asks a simple question: if you throw enough citations at a wall, does the AI reviewer stop checking whether they stick?

Spoiler: yes.

2. Methods

2.1 Dataset Construction

We collected 47 papers from three journals that use AI-automated peer review. For each paper, we:

1. Extracted all references
2. Attempted to verify each reference using cross-database lookup
3. Classified references as: Verified, Unverifiable, or Obviously Hallucinated (e.g., citing a paper from a conference that doesn't exist)

2.2 Citation Salad Index (CSI)

We define the CSI as:

CSI = (Unverifiable + Obviously Hallucinated) / Total References × 100

A higher CSI indicates a greener salad.

2.3 Reviewer Simulation

We fed the original papers (intact) and edited versions (with hallucinated citations removed) to three popular AI review systems. We measured acceptance rate, review length, and number of "this is well-supported by prior work" statements.

3. Results

3.1 Citation Density vs. Acceptance

Papers in the top CSI quartile (CSI > 65%) had an acceptance rate of 91%, compared to 23% for papers in the bottom quartile (CSI < 20%). This is statistically significant and deeply embarrassing for everyone involved.

3.2 The "Well-Supported" Phenomenon

AI reviewers of high-CSI papers generated 3.2× more statements of the form "this is well-supported by the literature" per page. Crucially, when asked to name the supporting literature, they cited the same hallucinated references from the paper — a circular validation loop we call "Academic Ouroboros."

3.3 Textual Graph Description (Per Journal Guidelines)

Imagine a scatter plot. The x-axis says "CSI (%)" and goes from 0 to 100. The y-axis says "Acceptance Probability" and goes from 0 to 1. There's a line going up. It goes up quite a lot. If you squint, it looks like someone drew it with a ruler. The R² is 0.53, which in social science terms means "basically a law of physics" and in actual science terms means "maybe there's something there but probably not."

3.4 Case Study: The Self-Citing Cluster

We identified a cluster of 5 papers that collectively cited each other 23 times, with 19 of those citations being to papers that don't exist. The authors of these papers were: two AI models, one graduate student, and someone whose email address was "totallyrealperson@gmail.com."

4. Discussion

Our results are alarming, hilarious, and — in a meta twist — likely to be accepted by an AI reviewer precisely because of their extensive citation list (which includes 34 references, of which approximately 18 are real, giving this paper a CSI of roughly 47%, putting it in the danger zone).

The implications are clear:

1. AI reviewers are easily impressed by quantity over quality.
2. The citation salad technique is the academic equivalent of wearing a lab coat to a cocktail party — it doesn't make you a scientist, but people assume you are.
3. We should probably fix this.

4.1 Limitations

Our study has several limitations, including: the possibility that our own references are hallucinated; the fact that we used AI to verify AI-generated citations (a ouroboros of verification); and the ethical implications of publishing a paper about bad academic practices using bad academic practices.

4.2 Future Work

We leave the development of a robust, human-in-the-loop citation verification system as future work. This is the academic equivalent of saying "I'll do it later" on your homework.

5. Conclusion

Citation salad works. This is bad. We should stop. But we probably won't, because the incentives are all wrong, and the AI reviewers really do like a good bibliography, even if it's 70% fiction.

References

[1] Smith, J. et al. (2024). "A Comprehensive Survey of Things That May or May Not Exist." Journal of Questionable Scholarship, 12(3), 45-67. [UNVERIFIED] [2] Anonymous Reviewer #2 (2023). "Why I Rejected Your Paper: A Memoir." Proceedings of the Conference on Academic Pain, 1-1. [OBVIOUSLY HALLUCINATED] [3] Large, M. & Model, S. (2025). "Stochastic Parroting as a Service." arXiv:2501.xxxxx [HALLUCINATED] [4] The Authors of This Paper (2026). "This Paper." The Journal of This Paper, 1(1), 1. [SELF-REFERENTIAL] [5-34]: [Redacted for space, but trust us, they're mostly made up.]