Presenter
Ashley Kulp
Document Type
Poster
Publication Date
2026
Abstract
AI-generated text is becoming increasingly common in educational and research settings, leading to a greater reliance on AI detection tools. While prior research has assessed the performance of AI detection tools, few studies have examined their performance at determining humanized and/or hybridized text This study intends to evaluate the accuracy of five publicly accessible AI detectors in distinguishing between human-written, AI-generated, humanized, and hybrid academic text. Discussion sections from 200 Alzheimer's disease research articles were collected. Of these articles, four versions were created: human only, AI only, humanoid, and hybrid (30% rewrite), resulting in 800 total samples. In the primary analysis, detectors were presented with all human-only and AI-only text, while the secondary analysis used a randomized subset of humanoid and hybrid texts. All detectors evaluated the same final set of 500 samples. Detector outputs were recorded as numeric scores and performance was assessed using area under the receiver operating characteristic curve (AUROC). The study's findings emphasize the need for further development of AI detection tools to properly distinguish AI from human writing and to increase their reliability. Further, given the rapid development of LLMs, detectors should be continuously revised to keep up with more advanced text transformations.
Faculty Mentor
Samantha Rosenthal, Ph.D., M.P.H.
Academic Discipline
College of Arts & Sciences
Repository Citation
Kulp, Ashley; McKinnon, Ruth; Jacob, Steve; Bruno, Samantha; Kasprzak, Marygrace; Silva, Shayanne; Pereira, Hannah; and Villa, Miriam Liz, "AI Detection in Academic Writing: Evaluating Performance" (2026). Student Research Design & Innovation Symposium. 340.
https://scholarsarchive.jwu.edu/innov_symposium/340
