Detecting ChatGPT-generated essays in a large-scale writing assessment: Is there a bias against non-native English speakers?

A recent study (Liang et al., 2023) showed that publicly available detectors of AI-generated texts are more likely to misclassify essays written by non-native English speakers than those written by native English speakers, leading to significant fairness concerns about using these detectors. Using carefully sampled large-scale data from the Graduate Record Examinations (GRE) writing assessment, we developed multiple detectors of ChatGPT-generated essays based on linguistic features from the ETS e-rater engine and text perplexity features, and investigated their performance and potential bias. Results show that our detectors not only achieve near-perfect detection accuracy, but also show no evidence of bias that disadvantages non-native English speakers.