Abstract Physiologically relevant drought stress is difficult to apply consistently, and the heterogeneity in experimental design, growth conditions, and sampling schemes makes it challenging to compare water deficit studies in plants. Here, we reanalyzed hundreds of drought gene expression experiments across diverse model and crop species and quantified the variability across studies. We found that drought studies are surprisingly incomparable, even when accounting for differences in genotype, environment, drought severity, and method of drying. Many studies, including most Arabidopsis (Arabidopsis thaliana) work, lack high-quality phenotypic and physiological datasets to accompany gene expression, making it challenging to assess the severity or consistency of water deficit stress events. To help address this, we developed supervised learning classifiers that can distinguish RNAseq samples that likely experienced drought stress. While not a substitute for direct measurements, these classifiers may aid in interpreting existing datasets and assessing drought severity in studies lacking physiological metadata. Together, our analyses highlight the importance of paired physiological data to quantify stress severity for reproducibility and future data analyses.