O-GlcNAcylation is a non-canonical form of protein glycosylation that occurs in nuclear, cytoplasmic, and mitochondrial proteins among all multicellular eukaryotes. There are only two enzymes that regulate this post-translational modification, one of which is O-GlcNAcase, a glycoside hydrolase that catalyzes the hydrolytic cleavage of O-GlcNAc from protein substrates. Related studies have shown that the reduction of O-GlcNAc levels is closely related to Alzheimer's disease, which is maintained by reducing the aggregation of tau via inhibiting O-GlcNAcase. Various smallmolecule O-GlcNAcase inhibitors with different chemical structures have been developed and used as chemical probes to explore the O-GlcNAc pathway. Although many reported inhibitors have shown that O-GlcNAcase activity has single-digit nmol IC50 values in binding assays, and molecules, such as LY-3372689, have entered phase II clinical studies, further exploration of novel OGlcNAcase inhibitors with higher inhibitory activity and specificity is still worthy of attention. This article reviews the pathogenesis and therapeutic role of O-GlcNAcase in Alzheimer's disease, as well as the recent progress of O-GlcNAcase small molecule inhibitors, including sugar-derived or non-sugar scaffolds, and summarizes the clinical progress and potential prospects of O-GlcNAcase inhibitors.