Research on Efficient Fine-tuning of Large Language Model Parameters for Text Classification: A Review from Gradient Descent Basics to LoRA/QLoRA Methods

Large Language Models (LLMs) perform well in many text classification tasks, such as sentiment analysis, spam detection, and topic classification. However, adapting large language models to downstream tasks via full-parameter fine-tuning often requires significant computing power and GPU memory, resulting in high training costs, which is impractical for course assignments or resource-constrained scenarios. To address this issue, Parameter Efficient Fine-Tuning (PEFT) has gradually become a popular approach. Its idea is to freeze most model parameters and train only a small number of additional parameters, thereby reducing training costs. This paper takes text classification as an example and begins with an introduction to fundamental concepts such as stochastic gradient descent, cross-entropy loss, and confusion matrices. It first reviews the technological evolution from traditional NLP models (such as bag-of-words, Naive Bayes, and N-grams) to word embeddings, Transformers, and ultimately large language models. Then, it focuses on the low-rank adaptation concept of LoRA and the 4-bit quantization training mechanism of QLoRA, analyzing their characteristics from the perspectives of parameter count, GPU memory consumption, and training stability. This paper also discusses key representation-level issues such as word segmentation methods (BPE, WordPiece, SentencePiece), Type/Token differentiation, vocabulary size, and sequence length, explaining how these factors affect the model's generalization ability. Overall, this paper aims to answer the question of "how to adapt large models to text classification tasks at a lower cost under limited resources," and provides a clearer theoretical framework and methodological approach for course learning and introductory research.