Semantic Chunking and Chain-Of-Thought Reasoning for Rag-Based Document Processing

This paper presents a novel two-phase semantic chunking methodology designed to enhance document processing within Retrieval-Augmented Generation (RAG) systems. The proposed approach utilizes Large Language Models (LLMs) and Chain of Thought (CoT) to systematically generate and refine document chunks, while concurrently producing associated metadata, such as hypothetical user queries and contextual tags. By integrating established information retrieval techniques-namely Best Matching 25 (BM25)-with the advanced semantic understanding capabilities of LLMs, the proposed method substantially improves the relevance and quality of retrieved context for Generative Artificial Intelligence (GenAI) applications. Empirical evaluations reveal that this approach yields significant improvements in response accuracy and contextual relevance when compared to traditional chunking techniques. The implementation utilizes the open-source Qwen 2.572 B model for its semantic processing operations, demonstrating how state-of-the-art language models can be effectively deployed in practical RAG systems.