Learn to Understand: Knowledge Exemplification via Multi-Agent Cooperation for Science Question Answering

Science Question Answering (SQA) is an important task for evaluating models' capability to reason with scientific knowledge. However, the extensive availability of scientific information (e.g., basic concepts in biology, physics, and chemistry) in pre-trained corpora may cause large language models (LLMs) to rely more on memorized information rather than actual reasoning when answering questions. This reliance persists even with techniques like Chain-of-Thought prompting, resulting in shallow understanding and limited reasoning based on scientific knowledge. Therefore, to enhance LLMs' capacity to comprehend and apply scientific knowledge, we propose a framework called Multi-Agent Cooperation-based Knowledge Exemplification (MCKE). Specifically, MCKE leverages knowledge alongside questions to create exemplified knowledge, promoting deeper understanding through innovative knowledge representation. To better evaluate the model's ability to reason and apply knowledge, we introduce NovSciQA, a multiple-choice question answering dataset based on newly created scientific knowledge. This dataset covers multi-subject scientific knowledge and questions that do not exist in reality, making it impossible for the model to rely on memorized answer-related information to answer questions. Experimental results show that the MCKE framework outperforms baselines, and the NovSciQA dataset effectively assesses models' knowledge understanding and application. Our code and dataset are available in https://anonymous.4open.science/r/MCKE-NovSciQA.