Challenges in knowledge graph generation for breast cancer using open-source LLMs and the role of mCODE.

e13704 Background: Generative AI has demonstrated potential for structuring electronic health record data, extracting meaningful insights, and mapping treatment guidelines. Knowlege graphs used for retrieval-augmented generation (graphRAG) have shown to produce more contextually-aware outputs from large language models (LLMs). However, the core entity-relationship generation requires manually construction of entities and relationships for the LLM to retrieve. A subsequent challenge is the LLM's ability to identify nodes and edges in a provided document, which becomes more challenging the more specialized the source information is. Microsoft recently demonstrated auto-tuning, a method wherein an LLM initially evaluates a document and identifies entity types and relationship types with specific prompting and tool calling. Methods: This study explores the generation of breast cancer-specific knowledge graphs using the NCCN guidelines for breast cancer, and the impact of incorporating mCODE as a structured framework. The models were utilized through Ollama, and included Triplex, phi3:mini, llama3.2, gemini2:2b, and qwen2.5. The NCCN Breast Cancer Guidelines (Version 6.2024) was converted to LLM-ingestible data using IBM’s docling. In the first series, an mCODE-derived entity list was used as entity generation, and a list of relationships was provided. Knowledge graph generation in a Neo4j GraphDatabase with each LLM was attempted and the graphs were visualized. In the second series, auto-tuning using LLMs with tool calling was attempted. Due to the poor extraction of nodes and edges in this series, knowledge graph generation was not completed. Evaluation metrics included knowledge graph completeness, relationship accuracy, and adherence to clinical guidelines. Results: With entity type and relationship types provided, Triplex, phi3:mini, and qwen2.5 produced nodes and edges, but were limited and incomplete (e.g., specific medications were not connected to specific treatments for specific breast cancers). Llama3.2 consistently failed to generate nodes and edges, appearing to struggle with medical terminology and unable to understand medically-specific relationships (e.g., brachytherapy was not connected to radiotherapy or treatments). While Gemini and Qwen were more effective at structuring data, their output often lacked oncology-specific details, particularly in aligning staging information with guideline-directed treatment pathways. Conclusions: This study highlights the significant challenges of knowledge graph generation in the specialized field of breast cancer and the need for LLMs trained with medical terminology to produce these graphs. mCODE may be a strong initial scaffold, but additional work such as input examples and additional entity type and relationship type development are needed to standardize breast cancer data representation for LLM-ingestion.