Leveraging tree-transformer VAE with fragment tokenization for high-performance large chemical model generation

Molecular generation models, especially chemical language model (CLM) utilizing SMILES, a string representation of compounds, face limitations in handling large and complex compounds while maintaining structural accuracy. To address these challenges, we propose the Fragment Tree-Transformer based VAE (FRATTVAE), which treats molecules as tree structures with fragments as nodes. FRATTVAE incorporates several innovative techniques to enhance molecular generation. Molecules are decomposed into fragments and organized into tree structures, allowing for efficient handling of large and complex compounds. Tree positional encoding assigns unique positional information to each fragment, preserving hierarchical relationships. The Transformer's self-attention mechanism models complex dependencies among fragments. This architecture allows FRATTVAE to surpass existing methods, making it a robust solution that is scalable to unprecedented dataset sizes and molecular complexities. Distribution learning across various benchmark datasets, from small molecules to natural compounds, showed that FRATTVAE consistently achieved high accuracy in all metrics while balancing reconstruction accuracy and generation quality. In molecular optimization tasks, FRATTVAE generated high-quality, stable molecules with desired properties, avoiding structural alerts. These results highlight FRATTVAE as a robust and versatile solution for molecular generation and optimization, making it well-suited for a variety of applications in cheminformatics and drug discovery.