Molecular deep learning at the edge of chemical space

Molecular machine learning models often fail to generalize beyond the chemical space of their training data, limiting their ability to reliably perform predictions on structurally novel bioactive molecules. To advance the ability of machine learning to go beyond the ‘edge’ of their training chemical space, we introduce a joint modeling approach that combines molecular property prediction with molecular reconstruction, enabling us to estimate model generalizability through a new reconstruction-based ‘unfamiliarity’ metric. Via a systematic analysis spanning more than 30 bioactivity datasets, we demonstrate that unfamiliarity not only effectively identifies out-of-distribution molecules but also serves as a reliable predictor of classifier performance. Even when faced with the presence of strong distribution shifts, unfamiliarity yields robust and meaningful molecular insights that go unnoticed by traditional methods. Our findings highlight that joint modelling can be a powerful strategy for extending the reach of machine learning models into uncharted regions of chemical space, advancing the discovery of diverse and novel molecules.