Deep learning for optoacoustic imaging of reversibly switchable proteins: training and performance testing using simulated multispectral optoacoustic tomography images

SignificanceReversibly switchable optoacoustic proteins (rsOAPs) are a promising candidate for sensitive and quantitative optoacoustic (OA) imaging of genetically modified cell populations, such as chimeric antigen receptor (CAR) T-cells used in cancer immunotherapy. Although detection of rsOAPs has been demonstrated using classical machine learning approaches, there is still a need for higher detection sensitivity and more accurate quantification.AimWe aim to develop deep learning approaches for improving the sensitivity of the detection and accuracy of quantification of rsOAPs with OA imaging and a 3D simulation framework to create synthetic datasets for machine learning experiments.ApproachWe developed a forward model to generate labeled synthetic OA images of rsOAPs that takes into account light transport, acoustic wave propagation, and light-driven transitions between two different forms of the proteins. We used the synthetic images to train and evaluate machine learning models, including two convolutional neural networks and a transformer neural network, on the binary semantic segmentation and pixel-level prediction (regression) of the spatial distribution of the rsOAPs.ResultsWith this dataset, fine-tuned convolutional and transformer neural networks substantially outperformed classical machine learning approaches in the binary semantic segmentation of rsOAPs within the inhomogeneities (regions representing tumors), increasing the sensitivity from around 0.38 to 0.55 for noiseless data and from around 0.15 to 0.53 under a high level of stochastic noise (SNRdB=8.8) while maintaining a specificity of around 0.84, down to protein concentrations of order 10−8 M.ConclusionsA new methodology for the generation of synthetic OA imaging data of rsOAPs is presented, and the feasibility of deep learning for the accurate semantic segmentation and quantification of rsOAPs in OA imaging is demonstrated.