Incorporating five or more metals into a single structure creates a new family of alloys, high-entropy alloys (HEAs), which hold several exceptional properties, such as outstanding stabilities and continuous electronic structures, making them promising catalysts for a range of chemical conversions. Due to the high-dimensional design space, machine learning algorithms are frequently used for the optimization of high-entropy alloys to achieve enhanced catalytic performance. The precision of machine learning depends on the structural features of high-entropy alloys, and this work aims to explore whether pair distribution function (PDF) data of HEAs can be adopted as input features for the probabilistic optimization of HEA-based catalysts through machine learning. Here, we first address the challenge of the high dimensionality of PDF data through principal component analysis (PCA), and then use the PCA-reduced PDF as input features to predict the Gibbs free energy of nitrate adsorption on the FeCoNiCuZn HEA surfaces via a hybrid framework comprising a transformer-based model and a fine-tuned large language model (LLM). The results show that the as-constructed hybrid framework can accurately predict the Gibbs free energy of nitrate adsorption using PCA-reduced PDF data, with performance significantly superior to that of conventional algorithms such as random forest, support vector regression, and gradient boosting. In the meantime, the use of LLM can further improve the prediction accuracy and extract interpretable insights from the data set, eventually allowing for the predictive design of HEA-based catalysts with optimized activity and selectivity.