Constructing a quantitative factor investment strategy based on hundreds of candidate factors is a critical challenge. Existing linear models do not account for nonlinearities and variable interactions, while complex machine learning models are easily overfitting. In this paper, motivated by the portfolio sorts methods in empirical asset pricing, we propose an alternative approach called grouping-based AdaBoost by adapting the existing AdaBoost. It introduces the experience of the financial field into the algorithm design to improve the performance and generalization of machine learning-based factor investing strategies. The proposed method restricts the factor to only predict the common part of the returns of the same groups and allows the potential nonlinear relationship between a factor and the return. Moreover, to enhance the model's ability to use factors with high correlation, we extend the single-grouping AdaBoost in a multi-grouping way. Experiments on the Chinese A-share market demonstrate the effectiveness of our approach in both stock performance classification and portfolio selection and provide intuitive evidence for the generalization of the proposed method.