Smart Trading Rule: A Modular Machine Learning Framework for Portfolio Optimization with Transaction Costs

Traditional approaches to portfolio optimization embed transaction costs directly into the optimization process, inherently constraining the model's ability to identify ideal investment strategies. Conversely, ignoring transaction costs entirely leads to excessive trading and diminished returns. This paper introduces a novel two-step "smart trading rule", which effectively resolves this fundamental trade-off by decoupling portfolio optimization from transaction cost management. Our approach first determines the optimal frictionless portfolio allocation and subsequently employs an independent cost-benefit decision rule to execute trades only when expected gains surpass the associated transaction costs. We rigorously test this methodology across realistic market conditions using optimization-based machine-learning frameworks (XGBoost and LSTM), evaluating its performance against conventional one-step approaches and established benchmarks. Empirical results demonstrate that our smart trading rule consistently provides significantly higher returns, superior Sharpe ratios, and reduced portfolio turnover. Moreover, robustness analyses confirm these benefits remain stable across bull and bear markets, including extreme conditions such as the COVID-19 crash and the inflation-induced 2022 market correction. By separating portfolio optimization from execution, our approach delivers operational efficiency, enhanced computational speed, and practical decision-making benefits, providing substantial value to portfolio managers and institutional investors facing real-world transaction costs.