Smart Trading Rule: A Modular Machine-Learning Framework for Portfolio Optimization with Transaction Costs

Traditional approaches to portfolio optimization embed transaction costs directly into the optimization process, inherently constraining the model’s ability to identify ideal investment strategies. Conversely, entirely ignoring transaction costs leads to excessive trading and diminished returns. This article introduces a modular two-step machine-learning framwork, the “smart trading rule,” that effectively resolves this fundamental trade-off by decoupling portfolio optimization from transaction cost management. Our approach first determines the optimal frictionless portfolio allocation, and then employs an independent cost-benefit decision rule to execute trades only when expected gains surpass the associated transaction costs. This design functions as a model-agnostic execution layer that can be integrated with predictive architectures without modifying their training objectives, thereby enhancing stability and scalability. We rigorously test this methodology across realistic market conditions using optimization-based machine-learning frameworks (XGBoost and LSTM), evaluating its performance against conventional one-step approaches and established benchmarks. Empirical results demonstrate that our smart trading rule consistently provides significantly higher returns, superior Sharpe ratios, and reduced portfolio turnover. By separating portfolio optimization from execution, our approach acts as an implicit regularization mechanism that mitigates overfitting while delivering operational efficiency and practical decision-making benefits to portfolio managers.