Bridging prediction and decision: Advances and challenges in data-driven optimization

Data-driven approaches have revolutionized traditional optimization methods by integrating prediction with decision-making. This review examines the theoretical foundations, strengths, recent advancements, and limitations of three key methods—sequential optimization, end-to-end learning, and direct learning—highlighting their practical applications in power grid scheduling, operations management, and intelligent autonomous control. A multidimensional comparison is presented, followed by a discussion of the challenges in data-centric methodology, optimization methodology, and decision-making application. This paper offers a methodological guide and outlines future directions for academia and industry to enhance decision-making in complex data environments. Broader context: As big data technologies advance and data volumes grow, effectively leveraging these resources for complex decision-making has become a critical challenge for academia and industry. This review examines the transformative impact of big data and intelligent systems on traditional optimization paradigms, highlighting the continuum of data-driven optimization from predictive modeling to decision implementation. Key methodologies such as ''sequential optimization,'' ''end-to-end learning,'' and ''direct learning'' are analyzed, offering both theoretical insights and practical implications. Notably, we discuss breakthroughs such as implicit differentiation techniques, surrogate loss functions, and perturbation methods, which provide methodological guidance for achieving data-driven decision-making through prediction. By emphasizing the critical challenges across multiple dimensions, including data quality, model efficiency, and resilient decision-making under uncertainty, our review offers forward-looking insights to guide future research and foster the broader application of these approaches in diverse real-world scenarios.