Model-Based Prior for Model-Free Reinforcement Learning in Process Control: An Offline Methodology for Semiconductor Manufacturing

Process control is essential in advanced manufacturing to minimize variations caused by disturbances from manufacturing environments. As precision requirements continue to rise, process models have become increasingly complex, making it difficult for model-based controllers to accurately characterize the relationship between control inputs and system outputs. Meanwhile, unstable and unobservable disturbances, which are inevitable during manufacturing processes, introduce significant variations for model-free controllers. To address this challenge, this work develops a novel model-free reinforcement learning (RL) control approach that leverages model-based prior knowledge. Although prior knowledge cannot define the process model, it helps simplify the model structure and reduce variations in model-free controllers. Unlike traditional RL methods that require extensive real-time experimentation, we introduce an offline RL approach that consists of offline learning and online control phases and relies solely on fixed historical data. During the offline learning phase, nonparametric regression is employed to approximate the unknown process, while Gaussian process is used to predict disturbances. These two methods are integrated with a policy-gradient-search RL algorithm in online control. Theoretical properties of our control methodology are discussed. Simulations and a crystallization case study are conducted to validate the effectiveness of the proposed method.