A Survey of System Scheduling for HPC and Big Data

In the rapidly expanding field of parallel processing, job schedulers act as the "operating systems" of the clusters, including modern big data architectures and supercomputing systems. Job schedulers manage and allocate system resources, dispatch the queued jobs, and control the execution of processes on the allocated resources. In this paper, we firstly make an introduction to the cluster schedulers. Then according to the scenarios, we make a comprehensive survey of schedulers for HPC and Big Data. We can conclude that most of these current schedulers are centralized, which means master assigns jobs to the slaves. We call this mode Push, which is different from our new idea that introduces Pull to the schedulers. We proposed a novel scheduling model that allow slaves to actively pull jobs from master to execute. By analyzing the execution time and resource requests of jobs in "Tianhe-II", we will clarify that scheduling based on Push & Pull is a direction worthy of in-depth study in the future.