计算机科学
资源配置
培训(气象学)
计算机网络
资源管理(计算)
分布式计算
资源(消歧)
钥匙(锁)
实时计算
调度(生产过程)
电信网络
光通信
集成光学
作者
Dianxuan Fu,Yihao Zhang,Xiaomin Liu,Qizhi Qiu,Yicheng Xu,xingyu liu,Xin Miao (1674739),Ping Du,Shikui Shen,Weisheng Hu,Qunbi Zhuge
摘要
The explosive development in artificial intelligence (AI) like large language models (LLMs) requires massive graphics processing units (GPUs) during training, making distributed machine learning (DML) across multiple data centers (DCs) a practical approach to obtaining sufficient training resources. Among cross-DC DML techniques, pipeline parallelism (PP) is a natural solution, which partitions LLM layers across GPUs and orchestrates point-to-point communication between adjacent nodes. However, the high communication latency induces severe GPU idle time (“bubble”) due to poor communication and computation time overlapping in inter-DC PP. While prior works have explored partitioning and reordering computation tasks, the resource allocation for inter-DC communication between pipelined computing tasks over optical networks has been overlooked, resulting in higher GPU idle time and request congestion. Therefore, it is extremely important and challenging to propose an advanced inter-DC network resource allocation solution under the complex PP workload. In this work, we propose a cross-domain communication-bound-aware (CD-CBA) PP scheduling framework for multi-DC optical networks. Simulation results show that the proposed framework lowers the LLM training iteration time by up to 31.25%, decreases the bubble ratio by up to 11.81%, and reduces the transmission request blocking by up to 19.33%.
科研通智能强力驱动
Strongly Powered by AbleSci AI