作者
Sha Yuan,Haitao Zhao,Shen Zhao,Jiahong Leng,Yangxiao Liang,Xiaozhi Wang,Jifan Yu,Xin Lv,Zhimin Shao,Jiaao He,Yankai Lin,Xu Han,Zhenghao Liu,Ning Ding,Yongming Rao,Yizhao Gao,Liang Zhang,Ming Ding,Fengyu Cong,Yisen Wang,Mingsheng Long,Jing Zhang,Yinpeng Dong,Tianyu Pang,Peng Cui,Lei Huang,Lirong Zheng,Huawei Shen,Hui Zhang,Quanshi Zhang,Qingxiu Dong,Zhixing Tan,Mingxuan Wang,Shuo Wang,Long Zhang,Haoran Li,Junwei Lucas Bao,Yingwei Pan,Weinan Zhang,Yu Zhou,Rui Yan,Chence Shi,Minghao Xu,Zuobai Zhang,Guoqiang Wang,Xiang Peng,Mengjie Li,Xiaoyu Chu,Zijun Yao,Fei Zhu,Shulin Cao,Weicheng Xue,Zixuan Ma,Zhengyan Zhang,Shengding Hu,Yujia Qin,Chaojun Xiao,Zheni Zeng,Guixiang Cui,Weize Chen,Weilin Zhao,Y. Yuan,Peng Li,Wenzhao Zheng,Wenliang Zhao,Ziyi Wang,Borui Zhang,Nanyi Fei,Anwen Hu,Zenan Ling,Haoyang Li,Boxi Cao,Xianpei Han,Wei Zhan,Baobao Chang,Han‐Dong Sun,Jiawen Deng,Chujie Zheng,Juanzi Li,Lei Hou,Xigang Cao,Jidong Zhai,Zhiyuan Liu,Maosong Sun,Jiwen Lu,Zhiwu Lu,Qiong‐Hua Jin,Rongfang Song,Ji-Rong Wen,Zhouchen Lin,Liwei Wang,Hang Su,Jun Zhu,Zhifang Sui,Jiajun Zhang,Yang Liu,Xiaodong He,Minlie Huang,J. Tang,Jie Tang
摘要
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm. Researchers have achieved various outcomes in the construction of BMs and the BM application in many fields. At present, there is a lack of research work that sorts out the overall progress of BMs and guides the follow-up research. In this paper, we cover not only the BM technologies themselves but also the prerequisites for BM training and applications with BMs, dividing the BM review into four parts: Resource, Models, Key Technologies and Application. We introduce 16 specific BM-related topics in those four parts, they are Data, Knowledge, Computing System, Parallel Training System, Language Model, Vision Model, Multi-modal Model, Theory&Interpretability, Commonsense Reasoning, Reliability&Security, Governance, Evaluation, Machine Translation, Text Generation, Dialogue and Protein Research. In each topic, we summarize clearly the current studies and propose some future research directions. At the end of this paper, we conclude the further development of BMs in a more general view.