过度拟合
铰链损耗
经验风险最小化
计算机科学
人工神经网络
人工智能
机器学习
交叉熵
均方误差
趋同(经济学)
熵(时间箭头)
功能(生物学)
模式识别(心理学)
数学
统计
支持向量机
物理
量子力学
进化生物学
经济
生物
经济增长
作者
Siyuan Zhang,Linbo Xie
标识
DOI:10.1016/j.neucom.2023.126735
摘要
Deep learning, based on Empirical Risk Minimization (ERM), typically aims to fit the ideal outputs of all samples due to its large capacity. However, models trained based on empirical losses like cross entropy (CE) or mean square error (MSE), often learn unnecessary information during classification, leading to premature overfitting. On the other hand, the result-focused loss functions, i.e., zero–one loss or hinge loss, are hard to optimize and thus are rarely applied directly in neural network. This paper proposes a novel leader learning in classification, where CE is gradually trained by classification results using sample-dependent cost-sensitive learning. As complementary, the stepwise-changed CE covers the deficiency on classification error while preserving the advantage of fast convergence. In this way, the deviation between CE and classification error can be corrected. Experimental results demonstrate that the proposed leader learning has a more significant convergence trend than the baseline algorithms. Moreover, the loss function learned from a specific dataset has broad generality that can be transferred to other models as prior knowledge.
科研通智能强力驱动
Strongly Powered by AbleSci AI