Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning

arXiv:2604.22838v1 Announce Type: cross Abstract: With the accumulation of resources in the era of big data and the rise of pre-trained models in deep learning, optimizing neural networks for various tasks often involves different strategies for fine-tuning pre-trained models versus training from scratch. However, existing optimizers primarily focus on reducing the loss function by updating model parameters, without fully addressing the unique demands of these two major paradigms. In this paper,