Minimization problems involving a finite sum as objective function often arise in machine learning applications. The number of components of the finite-sum term is typically very large, by making unfeasible the computation of its gradient. For this reason stochastic gradient methods are commonly considered. The performance of these approaches strongly relies on the selection of both the learning rate and the mini-batch size employed to compute the stochastic direction. In this paper we combine a recent idea to select the learning rate as a diagonal matrix based on stochastic Barzilai-Borwein rules together with an adaptive subsampling technique to fix the mini-batch size. Convergence results of the resulting stochastic gradient algorithm are shown for both convex and non-convex objective functions. Several numerical experiments on binary classification problems are carried out to compare the proposed method with other state-of-the-art schemes.
Diagonal Barzilai-Borwein Rules in Stochastic Gradient-Like Methods / Franchini, G.; Porta, F.; Ruggiero, V.; Trombini, I.; Zanni, L.. - 1824:(2023), pp. 21-35. (Intervento presentato al convegno 6th International Conference on Optimization and Learning, OLA 202 nel 3-5 May 2023) [10.1007/978-3-031-34020-8_2].
Diagonal Barzilai-Borwein Rules in Stochastic Gradient-Like Methods
Trombini I.;
2023-01-01
Abstract
Minimization problems involving a finite sum as objective function often arise in machine learning applications. The number of components of the finite-sum term is typically very large, by making unfeasible the computation of its gradient. For this reason stochastic gradient methods are commonly considered. The performance of these approaches strongly relies on the selection of both the learning rate and the mini-batch size employed to compute the stochastic direction. In this paper we combine a recent idea to select the learning rate as a diagonal matrix based on stochastic Barzilai-Borwein rules together with an adaptive subsampling technique to fix the mini-batch size. Convergence results of the resulting stochastic gradient algorithm are shown for both convex and non-convex objective functions. Several numerical experiments on binary classification problems are carried out to compare the proposed method with other state-of-the-art schemes.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.