The influence of parameter initialization on the training time and accuracy of a nonlinear regression model
- Authors: Burnaev E.V.1, Erofeev P.D.1
-
Affiliations:
- Kharkevich Institute for Information Transmission Problems
- Issue: Vol 61, No 6 (2016)
- Pages: 646-660
- Section: Mathematical Models and Computational Methods
- URL: https://journals.rcsi.science/1064-2269/article/view/197031
- DOI: https://doi.org/10.1134/S106422691606005X
- ID: 197031
Cite item
Abstract
In order to construct a nonlinear regression model we have to accurately (in some sense) initialize parameters of the model. In this work we performed comparison of several widely used methods and several newly developed approached for initialization of parameters of a regression model, represented as a decomposition in a linear dictionary of some parametric functions (sigmoids). We proposed a general deterministic approach for initialization, providing repeatability of results, reduction of a learning time and in some cases increase of a regression model accuracy; we developed two new algorithms (based on a piecewise-linear approximation and based on local properties of approximable dependency) in the framework of the proposed approach; we developed randomized initialization algorithm (spherical initialization) for effective approximation of high-dimensional dependencies; we improved the classical initialization method SCAWI (by locating centers of sigmoids in sample points), providing a regression model accuracy improvement on specific classes of dependencies (smooth functions and discontinuous functions with a number of local peculiarities in an input domain) when using RProp algorithm for learning; we performed comparison of classical and newly proposed initialization methods and highlighted the most efficient ones.
About the authors
E. V. Burnaev
Kharkevich Institute for Information Transmission Problems
Author for correspondence.
Email: burnaev@iitp.ru
Russian Federation, Bol’shoi Karetnyi per. 19, str. 1, Moscow, 127051
P. D. Erofeev
Kharkevich Institute for Information Transmission Problems
Email: burnaev@iitp.ru
Russian Federation, Bol’shoi Karetnyi per. 19, str. 1, Moscow, 127051