Complete Statistical Theory of Learning

Abstract

Existing mathematical model of learning requires using training data find in a given subset of admissible function the function that minimizes the expected loss. In the paper this setting is called Second selection problem. Mathematical model of learning in this paper along with Second selection problem requires to solve the so-called First selection problem where using training data one first selects from wide set of function in Hilbert space an admissible subset of functions that include the desired function and second selects in this admissible subset a good approximation to the desired function. Existence of two selection problems reflects fundamental property of Hilbert space, existence of two different concepts of convergence of functions: weak convergence (that leads to solution of the First selection problem) and strong convergence (that leads to solution of the Second selection problem). In the paper we describe simultaneous solution of both selection problems for functions that belong to Reproducing Kernel Hilbert space. The solution is obtained in closed form.