Reproducing kernel Hilbert space

In functional analysis (a branch of mathematics), a reproducing kernel Hilbert space (RKHS) is a Hilbert space of functions in which point evaluation is a continuous linear functional. Roughly speaking, this means that if two functions f {displaystyle f} and g {displaystyle g} in the RKHS are close in norm, i.e., ‖ f − g ‖ {displaystyle |f-g|} is small, then f {displaystyle f} and g {displaystyle g} are also pointwise close, i.e., | f ( x ) − g ( x ) | {displaystyle |f(x)-g(x)|} is small for all x {displaystyle x} . The reverse need not be true. | L x ( f ) | := | f ( x ) | ≤ M ‖ f ‖ H ∀ f ∈ H . {displaystyle |L_{x}(f)|:=|f(x)|leq M|f|_{H}{ ext{ }}forall fin H.,} (1) f ( x ) = L x ( f ) = ⟨ f , K x ⟩ ∀ f ∈ H . {displaystyle f(x)=L_{x}(f)=langle f, K_{x} angle quad forall fin H.} (2) K ( x , y ) = ⟨ φ ( x ) , φ ( y ) ⟩ . {displaystyle K(x,y)=langle varphi (x),varphi (y) angle .} (3) γ : X × Λ × X × Λ → R . {displaystyle gamma :X imes Lambda imes X imes Lambda o mathbb {R} .} (4) In functional analysis (a branch of mathematics), a reproducing kernel Hilbert space (RKHS) is a Hilbert space of functions in which point evaluation is a continuous linear functional. Roughly speaking, this means that if two functions f {displaystyle f} and g {displaystyle g} in the RKHS are close in norm, i.e., ‖ f − g ‖ {displaystyle |f-g|} is small, then f {displaystyle f} and g {displaystyle g} are also pointwise close, i.e., | f ( x ) − g ( x ) | {displaystyle |f(x)-g(x)|} is small for all x {displaystyle x} . The reverse need not be true. It is not entirely straightforward to construct a Hilbert space of functions which is not an RKHS. Note that L2 spaces are not Hilbert spaces of functions (and hence not RKHSs), but rather Hilbert spaces of equivalence classes of functions (for example, the functions f {displaystyle f} and g {displaystyle g} defined by f ( x ) = 0 {displaystyle f(x)=0} and g ( x ) = 1 Q {displaystyle g(x)=1_{mathbb {Q} }} are equivalent in L2). However, there are RKHSs in which the norm is an L2-norm, such as the space of band-limited functions (see the example below). An RKHS is associated with a kernel that reproduces every function in the space in the sense that for any x {displaystyle x} in the set on which the functions are defined, 'evaluation at x {displaystyle x} ' can be performed by taking an inner product with a function determined by the kernel. Such a reproducing kernel exists if and only if every evaluation functional is continuous. The reproducing kernel was first introduced in the 1907 work of Stanisław Zaremba concerning boundary value problems for harmonic and biharmonic functions. James Mercer simultaneously examined functions which satisfy the reproducing property in the theory of integral equations. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of Gábor Szegő, Stefan Bergman, and Salomon Bochner. The subject was eventually systematically developed in the early 1950s by Nachman Aronszajn and Stefan Bergman. These spaces have wide applications, including complex analysis, harmonic analysis, and quantum mechanics. Reproducing kernel Hilbert spaces are particularly important in the field of statistical learning theory because of the celebrated representer theorem which states that every function in an RKHS that minimises an empirical risk functional can be written as a linear combination of the kernel function evaluated at the training points. This is a practically useful result as it effectively simplifies the empirical risk minimization problem from an infinite dimensional to a finite dimensional optimization problem. For ease of understanding, we provide the framework for real-valued Hilbert spaces. The theory can be easily extended to spaces of complex-valued functions and hence include the many important examples of reproducing kernel Hilbert spaces that are spaces of analytic functions. Let X {displaystyle X} be an arbitrary set and H {displaystyle H} a Hilbert space of real-valued functions on X {displaystyle X} . The evaluation functional over the Hilbert space of functions H {displaystyle H} is a linear functional that evaluates each function at a point x {displaystyle x} , We say that H is a reproducing kernel Hilbert space if, for all x {displaystyle x} in X {displaystyle X} , L x {displaystyle L_{x}} is continuous at any f {displaystyle f} in H {displaystyle H} or, equivalently, if L x {displaystyle L_{x}} is a bounded operator on H {displaystyle H} , i.e. there exists some M > 0 such that While property (1) is the weakest condition that ensures both the existence of an inner product and the evaluation of every function in H {displaystyle H} at every point in the domain, it does not lend itself to easy application in practice. A more intuitive definition of the RKHS can be obtained by observing that this property guarantees that the evaluation functional can be represented by taking the inner product of f {displaystyle f} with a function K x {displaystyle K_{x}} in H {displaystyle H} . This function is the so-called reproducing kernel for the Hilbert space H {displaystyle H} from which the RKHS takes its name. More formally, the Riesz representation theorem implies that for all x {displaystyle x} in X {displaystyle X} there exists a unique element K x {displaystyle K_{x}} of H {displaystyle H} with the reproducing property,