Software Resources
Home
Table of Contents
Software Resources
About the Author
Ordering Information
Errata
This website contains the source code for all figures and a demo library of functions for classification learning. Please read the disclaimer before you proceed.

Figures

Most of the figures have been created using the language R. Note that the codes are designed to output the graphics directly into a postscript file. Replace the lines postscript(...) with x11() to redirect the output to the screen. Moreover, the output is done into the directory ../../ps/.

Chapter 1

bullet

Figure 1.1: Classification learning of handwritten digits. (demo.R,mnist100.dat)

bullet

Figure 1.2: Preference learning of Web pages.

bullet

Figure 1.3: Function learning in action (linear, cubic, 10th degree). (demo.R)

bullet

Figure 1.4: Clustering of 150 training points, Probability density of the estimated mixture model. (demo.R)

bullet

Figure 1.5: The first 49 digits, The 49 images in a data matrix. (demo.R, mnist100.dat)

bullet

Figure 1.6: Classification of three new images. (demo.R)

Chapter 2

bullet

Figure 2.1: Hypothesis space, Feature space. (grandcircle.m, plane.m, plot_hypothesis_space.m, plot_data_space.m)

bullet

Figure 2.2: A geometrical picture of the update step in the PLA.

bullet

Figure 2.3: Mapping of the unit square, Nine different decision surfaces. (feature.R)

bullet

Figure 2.4: Real valued function for varying values of (=0.5, =0.7, =1.0, =2.0). (density.m, demo.m)

bullet

Figure 2.5: Intensity plots of normalized Gram matrices when applying string kernels (bow kernel, subsequence kernel, substring kernel). (rbf_demo.R, strings.R, strings.c, data.tar)  

bullet

Figure 2.6: Geometrical margin of a plane (large margin, small margin). (demo.R)

bullet

Figure 2.7: Approximation to the Heaviside step function. (loss.R)

bullet

Figure 2.8: Largest inscribable ball in version space (3D, from top). (bayes_set.m, circle.m, grandcircle.m, plane.m, sphead.m)

Chapter 3

bullet

Figure 3.1: Effect of evidence maximization. (demo.R)

bullet

Figure 3.2: Gaussian process evidence maximization for a simple regression problem (log-evidence, data fit, noise fit). (demo.R, rbf_demo.R, train.dat)

bullet

Figure 3.3: A function sampled from the ARD prior (fast variations, slow variations). (demo.R, rbf_demo.R)

bullet

Figure 3.4: Latent variable model for classification. (demo.R, rbf_demo.R, train.dat).

bullet

Figure 3.5: Approximation to the zero-one loss by the sigmoidal loss, Likelihood model induced by the hinge loss. (loss.R).

bullet

Figure 3.6: Marginalized log-prior densities (n=1, n=2). (demo.R).

bullet

Figure 3.7: Five samples obtained by playing billiards on the sphere (3-D,

bullet

 2-D). (bpm.m, arc.m, grandcircle.m, plane.m, pol2car.m).

bullet

Figure 3.8: Fisher discriminant estimated from 80 data points, a geometrical interpretation of the Fisher discriminant. (demo.R).

Chapter 4

bullet

Figure 4.1: True densities underlying the data in Example 4.2.

bullet

Figure 4.2: Growth of the complexity as a function of the VC dimension (large scale, fine scale),(demo.R)

bullet

Figure 4.3: Curse of dimensionality (n=1,n=2,n=3).

bullet

Figure 4.4: Structural risk minimization in action. (demo.R)

bullet

Figure 4.5: 20 real-valued functions, a cover for the function class. (demo.R)

bullet

Figure 4.6: Relation between the real-valued output of a cover element and the real-valued of the covered output.

bullet

Figure 4.7: Two points on the real line, -covering.

bullet

Figure 4.8: Minimal training sample size. (demo.R).

Chapter 5

bullet

Figure 5.1: Expected risk of classifiers learned by a support vector machine with and without normalization (thyroid, sonar). (normalise.R, Linear toolbox, Sonar dataset, Thyroid dataset).

bullet

Figure 5.2: The clipped linear soft margin loss. (loss.R).

Appendix A

bullet

Figure A.1: Geometrical proof of Scheff�s theorem.  

bullet

Figure A.2: Probability mass functions (Bernoulli, Binomial, Possion, Uniform). (demo.R).

bullet

Figure A.3: Probability densities (Gaussian, Gamma, Exponential, Beta). (demo.R).

bullet

Figure A.4: Densities of the multidimensional normal distribution (Isotrpic covariance, general covariance). (demo.R).

bullet

Figure A.5: -covering and -packing.

Appendix C

bullet

Figure C.1: Graphical illustration of the main step in the basic lemma.

bullet

Figure C.2: Counting swappings (before, after).

bullet

Figure C.3: Counting swappings (easy, difficult).

bullet

Figure C.4: Geometrical proof of Lemma C.12.

bullet

Figure C.5: Comparison of the volume ratio bound and the exact value (n=10, n=100). (demo.R, volratio.c)

Learning Kernel Classifiers Toolbox

In addition to the R source code of the figures, I have implemented all algorithms mentioned in Appendix D in R. The code will run with R version 1.3.0; newer versions of R might not work (thanks to Thomas G�rtner). The PR_LOQO routine was written by Alexander Smola at GMD FIRST. In order to use this library, you are required to compile all the C source codes. If you are running UNIX, e.g. LINUX, just type

R SHLIB bayes.c

R SHLIB kernels.c

R SHLIB kperc.c

R SHLIB pr_loqo.c

 

In case, you are running Windows 98/NT/2000/XP, you have to replace R by Rcmd. At first, you have to download the sources.

bullet

Full TAR archive (contains both example datasets and source files).

bullet

Example dataset.

bullet

Source files: linear.R, kernels.R, kernels.c, bayes.c, kperc.c, pr_loqo.c, pr_loqo.h.

You can run several examples to get an idea how to use the many different algorithms.

bullet

exmp.bayes and exmp.bayes2 for Bayes point machines.

bullet

exmp.svm for support vector machines.

bullet

exmp.nu.svm for -support vector learning.

bullet

exmp.GP.class for Gaussian process classification.

bullet

exmp.GP.regress for Gaussian process regression.

bullet

exmp.RVM.class for classification learning with relevance vector machines.

bullet

exmp.RVM.regress for regression estimation with relevance vector machines.

bullet

exmp.fisher for Fisher discriminants.

bullet

exmp.perc for perceptron learning.

Accompanying Software

bulletR is a statistical software package which is under the GNU Public License and publicly available at http://www.r-project.org/. This link contains pre-compiled binaries for most operating systems. The language R is very similar to the language S  which is used in the commercially available package S-PLUS developed and distributed by Insightful. Documentation for this language can be found at http://cran.r-project.org/manuals.html.
bulletMatLab is a powerful and widely used software for numerical analysis and visualisation of scientific data (see http://www.mathworks.com/products/matlab for more information). MatLab is distributed by MathWorks Inc.).

Disclaimer

(C) Ralf Herbrich. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS FREE ONLY FOR NON-COMMERCIAL USE. IT MUST NOT BE MODIFIED AND DISTRIBUTED WITHOUT PRIOR PERMISSION OF THE AUTHOR. THIS SOFTWARE IS PROVIDED BY  "AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.