MLPACK is the first comprehensive scalable machine learning library. Developed by the Fundamental Algorithmic and Statistical Tools laboratory (FASTlab), MLPACK and its core functions library FASTlib are the much needed filling of an existing void. Previously, researchers had to either (a) settle for poorly-scaling collections of methods implemented for academic purposes, (b) hunt down the often difficult to find and difficult to apply yet fast code writen by algorithms' developers, or (c) reimplement solutions to their specific analysis problems from scratch. With MLPACK, we offer a fourth option, in which researchers may find all the methods they need designed favoring both speed and usability. 1. CONTENTS ----------- MLPACK currently includes the following algorithms: - allknn - A dual tree based $k$-nearest neighbor classifier using kd-trees. - allnn - A dual-tree based $k$-nearest neighbor classifier using kd-trees, optimized for $k = 1$. - fastica - Implements the FastICA Algorithm for Independent Component Analysis using fixed-point optimization with various independence-minded contrast functions. - hmm - Implements 3 types of Hidden Markov Models: discrete, gaussian and mixture of gaussians. - infomax_ica - Implements the Information Maximisation algorithm for Independent Component Analysis. - kalman - Implements of the Kalman filter ensuring the positive definiteness of the error covariance matrices by using QR and Cholesky factorizations in both the measurement and time update steps. - kde - Implements the following versions of kernel density estimation: Using depth first dual tree, multidimensional fast fourier transform, multidimensional fast gaussian transform and multidimensional improved fast gaussian transform. - mog - Implements parametric estimation of a mixture of Gaussians using two different loss functions - maximum likelihood and the L2 error. - naive_bayes - Implements the Naive Bayes Classifier. - optimization - Implements two optimizers - The Nelder-Mead algorithm and the Quasi-Newton algorithm. - series_expansion - Implements the series expansion needed for the fast $N$-body algorithm. Gaussian series expansion in $O(D^p)$ and $O(p^D)$ are implemented. - svm - Implements the Support Vector Machine classifier and regression. Includes the Sequential Minimial Optimization algorithm. All algorithms are available in both executable and linkable form. [More...] 2. HOWTO -------- 2.1. Quick start ---------------- The CMake build system has been tested to work on Linux or Windows XP/Vista under Cygwin using gcc4 and gcc3.4. It will probably work on Mac OSX using gcc also. In the future, we will test it under the other build systems that CMake supports, like Visual Studio and Eclipse, but for now a gcc/make development environment is your best bet. 1. Use the Cygwin installer or your Linux package manager to install: * gcc * g++ * g77 or gfortran (cmake wants a Fortran compiler--make it happy) * make * cmake 2.6 or higher * BLAS & LAPACK (usually separate in Linux, both included as "lapack" under "Math" in the Cygwin installer) * The library opt++, needed for optimization routines. This is technically optional, but very many machine learning algorithms need to do optimization. Optional, but strongly recommended: * Trilinos 10.0 or above, if you want sparse matrices. * The Boost C++ libraries (not directly used, but useful, may become mandatory later) 2. Install the opt++ optimization library. Again, this is optional, but will be useful if you are developing new ML algorithms. The opt++ library seems to only be supported on systems with GNU autotools. MS VC++ users are on your own. * The source code can be obtained from https://software.sandia.gov/opt++/ * Extract the tar file to some OPTPP_DIR. * In the file OPTPP_DIR/newmat11/include.h, uncomment the line "#define SETUP_C_SUBSCRIPTS". * Now just follow the directions in the INSTALL file for opt++. By default, it installs in /usr/local. It uses some fairly generic filenames, so you might have to install in another location if you get conflicts. 3. If you want sparse matrices, install the Trilinos package. * The source code can be obtained from http://trilinos.sandia.gov/ * Extract the tar file to some directory and cd there. * Create a build directory there. * Trilinos has very many configuration options. For the minimal configuration that will work: - cp FASTLIB/script/fastlib-trilinos-minimal-serial-debug-cmake \ TRILINOS/build - Optionally edit the script to make CMAKE_INSTALL_PREFIX point where you'd like. Fastlib will automatically check standard install dirs and /opt/trilinos during its build. - Make any other changes you'd like (see TRILINOS/sampleScripts and TRILINOS/cmake/TrilinosCMakeQuickstart.txt for ideas) and run the script. - You may get an error saying that you don't have a recent enough version of CMake. You can install a newer version, but the configuration script above seems to work with CMake 2.6.2. Just change the first line of TRILINOS/CMakeLists.txt to read "CMAKE_MINIMUM_REQUIRED(VERSION 2.6)" 4. Now all dependencies should be installed. For the rest of the instructions, assume you've extracted the MLPACK tar or zip file to /path/to/fastlib (it's the directory this file is in). 5. After this, you will need to make sure your environment variables are set up properly. Set FASTLIBPATH to this directory, e.g. export FASTLIBPATH=/path/to/fastlib (bash) setenv FASTLIBPATH=/path/to/fastlib (csh/tcsh) 6. Change to the fastlib subdirectory and create a build directory: cd $FASTLIBPATH/fastlib mkdir build (this can actually go anywhere you want) cd build 7. CMake's default is to install things in /usr/local/, which may not be what you want. So, let's configure our cmake project: % ccmake .. (if in build subdir) % ccmake $FASTLIBPATH/fastlib (if you put the build dir some weird place) This will bring up a curses interface (GUIs exist for cmake; feel free to install one). Hit "c" to configure. You should see a screen like this: CMAKE_BUILD_TYPE * CMAKE_INSTALL_PREFIX */usr/local FASTLIB_WITH_OPTIMIZERS *ON FASTLIB_WITH_SPARSE *ON OPTPP_BASE_DIR */usr/local TRILINOS_BASE_DIR */usr/local From the CMake docs: "CMAKE_BUILD_TYPE... Possible values are empty, Debug, Release, RelWithDebInfo and MinSizeRel. This variable is only supported for make based generators." One choice for the install prefix that will make things similar to the old build system is to use FASTLIBPATH as the CMAKE_INSTALL_PREFIX. This will put the libraries and header files in the lib/ and include/ subdirectories. So, go to the CMAKE_INSTALL_PREFIX line, hit Enter, change it to whatever you want, and hit Enter again. Change FASTLIB_WITH_OPTIMIZERS to OFF if you don't want to include opt++ support (e.g. if the opt++ build failed for some reason) Change FASTLIB_WITH_SPARSE to OFF if you don't want to include sparse matrices (e.g. if the Trilinos build failed for some reason) Hit "c" to update the configuration. If Trilinos or opt++ is required but not found, there should be an error. Otherwise, hit "g" to generate the build files and exit. There should now be a Makefile in the build directory. Now run % make (or "VERBOSE=1 make &> make.log" if you want to know all the details) % make install Let's assume you installed things to $FASTLIBPATH. If you look in $FASTLIBPATH/lib and $FASTLIBPATH/include, you should see a static library and a bunch of header files. At this point, you should be able to develop your code using whatever tools you want. 8. Quick Test For instance, let's test out the install by compiling one of the mlpack programs by hand: % cd $FASTLIBPATH/mlpack/allnn % g++ allnn_test.cc -o allnn_test -DDISABLE_DISK_MATRIX -L$FASTLIBPATH/lib \ -I$FASTLIBPATH/include -lfastlib -llapack % ./allnn_test (tests should pass) That "-DDISABLE_DISK_MATRIX" is an annoying define that you'll have to add until we add support for the memory manager to the CMake build. An example of building a program that requires opt++: % cd $FASTLIBPATH/examples/optimization % g++ optimizer_tests.cc -o optimizer_tests -DDISABLE_DISK_MATRIX \ -L$FASTLIBPATH/lib -I$FASTLIBPATH/include -lfastlib -lopt -lnewmat -llapack You might have to add the opt++ library and include paths if you installed it somewhere nonstandard. 9. Compiling MLPACK code with CMake % cd $FASTLIBPATH/mlpack % mkdir build && cd build % ccmake .. Hit 'c' to configure. You have to tell CMake where to find the fastlib you just installed. By default, it looks in /usr/local, so it won't find the library. Change FASTLIB_BASE_DIR to where you installed fastlib (e.g. the path in $FASTLIBPATH) and hit 'c' again. FASTLIB_LIB should now be correct. Hit 'g' to generate the Makefiles. % make This builds libraries and executables for each algorithm in its corresponding subdirectory. You can run the programs from there. There is no sensible 'make install' target yet. One easy(ish) way to develop your code would be to make a subdirectory for it in mlpack. Then you could copy and modify the CMakeLists.txt file from another subdirectory to create your libraries and executables. You also have to modify the CMakeLists.txt file in the mlpack directory to recurse into your subdirectory. Then, when you configure mlpack, it will also create make targets for your code. FUTURE PLANS ------------ MLPACK is growing quickly, and will soon also include: - Affinity Propagation. - C # versions of nearest neighbor. - Convex optimization routines. - Dual-tree nearest neighbor algorithm using Cover trees. - Disk-based algorithms using memory-mapped file implementation. - Euclidean Minimum Spanning Tree. - Graphical model inference. - Kernel Discriminant Analysis - Local linear regression. - Manifold learning algorithms (diffusion maps, Laplacian Eigenmaps, LLE, Isomap). - Nonnegative matrix factorization and many of its variants. - Nonnegative SVM. - Orthogonal range search. - Ranking SVM. - Sparse KDE using QP. Happy coding! CONTACTS -------- Please contact the following authors of the code for any problems: Ryan Riegel (rriegel@cc.gatech.edu) Nikolaos Vasiloglou (nvasil@ieee.org) Dongryeol Lee (dongryel@cc.gatech.edu)