2091 lines
98 KiB
TeX
2091 lines
98 KiB
TeX
|
|
%% Note: If you are using this template for a proposal, pass the proposal option to
|
|
%% the class. Then the toplevel section will be \section, not \chapter. In either case
|
|
%% the \appendix command, which should only be used in the EndMatter environment, can be considered
|
|
%% to be at the same toplevel (either section or chapter}.
|
|
|
|
%% NOTE: Hyperref package doesn't yet completely work with the proposal option. It compiles
|
|
%% but the pdf bookmarks aren't correct.
|
|
|
|
%% Please report bugs/problems to djallred@gmail.com
|
|
|
|
%% Begin Preamble Info
|
|
\documentclass[12pt,letterpaper,doublespaced,ETD,dvips,proposal]{gtthesis}
|
|
|
|
\title{Scalable Manifold Learning with Applications in Speech}
|
|
\author{Nikolaos Vasiloglou II}
|
|
\copyrightyear{2007}
|
|
\graddate{August 2007} %Appears at bottom of title page
|
|
\approvaldate{July 2007} %Appears on signature page (not needed for a proposal)
|
|
|
|
% Add people to signature list (not needed for a proposal)
|
|
\addadvisor{David Anderson}{Assoc. Professor, School of ECE}{Georgia
|
|
Institute of Technology}
|
|
|
|
\addadvisor{Alexander Gray}{Assist. Professor, School of CS}{Georgia
|
|
Institute of Technology}
|
|
|
|
\addchair{Dr. Sitting Bull}{Professor, School of ECE}{Georgia
|
|
Institute of Technology}
|
|
|
|
\addreader{Dr. Wayne Geronimo}{Asst. Professor, School of
|
|
ECE}{Georgia Institute of Technology}
|
|
|
|
\addreader{Dr. David Goliath}{Asst. Professor, School of
|
|
ECE}{Georgia Institute of Technology} \addreader{Dr. Marky
|
|
Mark}{Asst. Professor, School of ECE}{Georgia Institute of
|
|
Technology}
|
|
|
|
\addreader{Dr. Larry A. Stooge}{Asst. Professor, School of
|
|
ECE}{Georgia Institute of Technology}
|
|
\bibfiles{template}
|
|
|
|
%%Packages to use
|
|
\usepackage{graphicx}
|
|
\usepackage{insfig}
|
|
\usepackage{hyperref}
|
|
\usepackage{verbatim}
|
|
\usepackage{boxedminipage}
|
|
|
|
%%%% These commands set the default depth the TOC should be expanded
|
|
%% out to for the chapters and appendices
|
|
%%
|
|
\setchaptertocdepth{2}
|
|
\setappendixtocdepth{2}
|
|
|
|
%%%% To rename different sections of the document use these
|
|
%% set???string{ ...string...} commands where ??? is replaced
|
|
%% by one of the following three letter codes
|
|
%%
|
|
%% toc = table of contents chp = chapter
|
|
%% lof = list of figures app = appendix
|
|
%% lot = list of tables prt = part (i.e. volume)
|
|
%% bib = bibliography abs = abstract
|
|
%% ind = index los = list of symbols
|
|
%% ded = dedication ack = acknowledgement
|
|
%% glo = glossary,list of terms
|
|
%%
|
|
%% Regardless of capitalization all will become capitalized in
|
|
%% the final document.
|
|
%%
|
|
\settocstring{Table of Contents}
|
|
\setlofstring{List of Figures}
|
|
\setlotstring{List of Tables}
|
|
\setbibstring{References}
|
|
\setindstring{Index}
|
|
\setdedstring{Dedication}
|
|
\setglostring{List of Terms}
|
|
\setchpstring{Chapter}
|
|
\setappstring{Appendix}
|
|
\setprtstring{Volume}
|
|
\setabsstring{Summary}
|
|
\setlosstring{List of Symbols}
|
|
\setackstring{Acknowledgment}
|
|
|
|
%%%% Set the default page style for each main enviroment
|
|
%%
|
|
\setfrontpagestyle{plain}
|
|
\setbodypagestyle{plain}
|
|
\setendpagestyle{plain}
|
|
|
|
\begin{document}
|
|
\pagestyle{plain}
|
|
|
|
\begin{FrontMatter}
|
|
\contents %Generates the TOC, LOF, and LOT, glossary and LOS
|
|
%% Below are some optional environments that can be used here
|
|
\end{FrontMatter}
|
|
|
|
\begin{Body}
|
|
\section{Introduction}
|
|
\label{intro}The explosion of information over the last few decades has created data volumes of extraordinary size and dimensionality,
|
|
bringing with it a great demand for analysis of and information extraction from these data.
|
|
Two approaches for analyzing these data are common in machine learning: parametric \cite{bishop2006pra, hastie2001esl, haykin1994nnc}
|
|
and non-parametric \cite{wasserman2006ans}.
|
|
In parametric analysis there is an assumption of an underlying model and the model is estimated based on the data.
|
|
In non-parametric analysis the data serves as the model.
|
|
|
|
|
|
%Information explosion during the last decades has
|
|
%created data volumes of extraordinary size and dimensionality. There
|
|
%is a great demand for analysis and information extraction from data.
|
|
%Two approaches are common in machine learning, the parametric one
|
|
%\cite{bishop2006pra, hastie2001esl, haykin1994nnc} and
|
|
%the non-parametric one \cite{wasserman2006ans}. In the parametric case there is an underlying model
|
|
%assumption. The concept is to estimate the model parameters given
|
|
%the data. In the non-parametric case the data is the model.
|
|
|
|
In the parametric case we start with a very large amount of data $N$
|
|
and we train a model that is described with $k$ parameters, where
|
|
$k<<N$. So evaluation of the model for new data is quite fast, since
|
|
it involves operations that have to do with $k$ parameters. In
|
|
practice, parametric models lead to non-convex optimization problems,
|
|
so it is difficult to find the best parameters given the data. The
|
|
most difficult part is to find the right number of the parameters
|
|
that best fit the data. For example the number of neurons and the
|
|
topology of a neural network is something that can only be
|
|
determined experimentally after testing different combinations given
|
|
a dataset. In any case the value of parametric models should not be
|
|
underestimated since it can achieve very good performance in
|
|
practice even if the best parameters have not been chosen.
|
|
|
|
In the parametric case the data is the model. The model has $N$
|
|
parameters plus one or two that have to be estimated. For example in
|
|
the Kernel Density Estimation problem \cite{Silverman}, the bandwidth $h$ of the
|
|
kernel has to be tuned, i.e.
|
|
\begin{equation}
|
|
\hat{f}(x)=\sum_{i}^{N}e^{-\frac{||x-x_i||}{h}}
|
|
\end{equation}
|
|
There are cases when the bandwidth depends on the point so the above
|
|
equation changes to
|
|
\begin{equation}
|
|
\hat{f}(x)=\sum_{i}^{N}e^{-\frac{||x-x_i||}{h_i}}
|
|
\end{equation}
|
|
|
|
\insfig[2d_kernel_density_estimation]{2d_kernel_density_estimation.eps}{Kernel
|
|
Density of 2 dimensional points. Over every point a kernel is set.
|
|
In areas with with high concentration of points there is more
|
|
overlap on the kernels which shows higher probability}{8cm}
|
|
|
|
In contrast to parametric cases, a single evaluation of a non
|
|
parametric model requires evaluation of $N$ parameters. Naively
|
|
speaking training can be avoided at all if the parameter is set
|
|
ad-hoc. If we want to find the single parameter with cross
|
|
validation then that would require $N^2$ operations per candidate
|
|
value of $h$. However if data are in a more sophisticated way
|
|
it is possible to reduce the training of the non-parametric model to
|
|
linear time and the evaluation of a new point in logarithmic time.
|
|
Estimation of the few parameters of a non-parametric model can be
|
|
casted as an optimization problem which in most of the cases can be
|
|
convex. In addition, the non-parametric Kernel Principal Component
|
|
Analysis \cite{scholkopf1998nca} can be computed with the help
|
|
of semidefinite programming \cite{vandenberghe1996sp, boyd2004co}.
|
|
|
|
\section{Origin and history of the problem}
|
|
\label{origin}
|
|
|
|
After the information explosion, the problem of reducing/estimating
|
|
the real dimensionality, also known as the intrinsic dimensionality of a
|
|
given dataset, has been an active area of research \cite{kegl2003ide}, \cite{costa2003egm}.
|
|
It is well known that many
|
|
machine learning algorithms starting from the very fundamental, the
|
|
nearest neighbor, grow exponentially on the intrinsic dimensionality of the data. This
|
|
is also known as the curse of dimensionality \cite{donoho2000hdd}.
|
|
In the literature, dimensionality is used without the prefix intrinsic or
|
|
extrinsic, causing confusion. The extrinsic dimensionality is just
|
|
the number of variables needed to describe a point, i.e. the
|
|
extrinsic dimensionality of $x\in \Re^d$ is just $d$. Each variable
|
|
of the d-dimensional vector will be called dimension $1\dots d$. In
|
|
reality, some of the dimensions might be linearly or non-linearly
|
|
dependent on other dimensions, thus fewer than $d$ dimensions are
|
|
sufficient to describe a point. We can define the intrinsic
|
|
dimensionality of a dataset as the minimum number of "independent"
|
|
variables needed to describe a point in space. Still the term is not
|
|
very clear, for the scope of this thesis we mean that there is no
|
|
algorithm that can predict a dimension from the others. Principal
|
|
Component Analysis for example \cite{jolliffe2002pca}, finds the most important dimensions
|
|
on a dataset that are linearly independent. In practice because of
|
|
noise all dimensions appear to be strictly independent. Modern
|
|
kernel methods can capture non-lineal dependencies between
|
|
dimensions. Kernel PCA is the father of non-linear dimensionality
|
|
reduction methods \cite{scholkopf2002lks}.
|
|
|
|
Although Kernel PCA seems to be a very powerful tool, it suffers from
|
|
polynomial complexity. It requires $O(N^2)$ kernel computations,
|
|
which make it unsuitable for large scale problems. In this thesis we
|
|
will show how multidimensional trees can reduce
|
|
the complexity, along with other approximations to bring it down to
|
|
linear \cite{gray2000nbp}.
|
|
|
|
The driving force for investigating dimensionality reduction is the
|
|
estimation of the true dimensionality of speech. Many representations
|
|
have been recommended for speech including MFCC, LPC, NRAFF etc, \cite{quatieri2002dts,rabiner1978dps, ravindran:inr}, yet the true
|
|
dimensionality is unknown. Meanwhile there is theoretical evidence
|
|
that speech signals lie on manifolds
|
|
\cite{alder1991dss, sciamarella1999tsc, kumar1990ade, tishby1990dsa, jansen2006ifa}. In
|
|
this thesis we will attempt to estimate the true dimensionality of
|
|
speech based on several representations which may help in developing new methods for speech
|
|
recognition.
|
|
|
|
\section{Preliminary Research}
|
|
\label{prelim}
|
|
\subsection{The Nearest Neighbor Problem}
|
|
\label{The_Nearest_Neighbor_Problem}
|
|
|
|
One of the classical problems in statistics and in machine learning
|
|
is the nearest neighbor problem. Given a set of points in $S =
|
|
\{x_i, x_i\in\Re^d, i=0\dots N\}$ and a distance metric $M(x_i, y_i),
|
|
x_i, y_i \in S$, the nearest neighbor of $x_i \in \Re^d$ is
|
|
$argmin(M(x, y_i)), y_i \in S$. In many cases the problem of finding
|
|
all nearest neighbors appears. This problem can be formalized as
|
|
find $argmin(M(x, y_i)), y_i \in S, \forall x \in S'$. The problem
|
|
of finding nearest neighbor can be solved in a "naive" way, by
|
|
computing all the distances between the query point and the
|
|
reference points. The complexity is $O(N)$. Assuming that the number
|
|
of the query points $N$ is the same as the number of reference
|
|
points then the complexity of the naive method is $O(N^2)$. An
|
|
efficient way to reduce the complexity is to use multidimensional
|
|
trees. In the next sections we present kd-trees \cite{moore-tutorial}
|
|
and ball-trees \cite{moore2000ahu}.
|
|
|
|
As a clarification we mention that this is the problem of the exact
|
|
nearest neighbor. There are algorithms addressing the problem of
|
|
approximate neighbor. Local Sensitivity Hashing is one of the
|
|
algorithms that deal with this problem \cite{datar2004lsh}.
|
|
|
|
\subsubsection{Multidimensional Trees}
|
|
\label{Multidimensional_Trees}
|
|
|
|
The purpose of multidimensional trees is to create a hierarchical
|
|
partition of a dataset so that data points are grouped close to
|
|
their neighbors, which are usually the points of interest, and far
|
|
away enough from other points. It has been proven that there is not
|
|
a globally optimal partition strategy \cite{gyorfi2002dft}
|
|
and it is heavily dependant on the distribution of the data points
|
|
as well as on the dimensionality. In this section we present a particular
|
|
class of multidimensional
|
|
trees, the binary trees and more specifically kd-trees and ball or
|
|
metric trees. The key concept in binary trees is partitioning a set
|
|
recursively in two sets at every step.
|
|
|
|
\subsubsection{Kd-trees}
|
|
\label{Kd_trees}
|
|
|
|
The first multidimensional tree in the literature was the kd-tree \cite{bentley1975bst}. After
|
|
30 years it still performs comparably and in some cases, even
|
|
better than modern trees. At every level a partition (pivot)
|
|
dimension is chosen according to a user defined criterion and the
|
|
data is split in two subsets according to this dimension. Kd-trees
|
|
are off-line trees. In order to build a kd-tree all the data must be
|
|
available before building the tree in contrast to other trees that
|
|
can dynamically add or delete points.
|
|
|
|
Several strategies have been recommended for finding the splitting
|
|
dimension. The most popular are splitting on the dimension with the
|
|
maximum variance or maximum range. However both of these strategies are heuristics and
|
|
there is no underlying theory behind them that guarantees better
|
|
performance of the trees. Another issue once the splitting dimension
|
|
is found, is the actual numeric value of the split value. In other words,
|
|
the ratio of the points going in the two resulting sets. The mid
|
|
value of the range or the mid value of the variance will in general
|
|
give two sets that in most of the cases will have different number
|
|
of points. Another strategy is to find the value that gives equally
|
|
sized sets. This strategy in general gives balanced trees, while the
|
|
first one leads to unbalanced trees. In the following sections we
|
|
show that empirically it is preferable to have unbalanced splits
|
|
on the tree, since it favors pruning in the process of finding the
|
|
nearest neighbors.
|
|
|
|
\paragraph{Data structures for kd-trees}
|
|
|
|
Kd-trees consist of two types of data structures, nodes and
|
|
leafs. The basic data needed for the nodes are the
|
|
hyper-rectangles fig.~\ref{parent_kd_node}. Hyper-rectangles are the k-dimensional rectangles that describe the minimum bounding box for a given set of points.
|
|
Leafs hold the data points fig.~\ref{2_kd_node}. The number of points on a
|
|
leaf is user defined. The tree method for finding the nearest neighbor is faster than the naive
|
|
only after a threshold of points. This threshold is used as the maximum number of points
|
|
per leaf. Bounding boxes (Hyper-rectangles) and points are the minimum amount
|
|
of information required to be stored on nodes on the leaf. Extra
|
|
information about the data, such as the centroid of the box, higher
|
|
order moments, etc also called Cached Statistics can speed up
|
|
tremendously algorithms other than nearest neighbor. The algorithm
|
|
of nearest neighbor search with kd-trees will be described in the following section,
|
|
fig.~\ref{nearest_neighbor_searchon_kd}.
|
|
|
|
\begin{figure}[!htb]
|
|
\label{parent_kd_node}
|
|
\centerline{\includegraphics[height=8cm]{parent_kd_node.eps}}
|
|
\caption{The parent node of a 2
|
|
dimensional kd-tree}
|
|
\end{figure}
|
|
|
|
%\insfig[parent_kd_node]{parent_kd_node.eps}{The parent node of a 2
|
|
%dimensional kd-tree}{8cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\label{2_kd_node}
|
|
\centerline{\includegraphics[height=8cm]{2_kd_node.eps}}
|
|
\caption{A two node 2 dimensional
|
|
kd-tree}
|
|
\end{figure}
|
|
|
|
%\insfig[2_kd_node]{2_kd_node.eps}{A two node 2 dimensional
|
|
%kd-tree}{8cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\label{nearest_neighbor_searchon_kd}
|
|
\centerline{\includegraphics[height=8cm]{nearest_neighbor_searchon_kd.eps}}
|
|
\caption{Nearest neighbor with pruning}
|
|
\end{figure}
|
|
|
|
%\insfig[nearest_neighbor_searchon_kd]{nearest_neighbor_searchon_kd.eps}{Nearest
|
|
%neighbor with pruning}{8cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\label{nearest_neighbor_searchon_kd_true_neighbor}
|
|
\centerline{\includegraphics[height=8cm]{nearest_neighbor_searchon_kd_true_neighbor.eps}}
|
|
\caption{True nearest neighbor is out of the leaf}
|
|
\end{figure}
|
|
|
|
%\insfig[nearest_neighbor_searchon_kd_true_neighbor]{nearest_neighbor_searchon_kd_true_neighbor.eps}
|
|
%{True nearest neighbor is out of the leaf}{8cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\label{nearest_neighbor_searchon_kd_no_prune}
|
|
\centerline{\includegraphics[height=8cm]{nearest_neighbor_searchon_kd_no_prune.eps}}
|
|
\caption{Kd tree with no possible pruning}
|
|
\end{figure}
|
|
|
|
%\insfig[nearest_neighbor_searchon_kd_no_prune]{nearest_neighbor_searchon_kd_no_prune.eps}{
|
|
%Kd tree with no possible pruning}{8cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\label{kd_pathological_points}
|
|
\centerline{\includegraphics[height=8cm]{kd_pathological_points.eps}}
|
|
\caption{A two dimensional pathological kd-tree}
|
|
\end{figure}
|
|
|
|
%\insfig[kd_pathological_points]{kd_pathological_points.eps}{A two
|
|
%dimensional pathological kd-tree}{8cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\label{pathological_kd_tree}
|
|
\centerline{\includegraphics[height=8cm]{pathological_kd_tree.eps}}
|
|
\caption{In this case the partition is very bad and almost for every point pruning is not
|
|
feasible}
|
|
\end{figure}
|
|
|
|
%\insfig[pathological_kd_tree]{pathological_kd_tree.eps}{In this case
|
|
%the partition is very bad and almost for every point pruning is not
|
|
%feasible}{8cm}
|
|
|
|
|
|
|
|
\subsubsection{Ball trees}
|
|
One of the main disadvantage of kd-trees is that they can only
|
|
handle problems associated with the euclidian metric, or metrics
|
|
that can be cast in euclidian metric such as the hamming distance.
|
|
In many cases although the data is expressed in cartesian
|
|
coordinates and the notion of similarity is beyond the euclidian sense.
|
|
For example the cosine distance is used in many applications of
|
|
machine learning such as document retrieval.
|
|
|
|
Metric or Ball trees \cite{moore2000ahu} give solution to this problem. Instead of using
|
|
hyper-rectangles, they use hyper-spheres. Hyper-spheres are
|
|
generalized k-dimensional spheres:
|
|
\begin{equation}
|
|
x,x_0 \in \Re^k , d(x,x_0)=r, r\in \Re
|
|
\end{equation}
|
|
where $d(x, y)$ defines a distance metric in a k dimensional space.
|
|
|
|
At every split two points that are far away are chosen as anchor
|
|
points. Every other point of the initial set is classified according
|
|
to the minimum distance from the anchor points. The choice of anchor
|
|
points is done heuristically. One method that works in practice is
|
|
choosing a point randomly and then find the furthest point. This is
|
|
the first anchor point. The next anchor point is chosen as the
|
|
furthest point from the first.
|
|
|
|
\subsubsection{Nearest Neighbor Algorithm}
|
|
The purpose of using a tree in the nearest neighbor problem is to
|
|
gradually find bounds for the neighbor distance and prune parts of
|
|
the search space. In general trees give bounds for data partitions
|
|
that can be used in other problems that we will see in future
|
|
sections, such as fast gaussian summation.
|
|
|
|
Initially we start the search from the root node. At each step we
|
|
recurse down to the child node that contains the query point.
|
|
In kd-trees we check the pivot dimension of the query point and if
|
|
it is smaller than the pivot value we recurse on the left child
|
|
otherwise on the right. In ball trees we recurse to the child that
|
|
is closer to the query point. The recursion continues until we reach
|
|
a leaf. Then the distances of the query point and the points on the
|
|
leaf are computed. The closest point is a candidate nearest
|
|
neighbor. Moreover we know that the actual nearest neighbor can not
|
|
be further than the current nearest neighbor. So in the end of the
|
|
top down recursion we have an upper bound for the nearest neighbor.
|
|
Note that the top down recursion is logarithmic to the number of
|
|
points assuming a balanced tree. The next step is to backtrack. At
|
|
each step in backtracking we check if the candidate nearest neighbor
|
|
is in the current leaf/node. There are two ways to check that. First
|
|
of all when we are on a leaf we check whether the hyper-sphere
|
|
centered around the query point with radius equal to the distance
|
|
between the query point and the candidate nearest neighbor
|
|
$d_{min}$ intersects the bounding box. If not then this means it is
|
|
the nearest neighbor. If it does, then we have to check all the
|
|
bounding boxes that intersect the hyper-sphere. For kd-trees and
|
|
ball trees it is fairly easy to detect that.
|
|
|
|
In general for natural data-sets where data tend to cluster, it is
|
|
possible to heavily prune and achieve logarithmic performance in
|
|
search. In some cases when the partitioning is not very good, trees
|
|
behave poorly and eventually have to search almost the whole
|
|
data-set and they behave worse than the naive method since they
|
|
have extra recursion overhead.
|
|
|
|
\begin{itemize}
|
|
\item Recurse to the leaf that contains the point. To do that
|
|
just compare at every node the split value of the splitting
|
|
dimension of the point and recurse to the right direction
|
|
\item On the leaf compute the nearest neighbor with the naive
|
|
method. The distance to the nearest point on the leaf (candidate nearest neighbor)
|
|
is the upper bound for the nearest neighbor distance $r_{max}$,
|
|
fig~\ref{nearest_neighbor_searchon_kd}.
|
|
\item If the sphere centered on the query point and radius
|
|
$r_{max}$ doesn't intersect the bounding box terminate. The
|
|
candidate nearest neighbor is the actual nearest neighbor.
|
|
\item If not, then backtrack and visit all subtrees recursively
|
|
to the leaf and update the candidate nearest neighbor if you find
|
|
a point that is closer than $r_{max}$, fig~\ref{nearest_neighbor_searchon_kd_no_prune}.
|
|
\item Every time you backtrack check if the sphere centered on
|
|
the query point with the radius $r_{max}$ doesn't intersect the
|
|
bounding box terminate.
|
|
\end{itemize}
|
|
|
|
|
|
\subsubsection{All nearest Neighbors}
|
|
The problem of computing all nearest neighbors appears often in many
|
|
machine learning problems, usually in the training phase. As we will
|
|
see in the following sections the formation of the kernel matrix
|
|
involves all nearest neighbor computations. The all-nearest neighbor
|
|
problem is a special case of a more general class of problems called
|
|
N-body problems.
|
|
|
|
Having in mind the tree based algorithm it is natural to assume that
|
|
finding all nearest neighbors can be solved by executing N nearest
|
|
neighbor queries resulting in an $O(N\log N)$ complexity. We will
|
|
call this as the single tree all nearest neighbor algorithm. However
|
|
it is not difficult to notice that there are redundant computations.
|
|
For example consider a query set stored on a kd-tree and a
|
|
reference set stored on another kd-tree as well. If we do the top
|
|
down recursion for a point then we notice that most likely this top
|
|
down recursion might be exactly the same for all the points in the
|
|
same query leaf. It is also possible that the top down recursion
|
|
might be the same for a whole query subtree fig.~\ref{dualkdtree}.
|
|
|
|
This observation leads us to the conclusion that instead of
|
|
comparing distances between points in the search it is more useful
|
|
to compare bounding boxes. The algorithm is a four-way recursion fig.~\ref{dualtree_algorithm}.
|
|
In general the dual tree algorithm as we call it since the query and
|
|
the reference set lie on trees (they might share the same tree if
|
|
the query and the reference set are the same), gives linear
|
|
complexity over the number of data. This is empirical complexity. In
|
|
fig.~\ref{dual_tree} we give an example of dual-tree pruning.
|
|
|
|
\begin{figure}[!htb]
|
|
\label{dual_tree}
|
|
\centerline{\includegraphics[height=8cm]{dual_tree_box_pruning1.eps},
|
|
\includegraphics[height=8cm]{dual_tree_box_pruning.eps}}
|
|
\caption{The query node in red after top down recursion ends up in a leaf of
|
|
the reference tree. Then every point in the query node (red) finds
|
|
with the naive method its candidate nearest neighbor. Then for all
|
|
of them we compare all the candidate nearest distances and find the
|
|
maximum $r_{max}$. Now we know that if there is any node in distance
|
|
greater than $r_{max}$ there is no point in checking for candidate
|
|
nearest neighbors. As we see in the right figure the dashed box
|
|
doesn't intersect with the bounding box of the leaf. This means that
|
|
we can prune the yellow boxes.}
|
|
\end{figure}
|
|
|
|
%\instbfig[dual_tree]{dual_tree_box_pruning1.eps}{dual_tree_box_pruning.eps}
|
|
%{The query node in red after top down recursion ends up in a leaf of
|
|
%the reference tree. Then every point in the query node (red) finds
|
|
%with the naive method its candidate nearest neighbor. Then for all
|
|
%of them we compare all the candidate nearest distances and find the
|
|
%maximum $r_{max}$. Now we know that if there is any node in distance
|
|
%greater than $r_max$ there is no point in checking for candidate
|
|
%nearest neighbors. As we see in the right figure the dashed box
|
|
%doesn't intersect with the bounding box of the leaf. This means that
|
|
%we can prune the yellow boxes.}{8cm}{8cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\begin{boxedminipage}[c]{\linewidth}
|
|
\begin{verbatim}
|
|
recurse(q : KdTree, r : KdTree) {
|
|
if (can_prune(q, r)) {
|
|
summarize(q, r) /* prune */
|
|
} else if (IsLeaf(q) and IsLeaf(r)) {
|
|
do_all_pairs(q, r)
|
|
/* at leaves we must resort to n^2 */
|
|
} else if (<should recurse on q>) {
|
|
recurse(q.left, r)
|
|
recurse(q.right, r)
|
|
} else {
|
|
recurse(q, r.left)
|
|
recurse(q, r.right)
|
|
}
|
|
}
|
|
\end{verbatim}
|
|
\end{boxedminipage}
|
|
\caption{Pseudo-code for the dual-tree all nearest neighbor
|
|
algorithm} \label{dualtree_algorithm}
|
|
\end{figure}
|
|
|
|
|
|
\begin{figure}[!htb]
|
|
\label{kdtree}
|
|
\centerline{\includegraphics[height=6cm]{kdtree.eps}}
|
|
\caption{A two dimensional kd-tree}
|
|
\end{figure}
|
|
|
|
%\insfig[kdtree]{kdtree.eps}{A two dimensional kd-tree}{8cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\label{dualkdtree}
|
|
\centerline{\includegraphics[height=4cm]{dualtree_recursion.eps}}
|
|
\caption{Simulation of the dual tree algorithm}
|
|
\end{figure}
|
|
|
|
%\insfig[dualkdtree]{dualtree_recursion.eps}{Simulation of the dual
|
|
%tree algorithm}{8cm}
|
|
|
|
\subsection{Large scale trees, for out of core memory}
|
|
\label{Large_scale_trees}
|
|
|
|
Kd-trees have been proven to be quite successful in
|
|
multidimensional indexing and outperform several other
|
|
trees in statistical processing of data, mainly in the all nearest
|
|
neighbor problem, n-point correlation etc. In this section we
|
|
address different implementations of kd-trees for data that don't
|
|
fit in the RAM. We show whether virtual memory behaves better than
|
|
paging of nodes on the disk.
|
|
|
|
Kd-trees have been used quite successfully in statistics for
|
|
speeding up statistical algorithms (nearest neighbor, 2 point
|
|
correlation) over large datasets \cite{gray2000nbp}. Many other trees like ball trees,
|
|
vantage trees, cover trees etc \cite{samet2005fma} have been developed as rivals but
|
|
they haven't really shown any significant improvement, their
|
|
performance is most of the times comparable to the kd-trees. After
|
|
30 years they still remain the state of the art for developing
|
|
statistical algorithms. In most of the cases kd trees have been used
|
|
for datasets that fit in the main memory.
|
|
|
|
During the same period the database community developed structures
|
|
for multidimensional indexing mainly used for database applications.
|
|
Some examples of successful tree structures are R-trees,
|
|
$R^{*}$-trees, X-trees and TV-trees \cite{samet2005fma}. The main incentive for these
|
|
trees was to handle large volumes of data, dynamic insertions,
|
|
deletions and random accesses.
|
|
|
|
The size of the datasets for statistical processing is increasing
|
|
faster than the availability of main memory so it is necessary to
|
|
investigate the performance of kd-trees and various optimizations
|
|
over cached implementations. Unfortunately R-tree like structures
|
|
cannot be used or even compared to kd-trees since they are optimized
|
|
for different problems. It turns out that batch loading of spatial
|
|
trees can improve their performance. There are splitting strategies
|
|
in some of them (X-trees) that can guarantee non overlapping
|
|
hyper-rectangles which is considered to be a good thing for the
|
|
nearest neighbor performance. Eventually kd-trees can be considered
|
|
as a special case of spatial trees, so there is no point in
|
|
comparing them.
|
|
|
|
In this section we consider some caching implementation strategies for
|
|
kd-trees. The first approach is the use of virtual memory through
|
|
memory mapped files. The basic advantage is the the programming
|
|
simplicity, since data can be accessed through normal pointers. The
|
|
operating system takes care of the page faults. The second approach
|
|
is to take care of the cache so that we can use bigger page sizes
|
|
than the default 4096 in Linux. Page fault is handled on the
|
|
application level. In order to minimize the io cost between disk and
|
|
main memory we used the TPIE library that does efficient block
|
|
transfers.
|
|
|
|
Another parameter that affects the performance of the trees is the
|
|
layout of the tree on the memory, that has to be in such a way that
|
|
it guarantees good locality in terms of the access pattern of some
|
|
basic kd-tree algorithms, such as all nearest neighbors, n-point
|
|
correlation. There are different ways to build a kd-tree: a)Depth
|
|
first b)Breadth first and c)The combination of them building k-depth
|
|
trees breadth first. It has been shown experimentally that
|
|
unbalanced kd-trees tend to give better pruning performance, we also
|
|
consider balanced tree splits since they can give better locality.
|
|
|
|
The last implementational issue is prefetching. Queries in
|
|
databases are random most of the time so prefetching doesn't really
|
|
make sense. On the other hand statistical algorithms on trees
|
|
generate queries that are highly correlated, so we can take
|
|
advantage of that. Taking advantage of the multicore architecture it
|
|
is possible to do non blocking system call for prefetching.
|
|
|
|
We do not address the problem of building cache
|
|
efficiently trees since this issue has been extensively addressed in
|
|
the literature, with the most successful algorithm the buffer tree
|
|
\cite{arge2003btt}.
|
|
|
|
\subsubsection{Memory layout of Kd-trees} There are many ways to
|
|
build a kd-tree, depth first, breadth first and the combination for
|
|
both. It turns out that breadth first does not give good locality
|
|
for dual-tree algorithms. Another parameter of
|
|
kd-trees that has effect in the cache performance is the splitting
|
|
algorithm. Mid-point splits guarantee balanced trees, but do not
|
|
always give good performance in terms of punning.
|
|
|
|
\subsubsection{System architecture} There are several ways to
|
|
implement a disk kd-tree. As it has been stated above, speed is an
|
|
essential feature that we are particularly interested as well as
|
|
portability. Since building the tree for large scale data might take
|
|
a lot of time it is very important to be able to reload the tree and
|
|
perform other operations. Although it is desirable to keep the tree
|
|
as generic as possible so that we can use it for different
|
|
applications this might sacrifice the performance. In our
|
|
implementation we avoided virtual functions, structures that require
|
|
memory reallocations. In order to keep it generic we used simple
|
|
concepts of C++ template programming. Another issue that is of great
|
|
importance is memory locality. There are five basic memory
|
|
structures involved in a kd-tree, nodes, leafs, data points node
|
|
cached statistics and hyper-rectangles. Some of them can be combined
|
|
but it is always good to preserve the abstraction hierarchy Tree,
|
|
Node, bounding Box, Node Statistics, and Points. There is also
|
|
another one which can be treated separately, the result. This can be
|
|
the nearest neighbor(s), the kernel density estimation etc. They can
|
|
be stored either on the tree or on external structures linked from
|
|
the tree nodes. There are two major ways to lay out the tree on the
|
|
memory. The first method is to use a separate cache for each data
|
|
structure and the other one is to use a single cache for
|
|
everything. In some cases the output of the algorithm might as well
|
|
be orders of magnitude larger than the actual tree. For example
|
|
k/range nearest neighbors can give pretty big result sets for large
|
|
k or range.
|
|
|
|
\subsubsection{Memory Manager Architecture} In order to test
|
|
different memory manager strategies, we used smart pointers that
|
|
would collaborate with their corresponding memory allocators. Memory
|
|
Allocators implement a set of basic functions such as
|
|
{\verb"malloc" }.
|
|
One of the fundamental issues is to make sure the allocated memory for each
|
|
structure had the correct alignment. It is widely known that wrong
|
|
memory alignment can decrease the performance or even cause a
|
|
program crash. an easy and portable way to compute the memory
|
|
alignment is to use the simple macro depicted in \ref{Alignment}. Another issue
|
|
that has to be addressed is page fault handling. In the following
|
|
section we describe two different memory managers we implemented and
|
|
we compare their performance.
|
|
|
|
\begin{figure}[!htb]
|
|
\label{Alignment}
|
|
\begin{boxedminipage}[c]{\linewidth}
|
|
\begin{verbatim}
|
|
template <typename T>
|
|
struct Tchar {
|
|
T t;
|
|
char c;
|
|
} ;
|
|
#define strideof(T) \
|
|
((sizeof(Tchar<T>) > sizeof(T)) ? \
|
|
sizeof(Tchar<T>)-sizeof(T) : sizeof(T))
|
|
\end{verbatim}
|
|
\end{boxedminipage}
|
|
\caption{Computing the alignment of a struct}
|
|
\end{figure}
|
|
|
|
\paragraph{Memory Mapped Files} In this approach we used a single memory
|
|
mapped file big enough to fit the tree. The length of the file has
|
|
to be known in advance, but it is not hard to estimate it. The memory is
|
|
allocated linearly and since there are no reallocations or deletions
|
|
there is no need to worry about memory fragmentation. Paging is
|
|
taken care by the operating system. The page size is fixed 4096B for
|
|
Linux. Customizing the page size is something desirable but this
|
|
shouldn't be of great worry since very soon Linux will accommodate
|
|
for variable page size \cite{winwood2002mps}. This implementation is pretty simple and
|
|
memory access is achieved with normal pointers. It is also efficient
|
|
since all the paging is done in lower levels of the operating system
|
|
|
|
\paragraph{User defined cache, with TPIE} In this implementation a block of
|
|
memory is pre-allocated and page faults are handled in an LRU sense,
|
|
while the block transfers between main memory and the disk are done
|
|
with the TPIE engine. TPIE is a portable library for efficient block
|
|
transfers. It 's highly abstracted to accommodate for different
|
|
transfer models. It is currently implementing memory mapped mode,
|
|
Linux read/write and C fread/fwrite. In the future it will implement
|
|
the parallel disk model as well. The advantage of this memory
|
|
manager is that we can control/define several features, such as page
|
|
size, page replacement algorithm and disk/memory transfers. The main
|
|
disadvantage is the address translation that has to be done on the
|
|
application level and slows down the system. In order to minimize
|
|
this effect we sacrificed a little bit of the programming interface
|
|
simplicity. So the address translation is explicitly done in the
|
|
initialization of the pointer. So the pointer includes a universal
|
|
address (integer) and a pointer to the main memory. Upon request the
|
|
internal pointer p points to the data in the main memory. As it is
|
|
obvious this particular memory segment might be paged out during the
|
|
execution. To avoid that the programmer explicitly locks the
|
|
particular page as long as the pointer is in use and then it is
|
|
explicitly unlocked when the data pointing to is not necessary
|
|
anymore. This policy is always prone to deadlocks if the cache
|
|
size is small and the page size is large. Another possible case of
|
|
deadlock is when the tree building algorithm does not guarantee good
|
|
locality in the access pattern of the dual-tree algorithm. In
|
|
practice we never came across a deadlock, since depth first and
|
|
kdepth first give good enough locality for N-body algorithms. But
|
|
even in the case of bad locality we could make local copies of the
|
|
data and not lock the pages, sacrificing the performance of the
|
|
algorithm. It turns out though that the locking/unlocking strategy
|
|
is quite efficient because it avoids copies and it minimizes the
|
|
cost of address translation. This coding scheme is sensitive to bugs
|
|
because the programmer might accidently keep pages locked, which can
|
|
have an immediate effect in the program performance in the same way
|
|
memory leaks behave. For this reason we trace locks/unlocks in debug
|
|
mode to ensure that the program is free of this type of bugs.
|
|
Another important issue is marking the modified pages so that we
|
|
save unnecessary copies from main memory to disk. We used the
|
|
SIGSEGV library along with mprotect to address efficiently this.
|
|
Pages are write protected in memory and if a write occurs a signal
|
|
is raised and the page is marked as modified.
|
|
|
|
\subsubsection{Results}
|
|
We ran several experiments on a 1GByte Ram machine with data that where at least twice the
|
|
size of the cache (The system would not give more than 700MB to the running process).
|
|
In reality the memory requirements are much larger since there is other overhead stored on the
|
|
tree. The results are also stored as key-value pairs. for example: <point id, minimum distance>.
|
|
The size of memory requirements depends also on the k-parameter of k-nearest neighbors.
|
|
We tested two different strategies,
|
|
\begin{enumerate}
|
|
\item Keep a separate memory mapped output file for the
|
|
results. Every leaf points to this file, it needs to keep a list with the current candidate
|
|
nearest neighbors. Eventually when the algorithm is done, all the results are stores
|
|
on the external file. The results are shown in table \ref{on_a_separate_file}.
|
|
\item Store the results on the leafs and when the algorithm is done visit the leafs and
|
|
dump them on a file, table \ref{on_the_leaf}
|
|
\end{enumerate}
|
|
At this point we would like to stress that the range nearest neighbor problem is much more
|
|
easy in the implementation. Once a point is within the range it can immediately stored on the
|
|
disk, while in the the k-nearest search we know the true k-neighbors after the end of the
|
|
whole dual-tree algorithm.
|
|
|
|
The results state that keeping the results on a separate file breaks the good locality
|
|
behavior of the dual-tree algorithm, giving almost double execution time. The profiling
|
|
reports show almost 40\% less CPU utilization time because of the frequent page faults.
|
|
|
|
In table \ref{mmanager_comparison} we see the performance
|
|
of the custom made memory manager, with variable page size. The performance is almost 50\%
|
|
worse than the memory mapped files. Profiling reports showed that a great overhead is spent
|
|
on the memory translation.
|
|
|
|
\begin{table}[!htb]
|
|
\label{on_a_separate_file}
|
|
\footnotesize{ \centering
|
|
\begin{tabular}{|c|c|c|c|}
|
|
\hline
|
|
% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
|
|
number of points & dimensions & dual-tree time(sec)& initialization (sec)\\
|
|
140000000 & 2 & 2711 & 1730\\
|
|
100000000 & 3 & 4885 & 1338\\
|
|
5000000 & 5 & 8376 & 650\\
|
|
\hline
|
|
\end{tabular}
|
|
\caption{Cache performance for storing results in separate space. Memory mapped files}
|
|
}
|
|
\end{table}
|
|
|
|
\begin{table}[!htb]
|
|
\label{on_the_leaf}
|
|
\footnotesize{ \centering
|
|
\begin{tabular}{|c|c|c|c|}
|
|
\hline
|
|
% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
|
|
number of points & dimensions & dual-tree time(sec) & collection (sec)\\
|
|
140000000 & 2 & 2039 & 836\\
|
|
100000000 & 3 & 2648 & 711\\
|
|
5000000 & 5 & 4131 & 430\\
|
|
\hline
|
|
\end{tabular}
|
|
\caption{Performance for cache that stores everything on the tree. Memory mapped files}
|
|
}
|
|
\end{table}
|
|
|
|
\begin{table}[!htb]
|
|
\label{mmanager_comparison}
|
|
\footnotesize{ \centering
|
|
\begin{tabular}{|c|c|c|c|c|}
|
|
\hline
|
|
number of points & dimension & page size & dual-tree(sec) & collection(sec) \\
|
|
140000000 & 2 & 65536 & 25432 & 2482 \\
|
|
140000000 & 2 & 524288 & 4978 & 2026 \\
|
|
140000000 & 2 & 1048576 & 4438 & 1901 \\
|
|
140000000 & 2 & 4194304 & 4543 & 1819 \\
|
|
100000000 & 3 & 65536 & 8163 & 1603 \\
|
|
100000000 & 3 & 524288 & 6542 & 1499 \\
|
|
100000000 & 3 & 1048576 & 6094 & 1386 \\
|
|
100000000 & 3 & 4194304 & 6618 & 1283 \\
|
|
\hline
|
|
\end{tabular}
|
|
\caption{Cache performance for different page size and different datasets. Custom made cache}
|
|
}
|
|
\end{table}
|
|
|
|
|
|
\subsection{Kernel methods}
|
|
\label{Kernel_methods}
|
|
|
|
The kernel or Gram matrix is an informational representation
|
|
for a set of data. There are different interpretations for the
|
|
kernel matrix. The most mathematical interpretation is the dot
|
|
product matrix. Every element of the kernel matrix is the dot
|
|
product between two data points. It is a positive semidefinite
|
|
matrix and it can be considered as the adjacency matrix of a graph
|
|
where the points are nodes. This graph can also be considered as an
|
|
approximation of the surface that data lie on. Positive
|
|
semidefinitness is the only property required for the gram matrix.
|
|
In most of the methods that we discuss in this section, we prefer
|
|
the interpretation of the dot product matrix, in the general Hilbert
|
|
sense. The dot products in most of the cases are non-linear.
|
|
|
|
\subsubsection{Kernel principal component analysis}
|
|
\label{Kernel_principal_component_analysis}
|
|
|
|
Kernel Principal Component Analysis (KPCA) \cite{scholkopf2002lks}, is an extension of
|
|
principal component analysis where the data are mapped into a
|
|
different feature space through the kernel. In Principal Component
|
|
Analysis the goal is to find the principal components for the
|
|
covariance matrix $C$ for a given dataset $S=\{x_j, x_j\in \Re^d, j=1
|
|
\dots M\}$, where $C=\frac{1}{M}\sum_{j=1}^{M}x_j x_j^T$. We
|
|
define a mapping
|
|
\begin{equation}
|
|
\Phi : x \rightarrow X , x \in \Re^d, X\in \Re^f
|
|
\end{equation}
|
|
usually $f>>d$. So the nonlinear mapping, maps the data in a
|
|
higher dimensional space sometimes in an infinite dimensional space.
|
|
At this point it should be clarified that the mapping increases the
|
|
extrinsic dimensionality of the data not the intrinsic. In kernel PCA we
|
|
want to find the principal components of the covariance matrix
|
|
\begin{equation}
|
|
\tilde{C} = \frac{1}{M}\sum_{j=1}^{M}\phi(x_j)\phi(x_j)^T
|
|
\end{equation}
|
|
It turns out that the mapping $\Phi$ doesn't have to be known
|
|
explicitly, all we need is the kernel that represents the dot
|
|
product.
|
|
\begin{equation}
|
|
k(x, y) = \phi(x)\phi(y)^T
|
|
\end{equation}
|
|
For some kernels there is an analytic factorization while for others
|
|
there is not. Not every function can be a valid kernel (dot
|
|
product). There are conditions that must be fulfilled. Basically for
|
|
every dataset they must give a valid kernel matrix as described in
|
|
previous sections.
|
|
|
|
Kernel PCA has many cousins such as Laplacian eigenmaps
|
|
\cite{Belkin}, Diffusion maps \cite{Lafon}, Isomap \cite{tenenbaum2000ggf},
|
|
Local Linear Embedding \cite{roweis1993ndr},
|
|
Local Tangent Space Alignment \cite{zhang2002pma} etc.
|
|
%Both works present algorithms for computing the Laplace-Beltrami
|
|
%operator on a submanifold $\Gamma$ embedded in $\Re^d$.
|
|
Lafon has introduced a diffusion process on the manifold based on a
|
|
kernel $k(x,y)$ that defines the local geometry on the points that
|
|
belong to the manifold. The choice of the kernel affects essentially
|
|
the results of the diffusion process, since it affects the notion of
|
|
the neighborhood around the points which is critical for the
|
|
creation of the proximity graph of the manifold. In their work
|
|
Belkin \cite{Belkin} and Lafon don't deal with the problem of training the local
|
|
bandwidth of the kernel, assuming that the manifold has been
|
|
sufficiently sampled. Jenssen \cite{jenssen17lpd} has proven that
|
|
there is a theoretical equivalence between the Mercer kernel and the
|
|
kernel in kernel density estimation. Based on that fact we performed
|
|
several experiments using the optimal bandwidth for kernel density
|
|
estimation. The Adaptive Kernel Based Density
|
|
Estimation Algorithm (AKDEA) \cite{Silverman} was chosen. The
|
|
proposed algorithm is a partial solution because it doesn't deal
|
|
with the final objective function directly, which in some
|
|
applications of manifold learning it could be clustering or
|
|
dimensionality reduction.
|
|
|
|
The connection between manifold learning and Kernel Density Estimation (KDE) has also been addressed in \cite{girolami2002osd}
|
|
by using
|
|
orthogonal series density estimation rather than kernel density
|
|
estimation
|
|
|
|
%In section 2 the diffusion operator is reviewed based on the work
|
|
%of Lafon \cite{Lafon}. Section 3 outlines the adaptive kernel
|
|
%estimation algorithm. The last section presents examples of the
|
|
%diffusion operator when the kernel bandwidth is trained.
|
|
|
|
\subsubsection{Geometric diffusion on a manifold}
|
|
Let $(\Gamma,\mu)$
|
|
be a measure space, where $\Gamma$ is a finite set of
|
|
$d$-dimensional points and $\mu$ is a counting measure that
|
|
represents the distribution of the points on the data set. In other
|
|
words $\Gamma$ is a submanifold of $\Re^{d}$. Assume that the
|
|
geometry of $\Gamma$ is defined by a kernel $k(x,y)$. The kernel
|
|
$k(x,y)$ measures the degree of similarity between two points $x,y$.
|
|
The kernel satisfies the following conditions:
|
|
\begin{itemize}
|
|
\item $k$ is symmetric: $k(x,y)=k(y,x)$,
|
|
\item $k$ is positivity-preserving: for all $x,y$ in
|
|
$\Gamma, k(x,y)\geq 0$
|
|
\item $k$ is positive semi-definite: for all bounded
|
|
functions $f$ defined on $\Gamma$,
|
|
\[
|
|
\int_{\Gamma} \int_{\Gamma} k(x,y)f(x)f(y)d\mu(x)d\mu(y)\geq 0\]
|
|
\end{itemize}
|
|
It is assumed that $\Gamma$ is a subset of the
|
|
Euclidean space $\Re^{d}$. So for $x,y\in\Gamma$ the kernel
|
|
(similarity measure) is a function of the Euclidean distance
|
|
$\parallel x-y\parallel:$
|
|
\[
|
|
k(x,y)=\eta\left(\frac{\parallel x-y\parallel}{h}\right)
|
|
\]
|
|
In order to study the geometry of the submanifold $\Gamma$, an
|
|
oriented graph $G$ is formed. Every node corresponds to a data
|
|
point. Let $b(x,y)$ be the associated adjacency matrix, where
|
|
$b(x,y)=1$ if $x$ is in the neighborhood of $y$ and $b(x,y)=0$ if
|
|
$x$ is not in the neighborhood of $y$. The kernel $k$ defines the
|
|
notion of neighborhood between the points.
|
|
|
|
Let $u^2(x)=\int_{\Gamma}k(x,y)d\mu(y)$, then the normalized kernel
|
|
$a(x,y)=\frac{k(x,y)}{u(x)u(y)}$ is stochastic since:
|
|
\begin{equation}\label{kernel}
|
|
\int_{\Gamma}a(x,y)d\mu(y)=1.
|
|
\end{equation}
|
|
|
|
The kernel $a(x,y)$ can be considered as a transition matrix of a
|
|
Markov process on the submanifold $\Gamma$. So any admissible kernel
|
|
can be associated with a random walk on $\Gamma$. The operator
|
|
\[
|
|
Af(x)=\int_{\Gamma}a(x,y)f(y)d\mu(y)
|
|
\]
|
|
is called a diffusion operator. The eigenvalue analysis of the graph
|
|
with the adjacency matrix $a(x,y)$ is the basis for embedding the
|
|
data to a lower dimensional space.
|
|
\[
|
|
a(x,y)=\sum_{j=0}^{M} \lambda_{j}\phi_{j}(x)\phi_{j}(y)
|
|
\]
|
|
where $M$ is the number of data points, $x,y$ are nodes of the graph
|
|
and $\phi_j$ is the $j_{th}$ eigenvector of $a$ that corresponds to
|
|
the $\lambda_j$ eigenvalue.
|
|
|
|
|
|
%\subsubsection{Kernel PCA on a manifold}
|
|
%\label{Kernel_PCA_on_a_manifold}
|
|
%
|
|
%Let $(\Gamma,\mu)$ be a measure space, where $\Gamma$ is a finite
|
|
%set of $N$-dimensional points and $\mu$ is a counting measure that
|
|
%represents the distribution of the points on the data set. In other
|
|
%words $\Gamma$ is a submanifold of $\Re^{n}$. Assume that the
|
|
%geometry of $\Gamma$ is defined by a kernel $k(x,y)$. The kernel
|
|
%$k(x,y)$ measures the degree of similarity between two points $x,y$.
|
|
%The kernel satisfies the following conditions:
|
|
%\begin{itemize}
|
|
% \item $k$ is symmetric: $k(x,y)=k(y,x)$,
|
|
% \item $k$ is positivity-preserving: for all $x,y$ in
|
|
%$\Gamma, k(x,y)\geq 0$
|
|
% \item $k$ is positive semi-definite: for all bounded
|
|
%functions $f$ defined on $\Gamma$,
|
|
%\[
|
|
%\int_{\Gamma} \int_{\Gamma} k(x,y)f(x)f(y)d\mu(x)d\mu(y)\geq 0\]
|
|
%\end{itemize}
|
|
%It is assumed that $\Gamma$ is a subset of the Euclidean space
|
|
%$\Re^{n}$. So for $x,y\ni\Gamma$ the kernel (similarity measure) is
|
|
%a function of the Euclidean distance $\parallel x-y\parallel:$
|
|
%\[
|
|
% k(x,y)=\eta\left(\frac{\parallel x-y\parallel}{h}\right)
|
|
%\]
|
|
%The eigenvalue analysis of the graph with the adjacency matrix
|
|
%$k(x,y)$ is the basis for embedding the data to a lower dimensional
|
|
%space.
|
|
%\[
|
|
% k(x,y)=\sum_{j=0}^{M} \lambda_{j}\phi_{j}(x)\phi_{j}(y)
|
|
%\]
|
|
%
|
|
%where $M$ is the number of data points, $x$ and $y$ are nodes of the
|
|
%graph and $\phi_j$ is the $jth$ eigenvector that corresponds to the
|
|
%$\lambda_j$ eigenvalue.
|
|
%
|
|
%One problem is that exhaustive computation of the kernel will lead
|
|
%to a full matrix, which makes the computation of the eigenvector
|
|
%$O(N^2)$. Fortunately very few $(x,y)$ pairs have a non zero kernel
|
|
%product since it depends on their proximity. So again the
|
|
%computational bottleneck for kernel PCA is the computation of
|
|
%nearest neighbors which can be solved by the dual tree algorithm in
|
|
%linear time.
|
|
|
|
\subsection{Parameter estimation for manifold learning, through density estimation}
|
|
\label{Bandwidth_Tuning}
|
|
Manifold learning turns out to be a very useful tool for many
|
|
applications of machine learning, such as classification.
|
|
Unfortunately the existing algorithms use ad hoc selection of the
|
|
parameters that define the geometry of the manifold. The parameter
|
|
choice affects significantly the performance of manifold learning
|
|
algorithms. Recent theoretical work has proven the equivalence
|
|
between the Mercer kernel learning methods and the kernel in kernel
|
|
density estimation. Based on this fact the problem of kernel
|
|
parameter estimation for manifold learning is addressed based on the
|
|
nonparametric statistical theory estimation. An automatic way of
|
|
determining the local bandwidths that define the geometry is
|
|
introduced. The results show that the automatic bandwidth selection
|
|
leads to improved clustering performance and reduces the
|
|
computational load versus ad hoc selection.
|
|
|
|
\subsubsection{Non parametric density estimation} The problem of
|
|
density estimation of finite data set is very significant for most
|
|
of the well known machine learning algorithms. Lafon has proven that
|
|
the Laplace Beltrami operator on a manifold can be computed without
|
|
the knowledge of the density. Although the density is not necessary
|
|
for the computation of the operator it can be useful in the
|
|
estimation of the parameters of the kernel eq.~\ref{kernel}.
|
|
|
|
The density estimate $f$ of a finite set of data $X={x_i\ni
|
|
\Re^n,i=1,\dots,M}$ can be expressed with the following equation
|
|
\cite{Silverman}:
|
|
\begin{equation}\label{powerdensity}
|
|
f(x)=\frac{1}{M}\sum_{i=1}^{M}\frac{1}{h^d}k\left(\frac{\parallel x-x_i
|
|
\parallel}{h}\right)
|
|
\end{equation}
|
|
The shape of the density function varies significantly with
|
|
different values of $h$, which is the global bandwidth. Higher
|
|
values of $h$ lead to smoothed versions of the density. An
|
|
automatic way to determine the optimum value of $h$ is to maximize
|
|
the Leave One Out Cross Validation (LOOCV) criterion. In other words
|
|
the density is formed according to (\ref{powerdensity}) by leaving
|
|
one point out and then the probability of this point is evaluated
|
|
according to the density. The LOOCV can be mathematically expressed:
|
|
\begin{equation}
|
|
M(h)=\sum_{i=1}^{M}\log\sum_{i\neq j}f_j(x_i)
|
|
\end{equation}
|
|
where $f_j$ is the density computed according to
|
|
(\ref{powerdensity}) by leaving $x_j$ out.
|
|
|
|
Another restriction that can be imposed to the estimated density is
|
|
the minimization of the power of the second derivative of the
|
|
density $\int f''(x)^2$. This restriction tends to give smoother
|
|
densities and avoids the formation of isolated islands of points.
|
|
With the standard optimization techniques $h$ can be chosen.
|
|
|
|
The value of $h$ defines the neighborhood around a point through the
|
|
kernel $k$. For dense areas $h$ has to be small, while in sparse
|
|
areas $h$ must be high since the points are far away from each
|
|
other. This leads to the conclusion that $h$ should be different for
|
|
every point. The density can be written in the form:
|
|
\[
|
|
f(x)=\frac{1}{M}\sum_{i=1}^{M}\frac{1}{(\lambda_i h)^d}k(\frac{\parallel x-x_i
|
|
\parallel}{\lambda_i h})
|
|
\]
|
|
The $\lambda_i$ are called local bandwidths. The local bandwidths
|
|
can be estimated from the following algorithm:
|
|
\begin{itemize}
|
|
\item Find a pilot estimate $\tilde{f}(x)$ that satisfies
|
|
$\tilde{f(x_i)>0, i=1,\dots,M}$
|
|
\item The local bandwidth $\lambda_i$ is given by:
|
|
\[
|
|
\lambda_i=(\tilde{f}(x_i)/g)^{\frac{1}{d}}
|
|
\]
|
|
where g is the geometric mean of the $\tilde{f}(x_i)$:
|
|
\[
|
|
g=\sqrt[M]{\prod_{i=1}^{M}\tilde{f}(x_i)}
|
|
\]
|
|
\item The adaptive kernel estimate $f(x)$ is defined by:
|
|
\[
|
|
f(x)=\frac{1}{M}\sum_{i=1}^{M}\frac{1}{(\lambda_i
|
|
h)^d} k\left( \frac{\parallel x-x_i \parallel}{\lambda_i h} \right)
|
|
\]
|
|
\end{itemize}
|
|
The pilot estimate can be any ad hoc density estimate. A common
|
|
estimate is the \emph{kth nearest neighbor estimate} defined by:
|
|
\[
|
|
\tilde{f}(x)=\frac{1}{Md_k(x)}\sum_{i=1}^{M}k\left(\frac{
|
|
\parallel x-x_i \parallel}{d_k(x)}\right)
|
|
\]
|
|
where $d_k(x)$ is the distance between $x$ and the $kth$ nearest
|
|
neighbor. It turns out that the choice of $k$ doesn't affect the
|
|
values of $\lambda_i$. The adaptive kernel estimate can be improved
|
|
if we iterate the above procedure by replacing in every iteration
|
|
the pilot density estimate with the adaptive kernel estimate
|
|
computed in the last step.
|
|
\subsubsection{Eigenvalue analysis of the diffusion graph}
|
|
In the previous section we saw that the
|
|
non-parametric estimation can define the local bandwidth for every
|
|
point. This information can be used in the construction of the
|
|
adjacency matrix $G$. The local bandwidths as derived from the
|
|
previous section will usually give a graph that has connected
|
|
components, or equivalently the manifold is connected. A safe test
|
|
to prove that is to look into the eigenvalues also known as the
|
|
spectrum of the graph. According to \cite{Cvetkovic} the
|
|
multiplicity of the first eigenvalue reflects the number of
|
|
disconnected components in the graph. There are cases where the
|
|
local bandwidths might not give a connected manifold, then it is
|
|
necessary to increase $h$ until the multiplicity of the first
|
|
eigenvalue becomes 1.
|
|
|
|
\subsubsection{Kernel choice} There are many classes of kernels that
|
|
can be used for the diffusion operator on the manifold. The most
|
|
popular is the Gaussian kernel:
|
|
\[
|
|
k(x)=\frac{1}{(2\pi)^{-\frac{d}{2}}}e^{-\frac{1}{2}xx^T}
|
|
\]
|
|
|
|
The problem with the gaussian kernel is that it is of infinite
|
|
support and it leads to adjacency matrices that are not sparse. It
|
|
would be preferable to use a kernel of finite support. The most
|
|
popular one is the Epanechnikov kernel \cite{Silverman}:
|
|
\begin{equation}
|
|
k(x)=\left\{
|
|
\begin{array}{ll}
|
|
\frac{1}{2}c_{d}^{-1}(d+2)(1-x^Tx) & x^Tx<1 \\
|
|
0 & x^Tx\geq 1 \\
|
|
\end{array}
|
|
\right.
|
|
\end{equation}
|
|
|
|
\subsubsection{Examples} In this section two examples illustrated in
|
|
the two dimensional space are presented and a third one on real
|
|
speech data. The first example fig.~\ref{fig1} contains two clusters
|
|
of data. Each cluster contains 300 points uniformly distributed in a
|
|
unity area square. The Average Nearest Neighbor Distance (ANND) as a
|
|
choice for the bandwidth for the Epanechnikov kernel turns out to be
|
|
inadequate since it leads to a disconnected graph with 4 disjoint
|
|
components. This is probably because the two classes are sparsely
|
|
sampled. The AKDE based on maximization of LOOCV gives on the
|
|
average local bandwidths that are an order of magnitude higher than
|
|
the ANND, and lead to a smoother density. As it is shown in
|
|
fig.~\ref{fig1}b the density is not flat as it was expected to be
|
|
and this is because of the fact that the number of points that
|
|
sample the manifold is small. In fig.~\ref{fig1}c the first non
|
|
trivial eigenvector is depicted discriminating very sharply the two
|
|
classes. In fig.~\ref{fig1.1} the same data set is analyzed with a
|
|
higher global bandwidth. The results are very poor since there is
|
|
high intra-class deviation and the centroids of the two classes are
|
|
closer.
|
|
|
|
The second example fig.~\ref{fig2} illustrates two classes of data
|
|
that lie on two halves of a circle, slightly separated. The clusters
|
|
are highly the sampled around the angle of $90^0$ degrees. This
|
|
fact makes very difficult to chose a universal value for the local
|
|
bandwidths since the notion of the locality is significantly
|
|
different in different areas of the manifold. In fig.~\ref{fig2}b
|
|
the density of the data points is shown, reflecting the high
|
|
concentration of points around the angle of $90^0$. In this case the
|
|
AKDE trains the local bandwidths $\lambda_i$ and the global $h$, so
|
|
that the LOOCV is maximized. The resulting values give a manifold
|
|
graph that is disconnected in two classes. A slight increase in the
|
|
global bandwidth makes the graph connected. The first non-trivial
|
|
eigenvector of the diffusion operator identifies very clearly the
|
|
two classes as shown in fig.~\ref{fig2}c. If the global bandwidth is
|
|
increased by an order of magnitude, then still the first non-trivial
|
|
eigenvector identifies the two classes, but the classes are not
|
|
compact and their centroids are closer fig.\ref{fig3}c. Another
|
|
disadvantage of keeping $h$ large is that the adjacency matrix
|
|
$a(x,y)$ fig.~\ref{fig5}b of the manifold graph becomes dense. This
|
|
increases tremendously the memory requirement and the computation of
|
|
the eigenvectors.
|
|
|
|
In this third example the effect of the local bandwidth on the
|
|
clustering problem of three phoneme classes is presented. The points
|
|
correspond to the Fast Fourier Transform of 20 msec speech frames
|
|
sampled at 8KHz, of the TIMIT database. 200 points from the three
|
|
phoneme classes /aa/,/ih/,/sh/ where chosen. In fig.~\ref{fig4}a the
|
|
data are plotted in 2 dimensions after the dimensionality reduction
|
|
process. It is clear that the diffusion process separates the
|
|
classes very well. In fig.~\ref{fig4}b where the bandwidth is an
|
|
order of magnitude larger than the optimal, the diffusion process
|
|
still separates the classes, but the clusters are not very compact.
|
|
When the local bandwidth is an order of magnitude smaller than the
|
|
optimal, the results are very poor. The corresponding graph
|
|
fig.~\ref{fig5}c shows that the points are isolated and not
|
|
connected.
|
|
|
|
|
|
|
|
|
|
\subsubsection{Discussion} In this section the problem of constructing
|
|
the diffusion graph of a manifold defined by data points in $\Re^d$
|
|
was addressed. The parameters of the kernel that define the geometry
|
|
over the manifold were automatically trained through the estimation
|
|
of the adaptive kernel based density algorithm. Experiments on
|
|
synthetic and real data showed that if the global bandwidth is
|
|
smaller than the one that maximizes the LOOCV, then the graph that
|
|
represents the manifold is disconnected into many spurious
|
|
clusters. On the other hand if the bandwidth is higher, then the
|
|
diffusion operator will still identify the clusters, but their
|
|
distance is smaller and they are not as compact compared to the
|
|
optimal bandwidth. Moreover high bandwidth lead to dense graphs and
|
|
as a consequence more computations.
|
|
|
|
The approach followed is still heuristic, although it has an
|
|
underlying theoretical justification on the work described at
|
|
\cite{jenssen17lpd}. It is the first time that experimental results
|
|
are presented that show the effect of optimal kernel density
|
|
estimation on the manifold learning.
|
|
|
|
%\instbfig[fig1.1]{fig1.1b.eps}{fig1.1c.eps}{The same data with
|
|
%fig.~\ref{fig1}a analyzed with a larger global bandwidth. (a) The
|
|
%estimated density, (b) The first non-trivial
|
|
%eigenvector}{4.5cm}{4.5cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\centerline{\includegraphics[height=4.5cm]{fig1.1b.eps}(a)}
|
|
\centerline{\includegraphics[height=4.5cm]{fig1.1c.eps}(b)}
|
|
\caption{The same data with fig.~\ref{fig1}a analyzed with a
|
|
larger global bandwidth. (a) The estimated density, (b) The first
|
|
non-trivial eigenvector }
|
|
\label{fig1.1}
|
|
\end{figure}
|
|
|
|
%\inssssfig[fig1]{fig1a.eps}{fig1b.eps}{fig1c.eps}{(a) Two clusters
|
|
%of uniformly distributed 2-D points, (b) The estimated density, (c)
|
|
%The first non-trivial eigenvector}{4.5cm}{4.5cm}{4.5cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\centerline{\includegraphics[height=4.4cm]{fig1a.eps}(a)}
|
|
\centerline{\includegraphics[height=4.4cm]{fig1b.eps}(b)}
|
|
\centerline{\includegraphics[height=4.4cm]{fig1c.eps}(c)}
|
|
\caption{(a) Two clusters of uniformly distributed 2-D points,
|
|
(b) The estimated density, (c) The first non-trivial eigenvector }
|
|
\label{fig1}
|
|
\end{figure}
|
|
|
|
%\inssssfig[fig2]{fig2a.eps}{fig2b.eps}{fig2c.eps}{(a) Two clusters
|
|
%of non-uniformly distributed 2-D points, (b)The estimated density,
|
|
%(c) The first non-trivial eigenvector}{6cm}{6cm}{6cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\centerline{\includegraphics[height=6.0cm]{fig2a.eps}(a)}
|
|
\centerline{\includegraphics[height=6.0cm]{fig2b.eps}(b)}
|
|
\centerline{\includegraphics[height=6.0cm]{fig2c.eps}(c)}
|
|
\caption{(a) Two clusters of non-uniformly distributed 2-D points,
|
|
(b)The estimated density, (c) The first non-trivial eigenvector
|
|
\vspace{1.2cm}}
|
|
\label{fig2}
|
|
\end{figure}
|
|
|
|
%\inssssfig[fig3]{fig3a.eps}{fig3b.eps}{fig3c.eps}{The same data set
|
|
%with fig.~\ref{fig2} processed with a bandwidth one order of
|
|
%magnitude larger. Although the first eigenvector still separates the
|
|
%two classes, the classes are closer and not compact (a) Two clusters
|
|
%of non-uniformly distributes 2-D points, (b)The estimated density,
|
|
%(c) The first non trivial eigenvector}{6cm}{6cm}{6cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\centerline{\includegraphics[height=6cm]{fig3a.eps}(a)}
|
|
\centerline{\includegraphics[height=6cm]{fig3b.eps}(b)}
|
|
\centerline{\includegraphics[height=6cm]{fig3c.eps}(c)}
|
|
\caption{The same data set with fig.~\ref{fig2} processed with a
|
|
bandwidth one order of magnitude larger. Although the first
|
|
eigenvector still separates the two classes, the classes are closer
|
|
and not compact (a) Two clusters of non-uniformly distributes 2-D
|
|
points, (b)The estimated density, (c) The first non trivial
|
|
eigenvector }
|
|
\label{fig3}
|
|
\end{figure}
|
|
|
|
%\inssssfig[fig4]{fig4a.eps}{fig4c.eps}{fig4e.eps}{(a) The 3 phoneme
|
|
%classes after the dimensionality reduction with the optimal
|
|
%bandwidth, plotted in two dimensions, (b) The 3 phoneme classes with
|
|
%a larger bandwidth,(c) The 3 phoneme classes with a smaller
|
|
%bandwidth}{6cm}{6cm}{6cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\centerline{\includegraphics[height=6cm]{fig4a.eps}(a)}
|
|
\centerline{\includegraphics[height=6cm]{fig4c.eps}(b)}
|
|
\centerline{\includegraphics[height=6cm]{fig4e.eps}(c)}
|
|
\caption{(a) The 3 phoneme classes after the dimensionality reduction
|
|
with the optimal bandwidth, plotted in two dimensions, (b) The 3
|
|
phoneme classes with a larger bandwidth,(c) The 3 phoneme classes
|
|
with a smaller bandwidth}
|
|
\label{fig4}
|
|
\end{figure}
|
|
|
|
%\inssssfig[fig5]{fig4b.eps}{fig4d.eps}{fig4f.eps}{The diffusion
|
|
%graph for three different local bandwidths (a)Optimal, (b)An order
|
|
%of magnitude larger, (c)An order of magnitude
|
|
%smaller}{6cm}{6cm}{6cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\centerline{\includegraphics[height=6cm]{fig4b.eps}(a)}
|
|
\centerline{\includegraphics[height=6cm]{fig4d.eps}(b)}
|
|
\centerline{\includegraphics[height=6cm]{fig4f.eps}(c)}
|
|
\caption{The diffusion graph for three different local bandwidths
|
|
(a)Optimal, (b)An order of magnitude larger, (c)An order of
|
|
magnitude smaller}
|
|
\label{fig5}
|
|
\end{figure}
|
|
|
|
|
|
\subsection{Preliminary investigation of nearest neighbor search for speech recognition}
|
|
|
|
In this section we present a computationally efficient means of
|
|
characterizing different features for speech processing tasks. The
|
|
presented method can be used to compare the performance of features
|
|
free of interference from back-end recognition engines that might be
|
|
skewed to favor a particular feature. Understanding the intrinsic
|
|
characteristics of features would enable us to make the right choice
|
|
of features for a given task.
|
|
|
|
\subsubsection{Introduction}
|
|
Many features have been proposed for speech recognition, with Mel
|
|
Frequency Cepstrum Coefficients (MFCC) \cite{quatieri2002dts} being
|
|
the most popular. Other features have also been developed claiming
|
|
better performance than MFCC in limited tests over different data
|
|
sets. The most common metric is the speech recognition word accuracy
|
|
over TIMIT or any other data set. Unfortunately most of the speech
|
|
recognition systems are specifically tuned for MFCCs and it is
|
|
difficult to have objective comparison since systems always tend to
|
|
favor them.
|
|
%In this section we will introduce statistical algorithms
|
|
%that can scale easily for large data sets. One very interesting
|
|
%statistical measure for features that can be the basis for many
|
|
%machine learning algorithms is all nearest neighbors, which is a
|
|
%special case for the general N-body problem. The brute force method
|
|
%is of $O(N^2)$ complexity and virtually impossible to scale for
|
|
%large data sets. One way to speed up computations is to order the
|
|
%features on a multidimensional tree, such as kd or ball tree
|
|
%\cite{bentley1975bst}, \cite{moore2000ahu} which provide on the
|
|
%average $O(N\log_2N)$ complexity for all nearest neighbor search. A
|
|
%faster algorithm using 2 trees has made it possible to compute all
|
|
%nearest neighbors in linear time \cite{gray2000nbp}.
|
|
In this section
|
|
we demonstrate the performance of the fast all nearest neighbor
|
|
algorithm over the whole TIMIT database using two different
|
|
features, MFCC and Noise Robust Auditory Features (NRAF)
|
|
\cite{ravindran:inr}. Since the data are phonetically labeled, the
|
|
results are used for evaluating the k-nearest neighbor classifier
|
|
performance. This is also a good measure for the compactness and
|
|
separability of the phonetic classes for these two features.
|
|
|
|
Another fundamental problem in speech generation theory is the
|
|
estimation of the intrinsic dimensionality of the speech manifold. Recent
|
|
theories have proven that although speech is described with high
|
|
dimensional representations, it is actually embedded on a low
|
|
dimensional subspace \cite{jansen2006ifa}. The most easy but
|
|
inaccurate method to estimate the embedding of speech in a low
|
|
dimensional space is Principal Component Analysis (PCA), which
|
|
suffers from many problems, specially because of the presence of
|
|
non-linearities on the data. On the contrary many other methods for
|
|
dimensionality reduction have been proposed. These are kernel PCA
|
|
\cite{scholkopf:nca}, diffusion maps, Laplacian eigenmaps
|
|
\cite{coifman2006dm} etc. All these methods operate on a local level
|
|
and require the computation of the nearest (range or k) neighbors of
|
|
every point. The brute force and single tree methods have made the
|
|
cost prohibitive for large data sets, but the dual tree method makes
|
|
possible to evaluate the proximity graph between the points and
|
|
apply any of the above mentioned methods.
|
|
|
|
\subsubsection{Speech features} In our implementation, MFCC features
|
|
were extracted using the freely available code from \emph{Voicebox}.
|
|
The speech signal was divided into frames of length 25 msec, with 50
|
|
\% overlap and 13 MFCC coefficients (including the zeroth
|
|
coefficient) where extracted from each frame. Twenty-seven filters
|
|
were used to group the frequency bins into critical bands.
|
|
|
|
The NRAF features were derived from a model of the early auditory
|
|
system~\cite{shamma_main}. The input signal was passed through a
|
|
bandpass filter bank. We then performed a difference operation
|
|
between adjacent frequency channels. This is followed by a half-wave
|
|
rectification and a smoothing filter. We compute a discrete cosine
|
|
transform (DCT) of the logarithm of the output of the smoothing
|
|
filter to obtain the NRAFs. The filter bank consisted of 27
|
|
1/4$^{th}$ octave filters tuned from 76 Hz to 6887 Hz. The smoothing
|
|
(temporal integration) was done over 25 msec and sampled at a rate
|
|
of 80 Hz (in order to match MFCCs). As in the case of MFCCs, the
|
|
first 13 coefficients were used for further processing.
|
|
|
|
In both cases the delta and acceleration coefficients were extracted
|
|
and appended to each feature vector, increasing the dimensionality
|
|
from 13 to 39.
|
|
|
|
%\subsubsection{Fast N-body methods for feature comparison} One of the
|
|
%fundamental problems in statistics is the nearest neighbor
|
|
%computation for every point in a dataset. The brute force also known
|
|
%as the naive method requires $O(N^2)$ computations. Ordering the
|
|
%multidimensional data on a binary tree, gives $O(\log (N)$ search
|
|
%time per point and $O(N\log N)$ in total. There is a plethora of
|
|
%binary multidimensional trees ball tree, kd tree, cover tree etc. In
|
|
%our experiments we used the kd tree. A detailed description of kd
|
|
%trees can be found at \cite{moore-tutorial}. The main concept is to
|
|
%recursively partition data in disjoint rectangles. Each tree node
|
|
%contains a k-dimensional rectangle (hyper rectangle) that bounds all
|
|
%the points that are contained in the subtree. Only leafs contain
|
|
%points. In Fig.~\ref{kdtree} an example of a kd tree in 2 dimensions
|
|
%is given, while the algorithm for computing the nearest neighbor for
|
|
%a test point is depicted in table Fig.~\ref{dualtree_algorithm}. The
|
|
%main idea is to recursively find the leaf that contains the query
|
|
%point then compute a candidate nearest neighbor on the leaf. As the
|
|
%algorithm back tracks it determines if it is necessary to search
|
|
%other subtrees for a nearest neighbor.
|
|
%
|
|
%The use of dual trees in the all nearest neighbor search has proven
|
|
%to give significant speed-up in all-nearest neighbor computations.
|
|
%The basic idea is that the whole query set is structured on a tree.
|
|
%In our case the query tree and the reference tree are the same. The
|
|
%core of the algorithm relies on the fact that if the current maximum
|
|
%nearest neighbor distance in the query node is smaller than the
|
|
%distance between the query and the reference node distance then the
|
|
%reference node can be pruned. This saves a huge amount of
|
|
%computation in most of the cases since whole subtrees can be pruned
|
|
%out for a large subset of points at once. The all nearest neighbor
|
|
%search is described algorithmically in Table
|
|
%\ref{dualtree_algorithm}. The complexity of the algorithm, on
|
|
%average, turns out to be linear.
|
|
|
|
|
|
\subsubsection{Experimental Results} In order to illustrate the
|
|
applicability of the algorithms described above, the TIMIT and Noisy
|
|
TIMIT (NTIMIT) database were chosen. Our goal is to test our
|
|
algorithms in terms of scalability and time. Phoneme classification
|
|
with the nearest neighbor classifiers was chosen as the evaluation
|
|
task. The task was repeated for both set of features, namely, MFCC
|
|
and NRAF. We also illustrate the spectrum of kernel PCA for the two
|
|
different features.
|
|
|
|
\subsubsection{All nearest neighbor performance}
|
|
The timing results are illustrated in table \ref{timing}. The all
|
|
20-nearest neighbor method with the dual tree algorithm on kd-trees
|
|
is 30 to 50 times faster than naive in terms of time. It took about
|
|
3.6 hours for MFCCs and 2.2 hours for NRAF for all 20-nearest
|
|
neighbors. It is well known that the pruning performance of the
|
|
trees depends on the intrinsic dimension of the dataset. A simple
|
|
way to get an estimation of the intrinsic dimensionality is the
|
|
eigenvalue distribution of the Principal Component Analysis. As seen
|
|
from Fig.~\ref{pca} NRAFs have a more compact distribution than
|
|
MFCCs.
|
|
|
|
\subsubsection{Comparison of NRAF and MFCC}
|
|
\label{Feature_Comparison}
|
|
Two types of experiments were done for the comparison of the features. The first
|
|
one is the Leave One feature Cross Validation. So for every feature
|
|
we find the k-nearest neighbors and check whether the majority has
|
|
the same label as it \footnote{Instead of using the 61 different
|
|
phonetic classes of TIMIT we used the 39 categories of the CMU
|
|
dictionary}. In tables \ref{kneighborTIMIT} and
|
|
\ref{kneighborNTIMIT}, we demonstrate the performance of MFCC and
|
|
NRAF for TIMIT and NTIMIT for various k. NRAF perform better than
|
|
MFCC, specially in the noisy case.
|
|
|
|
In the second experiment we do Leave-One-Speaker-Out cross
|
|
validation. For every feature we find the k-nearest neighbors that
|
|
don't belong to the same speaker and check if the majority has the
|
|
same label as it. The results are illustrated again in
|
|
\ref{kneighborLOOCVTIMIT} and \ref{kneighborLOOCVNTIMIT}. The
|
|
results drop significantly and it interesting that MFCCs perform
|
|
better than NRAFs. The reason why the performance drops
|
|
significantly is because we haven't done any covariance
|
|
normalization on the data. It is well known in the speech
|
|
recognition literature that mean and variance normalization improves
|
|
recognition as it removes the speaker bias. It turns out that
|
|
variance normalization broadens significantly the PCA spectrum which
|
|
is tightly connected with the performance of kd-trees. We tried
|
|
variance normalization but the all nearest neighbor performance was
|
|
very slow. Another option is the Mahalanobis distance on a metric
|
|
tree, but again the covariance normalization would have an immediate
|
|
effect on the performance.
|
|
|
|
It turns out that variance normalization is a way of decorrelating
|
|
the MFCC/NRAF dimensions, which breaks the manifold structure. This
|
|
is a suggestion that we should move on with higher dimensional
|
|
transformations that remove the speaker bias intrinsically.
|
|
|
|
In fig.~\ref{kernel_pca} the kernel PCA spectrum is illustrated. The
|
|
Gaussian kernel with adaptive local bandwidth was used for the
|
|
20-nearest neighbors. The bandwidth is proportional to the nearest
|
|
neighbor distance and every point is connected with its 20 nearest
|
|
neighbors. It turns out that the kernel PCA spectrum is very wide.
|
|
This is a result of the fact that 20 nearest neighbors are not
|
|
enough to model correctly the manifold. It is an indication that
|
|
more neighbors should be used. We experimented with the bandwidths
|
|
and it turns out that no further improvement can be achieved by
|
|
tuning it. High bandwidths tend to give more compact spectrum, so
|
|
even in the extreme case of infinite bandwidth no more significant
|
|
improvement is shown for 10 nearest neighbors.
|
|
|
|
\subsubsection{Discussion} It has been shown previously that NRAFs
|
|
tend to be more robust to noise compared to
|
|
MFCCs~\cite{ravindran:inr}. From Tables~\ref{kneighborTIMIT}
|
|
and~\ref{kneighborNTIMIT} it is clear that NRAFs do provide
|
|
substantial improvement over MFCCs specially in noisy environments.
|
|
However, it is very interesting to note that MFCCs outperform NRAFs
|
|
in all Leave-One-Speaker-Out cross validation tests. This would
|
|
suggest that MFCCs are better than NRAFs at removing inter-speaker
|
|
variability. Inter-speaker variability usually is the result of
|
|
incorporating pitch information (usually encoded in the higher
|
|
coefficients of the decorrelation transform) and it is very much
|
|
possible that in ensuring NRAFs and MFCCs have the same number of
|
|
channels we may have introduced pitch information into the NRAF
|
|
features. Perhaps using lesser number of NRAF coefficients or
|
|
different filter spacing, the inter-speaker variability could be
|
|
removed.
|
|
|
|
\begin{figure}[!htb]
|
|
\label{pca}
|
|
\centerline{\includegraphics[height=8cm]{timit_pca.eps}}
|
|
\centerline{\includegraphics[height=8cm]{ntimit_pca.eps}}
|
|
\caption{Principal Component Analysis of TIMIT (top) and NTIMIT (bottom) for MFCC (solid line)
|
|
and for NRAF (dashed line).}
|
|
\end{figure}
|
|
|
|
%\instbfig[pca]{timit_pca.eps}{ntimit_pca.eps}{Principal Component
|
|
%Analysis of TIMIT (top) and NTIMIT (bottom) for MFCC (solid line)
|
|
%and for NRAF (dashed line).}{8cm}{8cm}
|
|
|
|
\begin{figure}[!htb]
|
|
\label{kernel_pca}
|
|
\centerline{\includegraphics[height=8cm]{kernel_pca.eps}}
|
|
\caption{Kernel PCA spectrum for TIMIT MFCC features}
|
|
\end{figure}
|
|
|
|
%\insfig[kernel_pca]{kernel_pca.eps}{Kernel PCA spectrum for TIMIT
|
|
%MFCC features}{8cm}
|
|
|
|
\begin{table}[!htb]
|
|
\footnotesize{ \centering
|
|
\begin{tabular}{|c|c|c|c|}
|
|
\hline
|
|
% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
|
|
feature & Naive & Single Tree & Dual Tree \\
|
|
\hline
|
|
MFCC & 100 & 4.3 & 3.6 \\
|
|
NRAF & 100 & 2.9 & 2.2 \\
|
|
\hline
|
|
\end{tabular}
|
|
\caption {CPU time (in hours) for evaluating all 20 nearest neighbors on
|
|
TIMIT database}
|
|
}
|
|
\label{timing}
|
|
\end{table}
|
|
|
|
\begin{table}[!htb]
|
|
\footnotesize{ \centering
|
|
\begin{tabular}{|c|c|c|c|c|c|}
|
|
\hline
|
|
% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
|
|
feature & 1-nearest & 3-nearest & 4-nearest & 7-nearest & 10-nearest \\
|
|
\hline
|
|
MFCC & 62.2\% & 70.4\% & 69.7\% & 65.3\% & 63.4\% \\
|
|
NRAF & 66.9\% & 74.1\% & 73.2\% & 67.3\% & 64.7\% \\
|
|
\hline
|
|
\end{tabular}
|
|
\caption{Nearest k-neighbor classfier for TIMIT database} }
|
|
\label{kneighborTIMIT}
|
|
\end{table}
|
|
|
|
\begin{table}[!htb]
|
|
\footnotesize{ \centering
|
|
\begin{tabular}{|c|c|c|c|c|c|}
|
|
\hline
|
|
% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
|
|
feature & 1-nearest & 3-nearest & 4-nearest & 7-nearest & 10-nearest \\
|
|
\hline
|
|
MFCC & 41.4\% & 55.5\% & 54.5\% & 50.7\% & 49.4\% \\
|
|
NRAF & 55.0\% & 65.1\% & 64.0\% & 57.5\% & 54.5\% \\
|
|
\hline
|
|
\end{tabular}
|
|
\caption{Nearest k-neighbor classfier for NTIMIT database} }
|
|
\label{kneighborNTIMIT}
|
|
\end{table}
|
|
|
|
\begin{table}[!htb]
|
|
\footnotesize{ \centering
|
|
\begin{tabular}{|c|c|c|c|c|c|}
|
|
\hline
|
|
% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
|
|
feature & 1-nearest & 3-nearest & 4-nearest & 7-nearest & 10-nearest \\
|
|
\hline
|
|
MFCC & 44.0\% & 56.3\% & 56.3\% & 54.7\% & 54.6\% \\
|
|
NRAF & 42.2\% & 53.5\% & 53.7\% & 52.4\% & 52.4\% \\
|
|
\hline
|
|
\end{tabular}
|
|
\caption{Nearest k-neighbor classifier for
|
|
Leave One Speaker out Cross Validation in TIMIT database}
|
|
} \label{kneighborLOOCVTIMIT}
|
|
\end{table}
|
|
|
|
\begin{table}[!htb]
|
|
\footnotesize{ \centering
|
|
\begin{tabular}{|c|c|c|c|c|c|}
|
|
\hline
|
|
% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
|
|
feature & 1-nearest & 3-nearest & 4-nearest & 7-nearest & 10-nearest \\
|
|
\hline
|
|
MFCC & 32.0\% & 46.7\% & 46.5\% & ?44.3\% & 43.8\% \\
|
|
NRAF & 30.1\% & 42.9\% & 42.9\% & 41.2\% & 40.1\% \\
|
|
\hline
|
|
\end{tabular}
|
|
\caption{Nearest k-neighbor classifier for Leave One Speaker Out
|
|
Cross validation in NTIMIT database}
|
|
} \label{kneighborLOOCVNTIMIT}
|
|
\end{table}
|
|
|
|
\pagebreak
|
|
\newpage
|
|
\section{Proposed Research}
|
|
\label{proposed}
|
|
The goal of this thesis is to estimate the intrinsic dimensionality of speech relying
|
|
on big databases like TIMIT \cite{garofolo1993tap}, Wall Street Journal and Broadcast
|
|
News \cite{graff1997bns}. Dimensionality reduction will implicitly lead to new feature
|
|
generation. The ultimate goal is to use these new features along
|
|
with kd-trees do speech recognition by nearest neighbor search. The simplest approach to dimensionality
|
|
reduction is Principal Component Analysis (PCA). Unfortunately PCA is a linear method that works well
|
|
only on gaussian and euclidian data. It fails to capture nonlinearities on the data manifold. Simple
|
|
examples of curves like the Swiss roll show that PCA fails to unfold it
|
|
and identify the principal components. Manifold techniques such as Local Linear Embedding, Isomap,
|
|
Local Tangent Space Alignment (LTSA), Kernel PCA, Laplacian Eigenmaps etc are more efficient in capturing
|
|
the principal components of a nonlinear manifold.
|
|
|
|
Fixed Kernel Methods such as Kernel PCA suffer from the kernel bandwidth tuning.
|
|
We saw in section \ref{Bandwidth_Tuning}
|
|
that we can tune it with heuristics. Although it can improve the performance we show in
|
|
section \ref{Feature_Comparison} fig.~\ref{kernel_pca} that the
|
|
choice of the kernel is crucial. A very promising direction as we will see in
|
|
section \ref{Customized_Kernels}
|
|
is to use semidefinite programming in customizing the kernel matrix (Semidefinite PCA). The results of \cite{weinberger2004lkm}
|
|
indicate superior perfoormance compared to standard Kernel Methods as well as Isomap and Local Linear Embedding.
|
|
|
|
All these methods require the k-neighborhood computation for every point. As we have already
|
|
mentioned the computational bottleneck can be reduced from $O(N^2)$ to linear with the dual-tree
|
|
algorithm. Another bottleneck is the computation of the eigenvalues for kernel matrices with
|
|
infinite support like the gaussian. Moreover the Semidefinite PCA suffers from polynomial complexity since it uses the interior
|
|
point methods. There are also some other implementational issues with the tree algorithms that we addressed
|
|
in section \ref{Large_scale_trees}
|
|
|
|
In the next sections we present optimization strategies for the fixed kernel methods as well as for the
|
|
Semidefinite PCA. In the last part the strategy for estimating the intrinsic dimension of speech and its
|
|
application in speech recognition is presented.
|
|
|
|
|
|
\subsection{Fast kernel summation}
|
|
Kernel summation is one of the most common tasks in kernel methods. The general formula
|
|
\begin{equation}
|
|
\label{kernel_sum}
|
|
G(x)=\sum_{r=0}^{N} w_i e^{-\frac{||x-x_r||}{\sigma^2}}
|
|
\end{equation}
|
|
appears in many cases, with the most interesting case the
|
|
eigenvalue/eigenvector computation of the kernel matrix. The kernel summation is a very
|
|
computationally intensive task. For Gaussian kernels there are
|
|
approximations that can speed up computations orders of magnitude.
|
|
We will briefly discuss them here. Apart from computations we can also
|
|
save memory. The gaussian kernel has infinite support, so the
|
|
kernel matrix $\{K : k(x,y)=e^{-\frac{||x-y||}{\sigma^2}} \}$ is not
|
|
sparse. Numerical iterative methods for eigenvector computation are
|
|
based on the general iteration rule $w^n=w^{n-1}+\lambda Kw^{n-1}$, where
|
|
the multiplication of each row of $K$ with $w^{n-1}$ is a kernel
|
|
summation problem. So we don't really need to store the kernel
|
|
matrix, we just have to compute the kernel sum for every row using
|
|
$w^{n-1}$ as a the weight vector. The term $Kw^{n-1}$ can be
|
|
expanded as
|
|
|
|
\begin{equation}
|
|
K_i\footnote{$K_i$ is the $ith$ row of the K matrix} w^{n-1} =
|
|
\sum_{j=1}^{N} K(i,j) w^{n-1}_{j}
|
|
\end{equation}
|
|
|
|
which is the gaussian summation if we replace $K(i,j)$ with the
|
|
Gaussian kernel
|
|
\begin{equation}
|
|
\sum_{j=1}^{N}w^{n-1}_j e^{-\frac{||x_i-x_j||}{\sigma^2}}
|
|
\end{equation}
|
|
|
|
\subsubsection{Gaussian Kernel factorization}
|
|
In general the kernel computation speedup is achieved by factoring
|
|
the kernel \cite{raykar2005fcs} in the form
|
|
\begin{equation}
|
|
k(x_r, x_q)=\sum_{k=1}^{p}\Phi_k(x_r)\Psi_k(x_q)+error
|
|
\end{equation}
|
|
Assuming that $x_r$ stands for the reference points and $x_q$ for
|
|
the query points, the terms that depend on the reference point can
|
|
be precomputed and be used for every query point. This essentially
|
|
reduces the computational complexity to $O(pN+pM)$, where $N$ is the
|
|
number of reference points and $M$ is the number of query points.
|
|
|
|
\paragraph{Factorization with Hermite Polynomials}
|
|
|
|
The gaussian can be approximated with the help of Hermite
|
|
polynomials \cite{strain1991fgt}. The Hermite polynomials $H_n(t)$ are defined by the
|
|
Rodrigues formula:
|
|
\begin{equation}
|
|
H_n(t)=(-1)^ne^{t^2}D^n e^{-t^2}, t\in \Re
|
|
\end{equation}
|
|
where $D=\frac{d}{dt}$.
|
|
|
|
The gaussian can now be expressed as
|
|
\begin{equation}
|
|
e^{-(t-s)^2}=\sum_{n=0}^{\infty}\frac{s^n}{n!}h_n(t)
|
|
\end{equation}
|
|
|
|
where $h_n(t)$ is defined by
|
|
\begin{equation}
|
|
h_n(t)=e^{-t^2}H_n(t).
|
|
\end{equation}
|
|
|
|
Eventually the Gaussian summation
|
|
\begin{equation}
|
|
G(x_q)=\sum_{i=1}^{N}w_i e^{-\frac{||x_r-x_q||}{\sigma^2}}
|
|
\end{equation}
|
|
|
|
can be expressed as
|
|
\begin{equation}
|
|
G(x_q)=\sum_{\alpha\geq 0} A_{\alpha} h_{\alpha}
|
|
\left(\frac{x_r-x_0}{\sigma}\right)
|
|
\end{equation}
|
|
|
|
where
|
|
|
|
\begin{equation}
|
|
A_{\alpha} = \frac{1}{\alpha!}
|
|
\sum_{r=1}^{N}w_r\left(\frac{x_r-x_0}{\sigma}\right)
|
|
\end{equation}
|
|
|
|
The error $E_H(p)$ due to truncating the series after $p^d$ terms is
|
|
bounded by
|
|
|
|
\begin{equation}
|
|
|E_H(p)|\leq (1.09)d\left(\frac{1}{p!}\right)^{\frac{d}{2}}
|
|
\left(\frac{\epsilon^{p+1}}{1-\epsilon}\right)^d
|
|
\end{equation}
|
|
|
|
The basic problem of this method and essentially of any analytical
|
|
factorization method that we know is that the error bound grows
|
|
exponentially to the dimension.
|
|
|
|
Moreover there is another issue. Let's say we want to compute Kernel
|
|
PCA. The goal of KPCA or any other spectral method is to find the
|
|
eigen-spectrum of a kernel matrix
|
|
|
|
\begin{equation}
|
|
K = \sum_{i=1}^{N}\lambda_i ww^T
|
|
\end{equation}
|
|
or to be more specific to express the $(i, j)$ element of the kernel matrix
|
|
\begin{equation}
|
|
k(i,j)=\sum_{i=1}^{N}\lambda_i w_i w_j
|
|
\end{equation}
|
|
|
|
and since $i$ corresponds to point $x$ and $j$ corresponds to $y$
|
|
we can rewrite it as
|
|
\begin{equation}
|
|
k(x,y)=\sum_{i=1}^{N}w(x)w(y)
|
|
\end{equation}
|
|
where $w(x)$ is the $ith$ element of $w$ and $w(y)$ the $jth$ of $w$.
|
|
It turns out that Kernel PCA does the same thing with analytic
|
|
kernel factorization, with the difference that the factorization on
|
|
Kernel PCA is done on a lower dimension since the kernel matrix is
|
|
evaluated on the data that lie on a lower dimensional manifold.
|
|
|
|
So there is no point in using analytic kernel expansion methods when
|
|
dealing with high dimensional data to speed up KPCA since we do many
|
|
redundant computations.
|
|
|
|
|
|
\subsubsection{Computing Gaussian Kernel Matrix with trees}
|
|
Instead of using analytic kernel expansion we revert to the original
|
|
idea of fast kernel summation. The initial approach to fast kernel
|
|
summation with trees was based on a nearest neighbor approach. Find
|
|
the range-$\epsilon$ nearest neighbors for the query point, evaluate
|
|
the kernel on them and sum \cite{gray2003rem}. This method scales better than analytic
|
|
kernel factorization since it depends on the nearest neighbor
|
|
algorithm which depends on the intrinsic dimensionality and not on
|
|
the extrinsic.
|
|
|
|
An interesting issue on KPCA that hasn't really been touched is the
|
|
effect of the truncation error on the Kernel matrix by setting all
|
|
the values less than $k(\epsilon)$ to zero. After searching the
|
|
literature it was found that there is an implicit method to find it.
|
|
|
|
Assume $K$ the exact kernel matrix which is positive definite matrix
|
|
and $\tilde{K}=K+E$ the perturbed matrix. Then let $\lambda_i$ be
|
|
the eigenvalues of $K$ and $\tilde{\lambda}_i$ the eigenvalues of
|
|
$\tilde{K}$. It turns out \cite{stewart1990mpt} that
|
|
|
|
\begin{equation}
|
|
\sqrt{\sum_{i=1}^{N}(\lambda_i-\tilde{\lambda}_i)^2}\leq ||E||_F
|
|
\end{equation}
|
|
where $||E||_F=\sqrt{\sum_{i=1}^{N}\sum_{j=1}^{N}e_{ij}^2}$, is the
|
|
Frobenious norm of $E$.
|
|
So we need a good estimate of $||E||_F$. A simple approach would be
|
|
to set all nonzero elements of $E$ which are the zero elements of
|
|
$K$ as $k(\epsilon)$. An open problem is to estimate a better bound
|
|
for the kernel sum of the furthest neighbors. Trees can give a good
|
|
estimate on that, by just getting information from the bounding
|
|
boxes that are pruned. This can be done by storing information in cached statistics of
|
|
a node about the the mean distance of the points in the node and out of the box.
|
|
|
|
\subsubsection{Applications}
|
|
As we have already mentioned an immediate application of a fast gaussian summation is the
|
|
computation of the eigenvalues of the kernel matrix. Another application is in gaussian process
|
|
regression and methods that relate to it. As a testbed we use the census database.
|
|
|
|
\subsection{Customized Kernels}
|
|
\label{Customized_Kernels}
|
|
There are infinite functions
|
|
that are valid kernels \cite{shawetaylor2004kmp}. The most popular ones are:
|
|
\begin{enumerate}
|
|
\item The gaussian kernel $k(x, y)=e^{-\frac{||x-y||^2}{\sigma^2}}$
|
|
\item The polynomial kernel $k(x, y)=(xy)^n$
|
|
\item The epanechnikov kernel $k(x, y)=\left\{ \begin{array}{cc}
|
|
1-||x-y||^2/\sigma^2 & ||x-y||\leq\sigma \\
|
|
0 & ||x-y||>\sigma
|
|
\end{array} \right.$
|
|
\item The k-nearest neighbor kernel
|
|
\end{enumerate}
|
|
Although these are valid kernels for any kernel method including
|
|
Kernel PCA they are not appropriate for every task. In some cases we
|
|
want to map data in higher dimensional spaces preserving distances
|
|
(isometry) while in other cases we want to make small distances
|
|
smaller and send high distances to infinity (clustering). The
|
|
gaussian kernel for example tends to behave better in clustering
|
|
since it fades out very quickly. On the other hand it fails to
|
|
unfold the data manifold properly so that the intrinsic dimension
|
|
can be estimated. Manifold unfolding is a very important procedure
|
|
since it can reveal the true dimensionality of the dataset.
|
|
|
|
In \cite{weinberger2004lkm} the authors introduced the idea of building the kernel matrix
|
|
from scratch without using any fixed kernel. As we have already
|
|
mentioned the kernel matrix has to be a positive semidefinite
|
|
matrix. Initially we have to define a k neighborhood for every
|
|
point. So at first we have to connect points to their neighbors and
|
|
initialize the non zero values of the matrix to 1. Then we
|
|
maximize the trace of the kernel matrix under the constraint that
|
|
the the distances in the kernel matrix are preserved. This leads to
|
|
a semidefinite programming problem. Here is the algorithm they
|
|
suggest:
|
|
|
|
\vspace{1cm}
|
|
\fbox{
|
|
\begin{minipage}[c]{0.6\linewidth}
|
|
Maximize $Trace(K)$ subject to:
|
|
\begin{enumerate}
|
|
\item $K \geq 0$
|
|
\item $\sum_{i=1}^{N}\sum_{j=1}^{N} K_{ij}=0$
|
|
\item $K_{ii}+K_{jj}-K_{ij}-K_{ji}=G_{ii}+G_{jj}-G_{ij}-G_{ji}$
|
|
for all $i, j$ such that i point is in the neighborhood of j point
|
|
\end{enumerate}
|
|
\end{minipage}
|
|
}
|
|
\vspace{1cm}
|
|
|
|
where $K$ is the kernel matrix, $G$ is the linear kernel matrix,
|
|
$G_{ij}=x_i x_j^T,\quad x_i,x_j\in \Re^d$ and
|
|
$Trace(K)=\sum_{i=0}^{N}K_{ii}$.
|
|
This is a typical semidefinite programming problem that can also be
|
|
posed as
|
|
\vspace{1cm}
|
|
|
|
\fbox{
|
|
\begin{minipage}[c]{0.8\linewidth}
|
|
maximize $Trace(K)$
|
|
\begin{itemize}
|
|
\item subject to $Trace(F_nK)=c_n, n=1,\dots,kN$
|
|
\item $K\geq 0$
|
|
\end{itemize}
|
|
where $F_i$ is a positive semidefinite matrix and in this case
|
|
\begin{itemize}
|
|
\item $F_0=\mathbf{1}$, where $\mathbf{1}$ is an $NxN$ matrix filled
|
|
with ones,
|
|
\item $F_n$ is a sparse matrix with nonzero values (set to 1)
|
|
the $\{i, j\}, \{j,j\}, \{i, j\}, \{j, i\}$ elements. In other words
|
|
$F_n$ represents the link between the $n$ point and its neighbor $m$
|
|
\item $c_n =G_{ii}+G_{jj}-G_{ij}-G_{ji}$
|
|
\end{itemize}
|
|
\end{minipage}
|
|
}
|
|
|
|
\vspace{1cm}
|
|
The inventors of this method named it Semidefinite PCA (SDPCA),
|
|
which is the term we will use form now on.
|
|
|
|
\subsubsection{Speeding up Semidefinite PCA}
|
|
Semidefinite problems are solved with the interior point method \cite{nesterov1994ipp}. The
|
|
basic concept of interior point method is to build a barrier
|
|
function that penalizes solutions that violate the inequality
|
|
constraints. The ideal function would be:
|
|
|
|
\begin{equation}
|
|
I_{-}(u)= \left\{
|
|
\begin{array}{cc}
|
|
0 & u\leq 0 \\
|
|
\infty & u>0
|
|
\end{array}
|
|
\right.
|
|
\end{equation}
|
|
|
|
|
|
It turns out that a good approximation of this function is
|
|
|
|
\begin{equation}
|
|
\hat{I}_{-}(u) = -\frac{1}{t}\log(-u), u\leq 0,
|
|
\end{equation}
|
|
|
|
It turns out that
|
|
|
|
\begin{equation}
|
|
\lim_{t\rightarrow \inf} \hat{I}_{-} = I_{-}
|
|
\end{equation}
|
|
|
|
It turns out that a good barrier function for Semidefinite
|
|
programming is
|
|
\begin{equation}
|
|
\phi(x)=\log \det(K)
|
|
\end{equation}
|
|
where $det(K)$ is the determinant of $K$ matrix.
|
|
|
|
At this stage here it is important to mention that there is a dual
|
|
form of the semidefinite programming posed above which eliminates
|
|
the equality constraint:
|
|
|
|
\vspace{1cm}
|
|
|
|
\fbox{
|
|
\begin{minipage}[c]{0.6\linewidth}
|
|
minimize $c^Tx$ \\
|
|
subject to $F(x)=x_0F_0(x)+x_1F_1+\dots+x_nF_n+I\leq 0$
|
|
\end{minipage}
|
|
}
|
|
|
|
\vspace{1cm}
|
|
|
|
The barrier function for this form is
|
|
|
|
\begin{equation}
|
|
\phi(x)=\frac{1}{t}\log\det(-F(x)^{-1})
|
|
\end{equation}
|
|
|
|
The nice property of this function is that the gradient has the form:
|
|
|
|
\begin{equation}
|
|
\frac{\partial\phi(x)}{\partial x_{n}} = Trace(-F(x)^{-1}F_n)
|
|
\end{equation}
|
|
|
|
The function that we want to minimize including the barrier
|
|
(penalty) function is
|
|
|
|
\begin{equation}
|
|
f(x)=c^Tx+\phi(x)
|
|
\end{equation}
|
|
|
|
and the derivative:
|
|
|
|
\begin{equation}
|
|
\frac{\partial f(x_n)}{\partial x_{n}} = c_n
|
|
+\frac{1}{t}Trace(-F(x)^{-1}F_n)
|
|
\end{equation}
|
|
|
|
If we follow a gradient descent method then we need to compute at
|
|
every iteration $F(x)^{-1}$ since the update of x will be given by
|
|
|
|
\begin{equation}
|
|
x_{n}^{new}=x_{n}^{old}+\lambda (c_n
|
|
+\frac{1}{t}Trace(-F(x)^{-1}F_n)
|
|
\end{equation}
|
|
|
|
where $\lambda$ is the learning rate.
|
|
|
|
There are two approaches on that:
|
|
\begin{enumerate}
|
|
\item Compute $F(x)^{-1}$ once and then do the updates directly on
|
|
$F(x)^{-1}$ using the approximation $(F+ \sigma^2\partial F)^{-1}
|
|
= F^{-1} + \sigma^2 F^{-1}\partial F F^{-1}$, provided $\sigma^2$
|
|
is smaller compared to the Frobenius norms of $Q, \partial F$
|
|
\item As we can see there is no need to compute all the elements
|
|
of $F(x)^{-1}$ since only the trace of $F(x)^{-1}F_n$ is required.
|
|
This means that for every $n$ only a few columns of $F(x)^{-1}$
|
|
are required. They can be computed very quickly by the conjugate
|
|
gradient method \cite{saad2003ims} which is specially designed for sparse positive
|
|
definite matrices \cite{saad2003ims}. This is also a stochastic gradient approach since
|
|
incomplete (noisy) instances of the gradient are computed \cite{spall2003iss}.
|
|
\end{enumerate}
|
|
|
|
\subsubsection{Accelerating Semidefinite PCA with domain
|
|
decomposition}
|
|
|
|
Zhang and Zha \cite{zhang:ddm} proposed a domain decomposition method for fast
|
|
manifold learning. This is a general decomposition method for
|
|
dividing the problem in subdomains, evaluating the spectral
|
|
decomposition and and then glue them. The algorithm is described
|
|
below:
|
|
|
|
\vspace{1cm}
|
|
|
|
\fbox{
|
|
\begin{minipage}[c]{0.8\linewidth}
|
|
Let $K$ be the k-neighborhood matrix. Let $I$ be the set of all
|
|
points. Partition the $K$ into $K_{1}=K(:,I_1)$, $K_{2}=K(:, I_2)$
|
|
so that $I_1 \bigcup I_2 = I$ and $I_1 \bigcap I_2 = I_0
|
|
\neq\emptyset$. Let $K_{01}, K_{02}$ be the the parts of $K_1, K_2$
|
|
that contain the overlap points corresponding to $I_0$
|
|
\begin{enumerate}
|
|
\item Compute the least squares problem $\min_w||K_{01}-W[e,
|
|
K_{02}^T]^T||_F$.
|
|
\item Affinely transform $K_2$ to $\hat{K}_2=W[e,K_{2}^T]^T$.
|
|
\item Set the global coordinate matrix $K$ by \\ $K(:,I_1\backslash I_0)=K_{11}, K(:,I_0)= \alpha
|
|
K_{01}+(1-\alpha)\hat{K}_{02}, K(:, I_2\backslash I_0)=\hat{K}_{12}$
|
|
\end{enumerate}
|
|
\end{minipage}
|
|
}
|
|
|
|
\vspace{1cm}
|
|
|
|
The above method assumes a graph between the points, usually formed with the k-nearest neighborhood
|
|
method. Then the authors use a partitioning method for the graph such as METIS \cite{karypis1995mug}
|
|
or Cuthill-McKee \cite{george1981csl}.
|
|
This adds extra computational cost. It is possible to combine the
|
|
partition of the data and the generation of the graph. This can be done with the use of trees.
|
|
The spectral decomposition (LTSA, Kernel PCA, etc) can be computed on every leaf individually
|
|
and then recursively glue them, following the tree form bottom to top. It is necessary though that
|
|
the trees are overlapping since the method requires the domains not to be disjoint. This would potentially
|
|
lead to a nearest neighbor independent solution which can scale better with the dimensionality.
|
|
Another alternative is to keep the kd-tree in the original form and perform nearest
|
|
neighbor search between the leafs to generate overlapping domains.
|
|
|
|
\paragraph{Monte Carlo for Semidefinite PCA.}
|
|
The authors of Semidefinite PCA have also suggested a speedup of the method focusing more on the
|
|
approximation of the graph by shrinking it \cite{weinberger2005ndr}. In \cite{scholkopf2002lks}
|
|
authors suggest a factorization that it is very similar for accelerating Kernel PCA. We suggest a
|
|
Monte Carlo approach based on a tree partition of the dataset. We can build $m$ kernel matrices
|
|
based on m subtrees and train them with semidefinite programming. Whenever we want to compute the
|
|
real dimensions of a point we have to project it to the other kernel matrices and average. Trees
|
|
can also help in pruning out redundant projections that can give projections close to zero.
|
|
|
|
|
|
\subsection{Speech recognition with nearest neighbor search, Dimensional Analysis of Speech}
|
|
The motivation for this thesis has been to develop a speech recognition engine based on nearest neighbor search. In the age of information explosion it is very easy to collect large collections of annotated speech. Most of the tv broadcasts are captioned. Although there are errors and misalignment, it is always possible to reduce them, with an open source Viterbi decoder, by force aligning them. So it is possible to extract the audio from the tv signal, segment it in word or phoneme level and then generate features such as MFCC or NRAF or any other customized features. Then we can index the features with kd trees associated with the caption. The problem of speech recognition now can be casted as a search problem. This is a simplistic approach that requires the solution of other problems.
|
|
|
|
First of all it is necessary to estimate the intrinsic dimension of the features representing speech. This is why we focused our interest in dimensionality reduction algorithms. This is important for "compressing" the data and saving space. The second issue is to build a cache for trees that don't fit in the main memory already addressed. The last step is to generate features on different time scales.
|
|
|
|
This is probably the most interesting step. Initially we start with features of a minimum durations, let's say 20msec MFCC 39 coefficients. After bringing it down to a certain dimension $d$ with KPCA or SDPCA. Then we merge the features creating vectors of $2d$. We keep repeating this until we reach a stage where the dimensionality cannot be decreased to a value that is beyond the strength of kd-trees.
|
|
|
|
Our goal is to apply this algorithm on the whole Broadcast News database \cite{graff1997bns}.
|
|
|
|
\subsection{Adaptive kernel Filters}
|
|
One of the problems of spectral methods is the out of sample
|
|
extension. In practice this means that once we found the new
|
|
dimensions (eigenvectors of the kernel matrix) we need to find a way
|
|
to compute the new dimensions of a test point $x$. The most common
|
|
method is the one based on Nystr\"{o}m's formula, which requires an
|
|
expression or at least a way to compute the kernel product of the
|
|
test point and it's nearest neighbors.
|
|
|
|
\begin{equation}
|
|
y_k(x) = \frac{1}{\lambda_k}\sum_{i=1}^{N} w_{ik}K(x, x_i)
|
|
\end{equation}
|
|
where
|
|
\begin{itemize}
|
|
\item $y_k(x)$ is the $kth$ embedded dimension of $x$, in other
|
|
words if $x$ was the $N+1$ point of the kernel matrix $K$, then
|
|
$y_k(x)$ would have been the $w_{(N+1)k}$.
|
|
\item $\lambda_k$ is the $kth$ eigenvalue
|
|
\item $w_k$ is the $kth$ eigenvector
|
|
\end{itemize}
|
|
Unfortunately for methods like Semidefinite PCA there is no
|
|
expression for the kernel.
|
|
One other approach \cite{vanvaerenbergh2006swk} has been to keep a certain size kernel matrix and
|
|
after the arrival of a new test point we remove the oldest from the
|
|
kernel and we add the new one. The kernel looks like a sliding
|
|
window over the data. There is a fast way to compute the eigenvalues
|
|
and eigenvectors of the updated matrix.
|
|
|
|
We propose an alternative approach to the problem. As we noticed in
|
|
Semidefinite PCA the optimization step is iterative. Addition of an
|
|
extra line/column in the kernel matrix will require one or two
|
|
iterations for convergence. Meanwhile the computation of the
|
|
eigenvalues/eigenvectors can be done with the online algorithm that
|
|
converges fast \cite{gorrell2006gha}.
|
|
|
|
\end{Body}
|
|
|
|
\begin{EndMatter}
|
|
%\appendix{This is my first appendix}
|
|
|
|
%% Below are some optional environments that can be used here
|
|
\bibliographystyle{ieeetr}
|
|
\bibliography{proposal}
|
|
%\references % Generates the bibliography page
|
|
\index % Generates the index - should be last
|
|
\end{EndMatter}
|
|
|
|
\end{document}
|