several changes, fixed images, changed title. Looks more like a document. Ready for proof reading
This commit is contained in:
vasiloglou
File diff suppressed because it is too large
Load Diff
@@ -520,3 +520,25 @@ YEAR= { 1980 } }
|
||||
year={2002},
|
||||
publisher={MIT Press}
|
||||
}
|
||||
|
||||
@book{garofolo1993tap,
|
||||
title={{TIMIT Acoustic-phonetic Continuous Speech Corpus.}},
|
||||
author={Garofolo, J.S. and National Institute of Standards and Technology (US and Linguistic Data Consortium and Information Science and Technology Office and United States and Defense Advanced Research Projects Agency},
|
||||
year={1993},
|
||||
publisher={Linguistic Data Consortium}
|
||||
}
|
||||
|
||||
@article{graff1997bns,
|
||||
title={{The 1996 Broadcast News Speech and Language-Model Corpus}},
|
||||
author={Graff, D. and others},
|
||||
journal={Proceedings of the DARPA Workshop on Spoken Language technology},
|
||||
pages={11--14},
|
||||
year={1997}
|
||||
}
|
||||
|
||||
@book{spall2003iss,
|
||||
title={{Introduction to Stochastic Search and Optimization}},
|
||||
author={Spall, J.C.},
|
||||
year={2003},
|
||||
publisher={John Wiley \& Sons, Inc. New York, NY, USA}
|
||||
}
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
%% Begin Preamble Info
|
||||
\documentclass[12pt,letterpaper,doublespaced,ETD,dvips,proposal]{gtthesis}
|
||||
|
||||
\title{Nearest Neighbor methods in machine learning with applications to speech}
|
||||
\title{Fast Kernel computation with applications in speech}
|
||||
\author{Nikolaos Vasiloglou II}
|
||||
\copyrightyear{2007}
|
||||
\graddate{August 2007} %Appears at bottom of title page
|
||||
@@ -272,7 +272,7 @@ show that empirically it is more preferable to have unbalance splits
|
||||
on the tree, since it favors pruning in the process of finding the
|
||||
nearest neighbors.
|
||||
|
||||
\paragraph{Data structures for -d trees}
|
||||
\paragraph{Data structures for kd-trees}
|
||||
|
||||
Kd-tree consists of two types of data structures, the nodes and the
|
||||
leafs. The basic data needed for the nodes are the
|
||||
@@ -289,28 +289,72 @@ nearest neighbors.
|
||||
of nearest neighbor search with kd-trees will be described in the following section,
|
||||
fig.~\ref{nearest_neighbor_searchon_kd}.
|
||||
|
||||
\begin{figure}[!htb]
|
||||
\label{parent_kd_node}
|
||||
\centerline{\includegraphics[height=8cm]{parent_kd_node.eps}}
|
||||
\caption{The parent node of a 2
|
||||
dimensional kd-tree}
|
||||
\end{figure}
|
||||
|
||||
\insfig[parent_kd_node]{parent_kd_node.eps}{The parent node of a 2
|
||||
dimensional kd-tree}{8cm}
|
||||
%\insfig[parent_kd_node]{parent_kd_node.eps}{The parent node of a 2
|
||||
%dimensional kd-tree}{8cm}
|
||||
|
||||
\insfig[2_kd_node]{2_kd_node.eps}{A two node 2 dimensional
|
||||
kd-tree}{8cm}
|
||||
\begin{figure}[!htb]
|
||||
\label{2_kd_node}
|
||||
\centerline{\includegraphics[height=8cm]{2_kd_node.eps}}
|
||||
\caption{A two node 2 dimensional
|
||||
kd-tree}
|
||||
\end{figure}
|
||||
|
||||
\insfig[nearest_neighbor_searchon_kd]{nearest_neighbor_searchon_kd.eps}{Nearest
|
||||
neighbor with pruning}{8cm}
|
||||
%\insfig[2_kd_node]{2_kd_node.eps}{A two node 2 dimensional
|
||||
%kd-tree}{8cm}
|
||||
|
||||
\insfig[nearest_neighbor_searchon_kd_true_neighbor]{nearest_neighbor_searchon_kd_true_neighbor.eps}
|
||||
{True nearest neighbor is out of the leaf}{8cm}
|
||||
\begin{figure}[!htb]
|
||||
\label{nearest_neighbor_searchon_kd}
|
||||
\centerline{\includegraphics[height=8cm]{nearest_neighbor_searchon_kd.eps}}
|
||||
\caption{Nearest neighbor with pruning}
|
||||
\end{figure}
|
||||
|
||||
\insfig[nearest_neighbor_searchon_kd_no_prune]{nearest_neighbor_searchon_kd_no_prune.eps}{
|
||||
Kd tree with no possible pruning}{8cm}
|
||||
%\insfig[nearest_neighbor_searchon_kd]{nearest_neighbor_searchon_kd.eps}{Nearest
|
||||
%neighbor with pruning}{8cm}
|
||||
|
||||
\insfig[kd_pathological_points]{kd_pathological_points.eps}{A two
|
||||
dimensional pathological kd-tree}{8cm}
|
||||
\begin{figure}[!htb]
|
||||
\label{nearest_neighbor_searchon_kd_true_neighbor}
|
||||
\centerline{\includegraphics[height=8cm]{nearest_neighbor_searchon_kd_true_neighbor.eps}}
|
||||
\caption{True nearest neighbor is out of the leaf}
|
||||
\end{figure}
|
||||
|
||||
\insfig[pathological_kd_tree]{pathological_kd_tree.eps}{In this case
|
||||
the partition is very bad and almost for every point pruning is not
|
||||
feasible}{8cm}
|
||||
%\insfig[nearest_neighbor_searchon_kd_true_neighbor]{nearest_neighbor_searchon_kd_true_neighbor.eps}
|
||||
%{True nearest neighbor is out of the leaf}{8cm}
|
||||
|
||||
\begin{figure}[!htb]
|
||||
\label{nearest_neighbor_searchon_kd_no_prune}
|
||||
\centerline{\includegraphics[height=8cm]{nearest_neighbor_searchon_kd_no_prune.eps}}
|
||||
\caption{Kd tree with no possible pruning}
|
||||
\end{figure}
|
||||
|
||||
%\insfig[nearest_neighbor_searchon_kd_no_prune]{nearest_neighbor_searchon_kd_no_prune.eps}{
|
||||
%Kd tree with no possible pruning}{8cm}
|
||||
|
||||
\begin{figure}[!htb]
|
||||
\label{kd_pathological_points}
|
||||
\centerline{\includegraphics[height=8cm]{kd_pathological_points.eps}}
|
||||
\caption{A two dimensional pathological kd-tree}
|
||||
\end{figure}
|
||||
|
||||
%\insfig[kd_pathological_points]{kd_pathological_points.eps}{A two
|
||||
%dimensional pathological kd-tree}{8cm}
|
||||
|
||||
\begin{figure}[!htb]
|
||||
\label{pathological_kd_tree}
|
||||
\centerline{\includegraphics[height=8cm]{pathological_kd_tree.eps}}
|
||||
\caption{In this case the partition is very bad and almost for every point pruning is not
|
||||
feasible}
|
||||
\end{figure}
|
||||
|
||||
%\insfig[pathological_kd_tree]{pathological_kd_tree.eps}{In this case
|
||||
%the partition is very bad and almost for every point pruning is not
|
||||
%feasible}{8cm}
|
||||
|
||||
|
||||
|
||||
@@ -425,8 +469,11 @@ the query and the reference set are the same), gives linear
|
||||
complexity over the number of data. This is empirical complexity. In
|
||||
fig.~\ref{dual_tree} we give an example of dual-tree pruning.
|
||||
|
||||
\instbfig[dual_tree]{dual_tree_box_pruning1.eps}{dual_tree_box_pruning.eps}
|
||||
{The query node in red after top down recursion ends up in a leaf of
|
||||
\begin{figure}[!htb]
|
||||
\label{dual_tree}
|
||||
\centerline{\includegraphics[height=8cm]{dual_tree_box_pruning1.eps},
|
||||
\includegraphics[height=8cm]{dual_tree_box_pruning.eps}}
|
||||
\caption{The query node in red after top down recursion ends up in a leaf of
|
||||
the reference tree. Then every point in the query node (red) finds
|
||||
with the naive method its candidate nearest neighbor. Then for all
|
||||
of them we compare all the candidate nearest distances and find the
|
||||
@@ -434,7 +481,19 @@ maximum $r_{max}$. Now we know that if there is any node in distance
|
||||
greater than $r_max$ there is no point in checking for candidate
|
||||
nearest neighbors. As we see in the right figure the dashed box
|
||||
doesn't intersect with the bounding box of the leaf. This means that
|
||||
we can prune the yellow boxes.}{8cm}{8cm}
|
||||
we can prune the yellow boxes.}
|
||||
\end{figure}
|
||||
|
||||
%\instbfig[dual_tree]{dual_tree_box_pruning1.eps}{dual_tree_box_pruning.eps}
|
||||
%{The query node in red after top down recursion ends up in a leaf of
|
||||
%the reference tree. Then every point in the query node (red) finds
|
||||
%with the naive method its candidate nearest neighbor. Then for all
|
||||
%of them we compare all the candidate nearest distances and find the
|
||||
%maximum $r_{max}$. Now we know that if there is any node in distance
|
||||
%greater than $r_max$ there is no point in checking for candidate
|
||||
%nearest neighbors. As we see in the right figure the dashed box
|
||||
%doesn't intersect with the bounding box of the leaf. This means that
|
||||
%we can prune the yellow boxes.}{8cm}{8cm}
|
||||
|
||||
\subsection{Large scale trees, for out of core memory}
|
||||
\label{Large_scale_trees}
|
||||
@@ -559,7 +618,7 @@ that has to be addressed is page fault handling. In the following
|
||||
section we describe two different memory managers we implemented and
|
||||
we compare their performance.
|
||||
|
||||
\begin{figure}[!H]
|
||||
\begin{figure}[!htb]
|
||||
\label{Alignment}
|
||||
\begin{boxedminipage}[c]{\linewidth}
|
||||
\begin{verbatim}
|
||||
@@ -630,7 +689,7 @@ SIGSEGV library along with mprotect to address efficiently this.
|
||||
Pages are write protected in memory and if a write occurs a signal
|
||||
is raised and the page is marked as modified.
|
||||
|
||||
\begin{table}[!h]
|
||||
\begin{table}[!htb]
|
||||
\footnotesize{ \centering
|
||||
\begin{tabular}{|c|c|c|c|}
|
||||
\hline
|
||||
@@ -645,7 +704,7 @@ is raised and the page is marked as modified.
|
||||
}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[!h]
|
||||
\begin{table}[!htb]
|
||||
\footnotesize{ \centering
|
||||
\begin{tabular}{|c|c|c|c|}
|
||||
\hline
|
||||
@@ -660,7 +719,7 @@ is raised and the page is marked as modified.
|
||||
}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[!h]
|
||||
\begin{table}[!htb]
|
||||
\footnotesize{ \centering
|
||||
\begin{tabular}{|c|c|c|c|c|}
|
||||
\hline
|
||||
@@ -1078,96 +1137,98 @@ fall under the general case of N-body problem computations. The
|
||||
speed can be significantly improved by using KD-trees as described
|
||||
in \cite{gray2001nbp} and reduce the complexity up to $O(n)$.
|
||||
|
||||
\instbfig[fig1.1]{fig1.1b.eps}{fig1.1c.eps}{The same data with
|
||||
fig.~\ref{fig1}a analyzed with a larger global bandwidth. (a) The
|
||||
estimated density, (b) The first non-trivial
|
||||
eigenvector}{4.5cm}{4.5cm}
|
||||
%\begin{figure}[h!]
|
||||
% \centerline{\includegraphics[height=4.5cm]{fig1.1b.eps}(a)}
|
||||
% \centerline{\includegraphics[height=4.5cm]{fig1.1c.eps}(b)}
|
||||
% \caption{The same data with fig.~\ref{fig1}a analyzed with a
|
||||
% larger global bandwidth. (a) The estimated density, (b) The first
|
||||
% non-trivial eigenvector }
|
||||
% \label{fig1.1}
|
||||
%\end{figure}
|
||||
%\instbfig[fig1.1]{fig1.1b.eps}{fig1.1c.eps}{The same data with
|
||||
%fig.~\ref{fig1}a analyzed with a larger global bandwidth. (a) The
|
||||
%estimated density, (b) The first non-trivial
|
||||
%eigenvector}{4.5cm}{4.5cm}
|
||||
|
||||
\inssssfig[fig1]{fig1a.eps}{fig1b.eps}{fig1c.eps}{(a) Two clusters
|
||||
of uniformly distributed 2-D points, (b) The estimated density, (c)
|
||||
The first non-trivial eigenvector}{4.5cm}{4.5cm}{4.5cm}
|
||||
%\begin{figure}[tb]
|
||||
% \centerline{\includegraphics[height=4.4cm]{fig1a.eps}(a)}
|
||||
% \centerline{\includegraphics[height=4.4cm]{fig1b.eps}(b)}
|
||||
% \centerline{\includegraphics[height=4.4cm]{fig1c.eps}(c)}
|
||||
% \caption{(a) Two clusters of uniformly distributed 2-D points,
|
||||
%(b) The estimated density, (c) The first non-trivial eigenvector }
|
||||
% \label{fig1}
|
||||
%\end{figure}
|
||||
\begin{figure}[!htb]
|
||||
\centerline{\includegraphics[height=4.5cm]{fig1.1b.eps}(a)}
|
||||
\centerline{\includegraphics[height=4.5cm]{fig1.1c.eps}(b)}
|
||||
\caption{The same data with fig.~\ref{fig1}a analyzed with a
|
||||
larger global bandwidth. (a) The estimated density, (b) The first
|
||||
non-trivial eigenvector }
|
||||
\label{fig1.1}
|
||||
\end{figure}
|
||||
|
||||
\inssssfig[fig2]{fig2a.eps}{fig2b.eps}{fig2c.eps}{(a) Two clusters
|
||||
of non-uniformly distributed 2-D points, (b)The estimated density,
|
||||
(c) The first non-trivial eigenvector}{6cm}{6cm}{6cm}
|
||||
%
|
||||
%\begin{figure}[tb]
|
||||
% \centerline{\includegraphics[height=6.0cm]{fig2a.eps}(a)}
|
||||
% \centerline{\includegraphics[height=6.0cm]{fig2b.eps}(b)}
|
||||
% \centerline{\includegraphics[height=6.0cm]{fig2c.eps}(c)}
|
||||
%\caption{(a) Two clusters of non-uniformly distributed 2-D points,
|
||||
%(b)The estimated density, (c) The first non-trivial eigenvector
|
||||
%\vspace{1.2cm}}
|
||||
% \label{fig2}
|
||||
%\end{figure}
|
||||
%\inssssfig[fig1]{fig1a.eps}{fig1b.eps}{fig1c.eps}{(a) Two clusters
|
||||
%of uniformly distributed 2-D points, (b) The estimated density, (c)
|
||||
%The first non-trivial eigenvector}{4.5cm}{4.5cm}{4.5cm}
|
||||
|
||||
\inssssfig[fig3]{fig3a.eps}{fig3b.eps}{fig3c.eps}{The same data set
|
||||
with fig.~\ref{fig2} processed with a bandwidth one order of
|
||||
magnitude larger. Although the first eigenvector still separates the
|
||||
two classes, the classes are closer and not compact (a) Two clusters
|
||||
of non-uniformly distributes 2-D points, (b)The estimated density,
|
||||
(c) The first non trivial eigenvector}{6cm}{6cm}{6cm}
|
||||
\begin{figure}[!htb]
|
||||
\centerline{\includegraphics[height=4.4cm]{fig1a.eps}(a)}
|
||||
\centerline{\includegraphics[height=4.4cm]{fig1b.eps}(b)}
|
||||
\centerline{\includegraphics[height=4.4cm]{fig1c.eps}(c)}
|
||||
\caption{(a) Two clusters of uniformly distributed 2-D points,
|
||||
(b) The estimated density, (c) The first non-trivial eigenvector }
|
||||
\label{fig1}
|
||||
\end{figure}
|
||||
|
||||
%\begin{figure}[tb]
|
||||
% \centerline{\includegraphics[height=6cm]{fig3a.eps}(a)}
|
||||
% \centerline{\includegraphics[height=6cm]{fig3b.eps}(b)}
|
||||
% \centerline{\includegraphics[height=6cm]{fig3c.eps}(c)}
|
||||
%\caption{The same data set with fig.~\ref{fig2} processed with a
|
||||
%bandwidth one order of magnitude larger. Although the first
|
||||
%eigenvector still separates the two classes, the classes are closer
|
||||
%and not compact (a) Two clusters of non-uniformly distributes 2-D
|
||||
%points, (b)The estimated density, (c) The first non trivial
|
||||
%eigenvector }
|
||||
% \label{fig3}
|
||||
%\end{figure}
|
||||
%\inssssfig[fig2]{fig2a.eps}{fig2b.eps}{fig2c.eps}{(a) Two clusters
|
||||
%of non-uniformly distributed 2-D points, (b)The estimated density,
|
||||
%(c) The first non-trivial eigenvector}{6cm}{6cm}{6cm}
|
||||
|
||||
\inssssfig[fig4]{fig4a.eps}{fig4c.eps}{fig4e.eps}{(a) The 3 phoneme
|
||||
classes after the dimensionality reduction with the optimal
|
||||
bandwidth, plotted in two dimensions, (b) The 3 phoneme classes with
|
||||
a larger bandwidth,(c) The 3 phoneme classes with a smaller
|
||||
bandwidth}{6cm}{6cm}{6cm}
|
||||
\begin{figure}[!htb]
|
||||
\centerline{\includegraphics[height=6.0cm]{fig2a.eps}(a)}
|
||||
\centerline{\includegraphics[height=6.0cm]{fig2b.eps}(b)}
|
||||
\centerline{\includegraphics[height=6.0cm]{fig2c.eps}(c)}
|
||||
\caption{(a) Two clusters of non-uniformly distributed 2-D points,
|
||||
(b)The estimated density, (c) The first non-trivial eigenvector
|
||||
\vspace{1.2cm}}
|
||||
\label{fig2}
|
||||
\end{figure}
|
||||
|
||||
%\begin{figure}[tb]
|
||||
% \centerline{\includegraphics[height=6cm]{fig4a.eps}(a)}
|
||||
% \centerline{\includegraphics[height=6cm]{fig4c.eps}(b)}
|
||||
%\centerline{\includegraphics[height=6cm]{fig4e.eps}(c)}
|
||||
% \caption{(a) The 3 phoneme classes after the dimensionality reduction
|
||||
%with the optimal bandwidth, plotted in two dimensions, (b) The 3
|
||||
%phoneme classes with a larger bandwidth,(c) The 3 phoneme classes
|
||||
%with a smaller bandwidth}
|
||||
% \label{fig4}
|
||||
%\end{figure}
|
||||
%\inssssfig[fig3]{fig3a.eps}{fig3b.eps}{fig3c.eps}{The same data set
|
||||
%with fig.~\ref{fig2} processed with a bandwidth one order of
|
||||
%magnitude larger. Although the first eigenvector still separates the
|
||||
%two classes, the classes are closer and not compact (a) Two clusters
|
||||
%of non-uniformly distributes 2-D points, (b)The estimated density,
|
||||
%(c) The first non trivial eigenvector}{6cm}{6cm}{6cm}
|
||||
|
||||
\inssssfig[fig5]{fig4b.eps}{fig4d.eps}{fig4f.eps}{The diffusion
|
||||
graph for three different local bandwidths (a)Optimal, (b)An order
|
||||
of magnitude larger, (c)An order of magnitude
|
||||
smaller}{6cm}{6cm}{6cm}
|
||||
\begin{figure}[!htb]
|
||||
\centerline{\includegraphics[height=6cm]{fig3a.eps}(a)}
|
||||
\centerline{\includegraphics[height=6cm]{fig3b.eps}(b)}
|
||||
\centerline{\includegraphics[height=6cm]{fig3c.eps}(c)}
|
||||
\caption{The same data set with fig.~\ref{fig2} processed with a
|
||||
bandwidth one order of magnitude larger. Although the first
|
||||
eigenvector still separates the two classes, the classes are closer
|
||||
and not compact (a) Two clusters of non-uniformly distributes 2-D
|
||||
points, (b)The estimated density, (c) The first non trivial
|
||||
eigenvector }
|
||||
\label{fig3}
|
||||
\end{figure}
|
||||
|
||||
%\begin{figure}[tb]
|
||||
% \centerline{\includegraphics[height=6cm]{fig4b.eps}(a)}
|
||||
%\centerline{\includegraphics[height=6cm]{fig4d.eps}(b)}
|
||||
%\centerline{\includegraphics[height=6cm]{fig4f.eps}(c)}
|
||||
% \caption{The diffusion graph for three different local bandwidths
|
||||
%(a)Optimal, (b)An order of magnitude larger, (c)An order of
|
||||
%magnitude smaller}
|
||||
% \label{fig5}
|
||||
%\end{figure}
|
||||
%\inssssfig[fig4]{fig4a.eps}{fig4c.eps}{fig4e.eps}{(a) The 3 phoneme
|
||||
%classes after the dimensionality reduction with the optimal
|
||||
%bandwidth, plotted in two dimensions, (b) The 3 phoneme classes with
|
||||
%a larger bandwidth,(c) The 3 phoneme classes with a smaller
|
||||
%bandwidth}{6cm}{6cm}{6cm}
|
||||
|
||||
\begin{figure}[!htb]
|
||||
\centerline{\includegraphics[height=6cm]{fig4a.eps}(a)}
|
||||
\centerline{\includegraphics[height=6cm]{fig4c.eps}(b)}
|
||||
\centerline{\includegraphics[height=6cm]{fig4e.eps}(c)}
|
||||
\caption{(a) The 3 phoneme classes after the dimensionality reduction
|
||||
with the optimal bandwidth, plotted in two dimensions, (b) The 3
|
||||
phoneme classes with a larger bandwidth,(c) The 3 phoneme classes
|
||||
with a smaller bandwidth}
|
||||
\label{fig4}
|
||||
\end{figure}
|
||||
|
||||
%\inssssfig[fig5]{fig4b.eps}{fig4d.eps}{fig4f.eps}{The diffusion
|
||||
%graph for three different local bandwidths (a)Optimal, (b)An order
|
||||
%of magnitude larger, (c)An order of magnitude
|
||||
%smaller}{6cm}{6cm}{6cm}
|
||||
|
||||
\begin{figure}[!htb]
|
||||
\centerline{\includegraphics[height=6cm]{fig4b.eps}(a)}
|
||||
\centerline{\includegraphics[height=6cm]{fig4d.eps}(b)}
|
||||
\centerline{\includegraphics[height=6cm]{fig4f.eps}(c)}
|
||||
\caption{The diffusion graph for three different local bandwidths
|
||||
(a)Optimal, (b)An order of magnitude larger, (c)An order of
|
||||
magnitude smaller}
|
||||
\label{fig5}
|
||||
\end{figure}
|
||||
|
||||
|
||||
\subsection{Preliminary Investigation in Nearest Neighbor research for speech recognition}
|
||||
@@ -1284,7 +1345,7 @@ from 13 to 39.
|
||||
%\ref{dualtree_algorithm}. The complexity of the algorithm, on
|
||||
%average, turns out to be linear.
|
||||
|
||||
\begin{figure}[!H]
|
||||
\begin{figure}[!htb]
|
||||
\begin{boxedminipage}[c]{\linewidth}
|
||||
\begin{verbatim}
|
||||
recurse(q : KdTree, r : KdTree) {
|
||||
@@ -1308,12 +1369,22 @@ algorithm} \label{dualtree_algorithm}
|
||||
\end{figure}
|
||||
|
||||
|
||||
\begin{figure}[!htb]
|
||||
\label{kdtree}
|
||||
\centerline{\includegraphics[height=6cm]{kdtree.eps}}
|
||||
\caption{A two dimensional kd-tree}
|
||||
\end{figure}
|
||||
|
||||
\insfig[kdtree]{kdtree.eps}{A two dimensional kd-tree}{8cm}
|
||||
%\insfig[kdtree]{kdtree.eps}{A two dimensional kd-tree}{8cm}
|
||||
|
||||
\begin{figure}[!htb]
|
||||
\label{dualkdtree}
|
||||
\centerline{\includegraphics[height=4cm]{dualtree_recursion.eps}}
|
||||
\caption{Simulation of the dual tree algorithm}
|
||||
\end{figure}
|
||||
|
||||
\insfig[dualkdtree]{dualtree_recursion.eps}{Simulation of the dual
|
||||
tree algorithm}{8cm}
|
||||
%\insfig[dualkdtree]{dualtree_recursion.eps}{Simulation of the dual
|
||||
%tree algorithm}{8cm}
|
||||
|
||||
|
||||
\subsubsection{Experimental Results} In order to illustrate the
|
||||
@@ -1401,16 +1472,28 @@ features. Perhaps using lesser number of NRAF coefficients or
|
||||
different filter spacing, the inter-speaker variability could be
|
||||
removed.
|
||||
|
||||
\begin{figure}[!htb]
|
||||
\label{pca}
|
||||
\centerline{\includegraphics[height=8cm]{timit_pca.eps}}
|
||||
\centerline{\includegraphics[height=8cm]{ntimit_pca.eps}}
|
||||
\caption{Principal Component Analysis of TIMIT (top) and NTIMIT (bottom) for MFCC (solid line)
|
||||
and for NRAF (dashed line).}
|
||||
\end{figure}
|
||||
|
||||
\instbfig[pca]{timit_pca.eps}{ntimit_pca.eps}{Principal Component
|
||||
Analysis of TIMIT (top) and NTIMIT (bottom) for MFCC (solid line)
|
||||
and for NRAF (dashed line).}{8cm}{8cm}
|
||||
%\instbfig[pca]{timit_pca.eps}{ntimit_pca.eps}{Principal Component
|
||||
%Analysis of TIMIT (top) and NTIMIT (bottom) for MFCC (solid line)
|
||||
%and for NRAF (dashed line).}{8cm}{8cm}
|
||||
|
||||
\begin{figure}[!htb]
|
||||
\label{kernel_pca}
|
||||
\centerline{\includegraphics[height=8cm]{kernel_pca.eps}}
|
||||
\caption{Kernel PCA spectrum for TIMIT MFCC features}
|
||||
\end{figure}
|
||||
|
||||
\insfig[kernel_pca]{kernel_pca.eps}{Kernel PCA spectrum for TIMIT
|
||||
MFCC features}{8cm}
|
||||
%\insfig[kernel_pca]{kernel_pca.eps}{Kernel PCA spectrum for TIMIT
|
||||
%MFCC features}{8cm}
|
||||
|
||||
\begin{table}[h!]
|
||||
\begin{table}[!htb]
|
||||
\footnotesize{ \centering
|
||||
\begin{tabular}{|c|c|c|c|}
|
||||
\hline
|
||||
@@ -1427,7 +1510,7 @@ MFCC features}{8cm}
|
||||
\label{timing}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[htb]
|
||||
\begin{table}[!htb]
|
||||
\footnotesize{ \centering
|
||||
\begin{tabular}{|c|c|c|c|c|c|}
|
||||
\hline
|
||||
@@ -1442,7 +1525,7 @@ MFCC features}{8cm}
|
||||
\label{kneighborTIMIT}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[htb]
|
||||
\begin{table}[!htb]
|
||||
\footnotesize{ \centering
|
||||
\begin{tabular}{|c|c|c|c|c|c|}
|
||||
\hline
|
||||
@@ -1457,7 +1540,7 @@ MFCC features}{8cm}
|
||||
\label{kneighborNTIMIT}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[htb]
|
||||
\begin{table}[!htb]
|
||||
\footnotesize{ \centering
|
||||
\begin{tabular}{|c|c|c|c|c|c|}
|
||||
\hline
|
||||
@@ -1473,7 +1556,7 @@ MFCC features}{8cm}
|
||||
} \label{kneighborLOOCVTIMIT}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[htb]
|
||||
\begin{table}[!htb]
|
||||
\footnotesize{ \centering
|
||||
\begin{tabular}{|c|c|c|c|c|c|}
|
||||
\hline
|
||||
@@ -1489,7 +1572,8 @@ MFCC features}{8cm}
|
||||
} \label{kneighborLOOCVNTIMIT}
|
||||
\end{table}
|
||||
|
||||
|
||||
\pagebreak
|
||||
\newpage
|
||||
\section{Proposed Research}
|
||||
\label{proposed}
|
||||
|
||||
@@ -1498,19 +1582,21 @@ Efficient computation of the kernel matrix for large volumes of
|
||||
data. Special emphasis is given in high dimensional data $d>30$. One
|
||||
of the goals is to use Kernel PCA for unfolding the speech manifold.
|
||||
In other words estimate the intrinsic dimensionality of speech by
|
||||
using big databases like timit, wall street journal and broadcast
|
||||
news [?]. Implicitly dimensionality reduction will lead to new feature
|
||||
using big databases like TIMIT \cite{garofolo1993tap}, Wall Street Journal and Broadcast
|
||||
News \cite{graff1997bns}. Implicitly dimensionality reduction will lead to new feature
|
||||
generation . The ultimate goal is to use these new features along
|
||||
with kd-trees do speech recognition by nearest neighbor search.
|
||||
|
||||
\subsection{Customized Kernels}
|
||||
|
||||
There are infinite functions
|
||||
that can act as valid kernels \cite{shawetaylor2004kmp}. The most popular kernels are:
|
||||
\begin{enumerate}
|
||||
\item The gaussian kernel $k(x, y)=e^{-\frac{||x-y||^2}{\sigma^2)}}$
|
||||
\item The polynomial kernel $k(x, y)=(xy)^n$
|
||||
\item The epanechnikov kernel $k(x, y)=1-||x-y||^2/\sigma^2, ||x-y||<\sigma, else 0$
|
||||
\item The epanechnikov kernel $k(x, y)=\left\{ \begin{array}{cc}
|
||||
1-||x-y||^2/\sigma^2 & ||x-y||\leq\sigma \\
|
||||
0 & ||x-y||>\sigma
|
||||
\end{array} \right.$
|
||||
\item The k-nearest neighbor kernel
|
||||
\end{enumerate}
|
||||
|
||||
@@ -1530,7 +1616,7 @@ from scratch without using any fixed kernel. As we have already
|
||||
mentioned the kernel matrix has to be a positive semidefinite
|
||||
matrix. Initially we have to define a k neighborhood for every
|
||||
point. So at first we have to connect points to their neighbors and
|
||||
build a initialize the non zero values of the matrix to 1. Then we
|
||||
initialize the non zero values of the matrix to 1. Then we
|
||||
maximize the trace of the kernel matrix under the constraint that
|
||||
the the distances in the kernel matrix are preserved. This leads to
|
||||
a semidefinite programming problem. Here is the algorithm they
|
||||
@@ -1556,9 +1642,11 @@ where $K$ is the kernel matrix, $G$ is the linear kernel matrix,
|
||||
$G_{ij}=x_i x_j^T,\quad x_i,x_j\in \Re^d$ and
|
||||
$Trace(K)=\sum_{i=0}^{N}K_{ii}$.
|
||||
|
||||
This is a typical semidefinite programming problem it can also be
|
||||
This is a typical semidefinite programming problem that can also be
|
||||
posed as
|
||||
|
||||
\vspace{1cm}
|
||||
|
||||
\fbox{
|
||||
\begin{minipage}[c]{0.8\linewidth}
|
||||
maximize $Trace(K)$
|
||||
@@ -1583,10 +1671,10 @@ where $F_i$ is a positive semidefinite matrix and in this case
|
||||
The inventors of this method named it Semidefinite PCA (SDPCA),
|
||||
which is the term we will use form now on.
|
||||
|
||||
\subsubsection{Speeding up SDPCA}
|
||||
\subsubsection{Speeding up Semidefinite PCA}
|
||||
Semidefinite problems are solved with the interior point method. The
|
||||
basic concept of interior point method \cite{nesterov1994ipp} is to build a barrier
|
||||
function that penalize solutions that violate the inequality
|
||||
function that penalizes solutions that violate the inequality
|
||||
constraints. The ideal function would be:
|
||||
|
||||
\begin{equation}
|
||||
@@ -1602,13 +1690,13 @@ I_{-}(u)= \left\{
|
||||
It turns out that a good approximation of this functions is
|
||||
|
||||
\begin{equation}
|
||||
\hat{I_{-}(u)} = -\frac{1}{t}\log(-u), u\leq 0,
|
||||
\hat{I}_{-}(u) = -\frac{1}{t}\log(-u), u\leq 0,
|
||||
\end{equation}
|
||||
|
||||
It turns out that
|
||||
|
||||
\begin{equation}
|
||||
\lim_{t\rightarrow \inf} \hat{I_{-}} = I_{-}
|
||||
\lim_{t\rightarrow \inf} \hat{I}_{-} = I_{-}
|
||||
\end{equation}
|
||||
|
||||
It turns out that a good barrier function for Semidefinite
|
||||
@@ -1616,6 +1704,7 @@ programming is
|
||||
\begin{equation}
|
||||
\phi(x)=\log \det(K)
|
||||
\end{equation}
|
||||
where $det(K)$ is the determinant of $K$ matrix.
|
||||
|
||||
At this stage here it is important to mention that there is a dual
|
||||
form of the semidefinite programming posed above which eliminates
|
||||
@@ -1638,7 +1727,7 @@ The barrier function for this form is
|
||||
\phi(x)=\frac{1}{t}\log\det(-F(x)^{-1})
|
||||
\end{equation}
|
||||
|
||||
The nice of this function is that the gradient has the form:
|
||||
The nice property of this function is that the gradient has the form:
|
||||
|
||||
\begin{equation}
|
||||
\frac{\partial\phi(x)}{\partial x_{n}} = Trace(-F(x)^{-1}F_n)
|
||||
@@ -1679,8 +1768,8 @@ There are two approaches on that:
|
||||
This means that for every $n$ only a few columns of $F(x)^{-1}$
|
||||
are required. They can be computed very quickly by the conjugate
|
||||
gradient method which is specially designed for sparse positive
|
||||
definite matrices. This is also a stochastic gradient approach as
|
||||
we show next.
|
||||
definite matrices \cite{saad2003ims}. This is also a stochastic gradient approach as
|
||||
we will show next \cite{spall2003iss}.
|
||||
\end{enumerate}
|
||||
|
||||
\subsubsection{Accelerating Semidefinite PCA with domain
|
||||
@@ -1733,7 +1822,7 @@ computationally intensive task. For Gaussian kernels there are
|
||||
approximations that can speed up computations orders of magnitude.
|
||||
We will briefly discuss them here. Besides computation saving we can
|
||||
save memory as well. The gaussian kernel has infinite support so the
|
||||
kernel matrix $K, K(x,y)=e^{-\frac{||x-y||}{\sigma^2}}$ is not
|
||||
kernel matrix $\{K : k(x,y)=e^{-\frac{||x-y||}{\sigma^2}} \}$ is not
|
||||
sparse. Numerical iterative methods for eigenvector computation are
|
||||
based on the general iteration $w^n=w^{n-1}+\lambda Kw^{n-1}$, where
|
||||
the multiplication of each row of $K$ with $w^{n-1}$ is a kernel
|
||||
@@ -1822,13 +1911,13 @@ eigen-spectrum of a kernel matrix
|
||||
\begin{equation}
|
||||
K = \sum_{i=1}^{N}\lambda_i ww^T
|
||||
\end{equation}
|
||||
or to be more specific the $(i, j)$ element of the kernel matrix
|
||||
or to be more specific to express the $(i, j)$ element of the kernel matrix
|
||||
\begin{equation}
|
||||
k(i,j)=\sum_{i=1}^{N}\lambda_i w_i w_j
|
||||
\end{equation}
|
||||
|
||||
and since $i$ corresponds to point $x$ and $j$ corresponds to $y$
|
||||
then we can rewrite it as
|
||||
we can rewrite it as
|
||||
|
||||
\begin{equation}
|
||||
k(x,y)=\sum_{i=1}^{N}w(x)w(y)
|
||||
@@ -1887,7 +1976,7 @@ First of all it is necessary to estimate the intrinsic dimension of the features
|
||||
|
||||
This is probably the most interesting step. Initially we start with features of a minimum durations, let's say 20msec MFCC 39 coefficients. After bringing it down to a certain dimension $d$ with KPCA or SDPCA. Then we merge the features creating vectors of $2d$. We keep repeating this until we reach a stage where the dimensionality cannot be decreased to a value that is beyond the strength of kd-trees.
|
||||
|
||||
Our goal is to apply this algorithm on the whole broadcast news database.
|
||||
Our goal is to apply this algorithm on the whole Broadcast News database \cite{graff1997bns}.
|
||||
|
||||
\subsection{Adaptive kernel Filters}
|
||||
One of the problems of spectral methods is the out of sample
|
||||
@@ -1909,7 +1998,6 @@ where
|
||||
\item $\lambda_k$ is the $kth$ eigenvalue
|
||||
\item $w_k$ is the $kth$ eigenvector
|
||||
\end{itemize}
|
||||
|
||||
Unfortunately for methods like Semidefinite PCA there is no
|
||||
expression for the kernel.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user