284 lines
47 KiB
HTML
284 lines
47 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
|
|
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
|
|
<meta name="generator" content="Doxygen 1.8.11"/>
|
|
<title>mlpack: K-Means tutorial (kmeans)</title>
|
|
<link href="tabs.css" rel="stylesheet" type="text/css"/>
|
|
<script type="text/javascript" src="jquery.js"></script>
|
|
<script type="text/javascript" src="dynsections.js"></script>
|
|
<link href="search/search.css" rel="stylesheet" type="text/css"/>
|
|
<script type="text/javascript" src="search/searchdata.js"></script>
|
|
<script type="text/javascript" src="search/search.js"></script>
|
|
<script type="text/javascript">
|
|
$(document).ready(function() { init_search(); });
|
|
</script>
|
|
<link href="doxygen.css" rel="stylesheet" type="text/css" />
|
|
<link href="extra-stylesheet.css" rel="stylesheet" type="text/css"/>
|
|
</head>
|
|
<body>
|
|
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
|
|
<div id="titlearea">
|
|
<table cellspacing="0" cellpadding="0">
|
|
<tbody>
|
|
<tr style="height: 56px;">
|
|
<td id="projectalign" style="padding-left: 0.5em;">
|
|
<div id="projectname">mlpack
|
|
 <span id="projectnumber">master</span>
|
|
</div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<!-- end header part -->
|
|
<!-- Generated by Doxygen 1.8.11 -->
|
|
<script type="text/javascript">
|
|
var searchBox = new SearchBox("searchBox", "search",false,'Search');
|
|
</script>
|
|
<div id="navrow1" class="tabs">
|
|
<ul class="tablist">
|
|
<li><a href="index.html"><span>Main Page</span></a></li>
|
|
<li class="current"><a href="pages.html"><span>Related Pages</span></a></li>
|
|
<li><a href="namespaces.html"><span>Namespaces</span></a></li>
|
|
<li><a href="annotated.html"><span>Classes</span></a></li>
|
|
<li><a href="files.html"><span>Files</span></a></li>
|
|
<li>
|
|
<div id="MSearchBox" class="MSearchBoxInactive">
|
|
<span class="left">
|
|
<img id="MSearchSelect" src="search/mag_sel.png"
|
|
onmouseover="return searchBox.OnSearchSelectShow()"
|
|
onmouseout="return searchBox.OnSearchSelectHide()"
|
|
alt=""/>
|
|
<input type="text" id="MSearchField" value="Search" accesskey="S"
|
|
onfocus="searchBox.OnSearchFieldFocus(true)"
|
|
onblur="searchBox.OnSearchFieldFocus(false)"
|
|
onkeyup="searchBox.OnSearchFieldChange(event)"/>
|
|
</span><span class="right">
|
|
<a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a>
|
|
</span>
|
|
</div>
|
|
</li>
|
|
</ul>
|
|
</div>
|
|
<!-- window showing the filter options -->
|
|
<div id="MSearchSelectWindow"
|
|
onmouseover="return searchBox.OnSearchSelectShow()"
|
|
onmouseout="return searchBox.OnSearchSelectHide()"
|
|
onkeydown="return searchBox.OnSearchSelectKey(event)">
|
|
</div>
|
|
|
|
<!-- iframe showing the search results (closed by default) -->
|
|
<div id="MSearchResultsWindow">
|
|
<iframe src="javascript:void(0)" frameborder="0"
|
|
name="MSearchResults" id="MSearchResults">
|
|
</iframe>
|
|
</div>
|
|
|
|
</div><!-- top -->
|
|
<div class="header">
|
|
<div class="headertitle">
|
|
<div class="title">K-Means tutorial (kmeans) </div> </div>
|
|
</div><!--header-->
|
|
<div class="contents">
|
|
<div class="textblock"><h1><a class="anchor" id="intro_kmtut"></a>
|
|
Introduction</h1>
|
|
<p>The popular k-means algorithm for clustering has been around since the late 1950s, and the standard algorithm was proposed by Stuart Lloyd in 1957. Given a set of points <img class="formulaInl" alt="$ X $" src="form_81.png"/>, k-means clustering aims to partition each point <img class="formulaInl" alt="$ x_i $" src="form_125.png"/> into a cluster <img class="formulaInl" alt="$ c_j $" src="form_126.png"/> (where <img class="formulaInl" alt="$ j \le k $" src="form_127.png"/> and <img class="formulaInl" alt="$ k $" src="form_128.png"/>, the number of clusters, is a parameter). The partitioning is done to minimize the objective function</p>
|
|
<p class="formulaDsp">
|
|
<img class="formulaDsp" alt="\[ \sum_{j = 1}^{k} \sum_{x_i \in c_j} \| x_i - \mu_j \|^2 \]" src="form_129.png"/>
|
|
</p>
|
|
<p>where <img class="formulaInl" alt="$\mu_j$" src="form_130.png"/> is the centroid of cluster <img class="formulaInl" alt="$c_j$" src="form_131.png"/>. The standard algorithm is a two-step algorithm:</p>
|
|
<ul>
|
|
<li><b>Assignment</b> <b>step</b>. Each point <img class="formulaInl" alt="$x_i$" src="form_132.png"/> in <img class="formulaInl" alt="$X$" src="form_133.png"/> is assigned to the cluster whose centroid it is closest to.</li>
|
|
<li><b>Update</b> <b>step</b>. Using the new cluster assignments, the centroids of each cluster are recalculated.</li>
|
|
</ul>
|
|
<p>The algorithm has converged when no more assignment changes are happening with each iteration. However, this algorithm can get stuck in local minima of the objective function and is particularly sensitive to the initial cluster assignments. Also, situations can arise where the algorithm will never converge but reaches steady state – for instance, one point may be changing between two cluster assignments.</p>
|
|
<p>There is vast literature on the k-means algorithm and its uses, as well as strategies for choosing initial points effectively and keeping the algorithm from converging in local minima. <b>mlpack</b> does implement some of these, notably the Bradley-Fayyad algorithm (see the reference below) for choosing refined initial points. Importantly, the C++ <code>KMeans</code> class makes it very easy to improve the k-means algorithm in a modular way.</p>
|
|
<div class="fragment"><div class="line">@inproceedings{bradley1998refining,</div><div class="line"> title={Refining initial points <span class="keywordflow">for</span> k-means clustering},</div><div class="line"> author={Bradley, Paul S. and Fayyad, Usama M.},</div><div class="line"> booktitle={Proceedings of the Fifteenth International Conference on Machine</div><div class="line"> Learning (ICML 1998)},</div><div class="line"> volume={66},</div><div class="line"> year={1998}</div><div class="line">}</div></div><!-- fragment --><p><b>mlpack</b> provides:</p>
|
|
<ul>
|
|
<li>a <a class="el" href="kmtutorial.html#cli_kmtut">simple command-line executable</a> to run k-means</li>
|
|
<li>a <a class="el" href="kmtutorial.html#kmeans_kmtut">simple C++ interface</a> to run k-means</li>
|
|
<li>a <a class="el" href="kmtutorial.html#kmeans_template_kmtut">generic, extensible, and powerful C++ class</a> for complex usage</li>
|
|
</ul>
|
|
<h1><a class="anchor" id="toc_kmtut"></a>
|
|
Table of Contents</h1>
|
|
<p>A list of all the sections this tutorial contains.</p>
|
|
<ul>
|
|
<li><a class="el" href="kmtutorial.html#intro_kmtut">Introduction</a></li>
|
|
<li><a class="el" href="kmtutorial.html#toc_kmtut">Table of Contents</a></li>
|
|
<li><a class="el" href="kmtutorial.html#cli_kmtut">Command-Line 'kmeans'</a><ul>
|
|
<li><a class="el" href="kmtutorial.html#cli_ex1_kmtut">Simple k-means clustering</a></li>
|
|
<li><a class="el" href="kmtutorial.html#cli_ex2_kmtut">Saving the resulting centroids</a></li>
|
|
<li><a class="el" href="kmtutorial.html#cli_ex3_kmtut">Allowing empty clusters</a></li>
|
|
<li><a class="el" href="kmtutorial.html#cli_ex4_kmtut">Limiting the maximum number of iterations</a></li>
|
|
<li><a class="el" href="kmtutorial.html#cli_ex6_kmtut">Using Bradley-Fayyad "refined start"</a></li>
|
|
<li><a class="el" href="kmtutorial.html#cli_ex7_kmtut">Using different k-means algorithms</a></li>
|
|
</ul>
|
|
</li>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_kmtut">The 'KMeans' class</a><ul>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_ex1_kmtut">Running k-means and getting cluster assignments</a></li>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_ex2_kmtut">Running k-means and getting centroids of clusters</a></li>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_ex3_kmtut">Limiting the maximum number of iterations</a></li>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_ex5_kmtut">Setting initial cluster assignments</a></li>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_ex6_kmtut">Setting initial cluster centroids</a></li>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_ex7_kmtut">Running sparse k-means</a></li>
|
|
</ul>
|
|
</li>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_template_kmtut">Template parameters for the 'KMeans' class</a><ul>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_metric_kmtut">Changing the distance metric used for k-means</a></li>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_initial_partition_kmtut">Changing the initial partitioning strategy used for k-means</a></li>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_empty_cluster_kmtut">Changing the action taken when an empty cluster is encountered</a></li>
|
|
<li><a class="el" href="kmtutorial.html#kmeans_lloyd_kmtut">The LloydStepType template parameter</a></li>
|
|
</ul>
|
|
</li>
|
|
<li><a class="el" href="kmtutorial.html#further_doc_kmtut">Further documentation</a></li>
|
|
</ul>
|
|
<h1><a class="anchor" id="cli_kmtut"></a>
|
|
Command-Line 'kmeans'</h1>
|
|
<p><b>mlpack</b> provides a command-line executable, <code>mlpack_kmeans</code>, to allow easy execution of the k-means algorithm on data. Complete documentation of the executable can be found by typing</p>
|
|
<div class="fragment"><div class="line">$ mlpack_kmeans --help</div></div><!-- fragment --><p>As of October 2014, support for overclustering has been removed due to bugs and lack of usage. If this is support you were using, or are interested, please file a bug or get in touch with the <b>mlpack</b> developers in some way so that the support can be re-implemented.</p>
|
|
<p>Below are several examples demonstrating simple use of the <code>mlpack_kmeans</code> executable.</p>
|
|
<h2><a class="anchor" id="cli_ex1_kmtut"></a>
|
|
Simple k-means clustering</h2>
|
|
<p>We want to find 5 clusters using the points in the file dataset.csv. By default, if any of the clusters end up empty, that cluster will be reinitialized to contain the point furthest from the cluster with maximum variance. The cluster assignments of each point will be stored in assignments.csv. Each row in assignments.csv will correspond to the row in dataset.csv.</p>
|
|
<div class="fragment"><div class="line">$ mlpack_kmeans -c 5 -i dataset.csv -v -o assignments.csv</div></div><!-- fragment --><h2><a class="anchor" id="cli_ex2_kmtut"></a>
|
|
Saving the resulting centroids</h2>
|
|
<p>Sometimes it is useful to save the centroids of the clusters found by k-means; one example might be for plotting the points. The <code>-C</code> (<code>–centroid_file</code>) option allows specification of a file into which the centroids will be saved (one centroid per line, if it is a CSV or other text format).</p>
|
|
<div class="fragment"><div class="line">$ mlpack_kmeans -c 5 -i dataset.csv -v -o assignments.csv -C centroids.csv</div></div><!-- fragment --><h2><a class="anchor" id="cli_ex3_kmtut"></a>
|
|
Allowing empty clusters</h2>
|
|
<p>If you would like to allow empty clusters to exist, instead of reinitializing them, simply specify the <code>-e</code> (<code>–allow_empty_clusters</code>) option. Note that when you save your clusters, some of the clusters may be filled with NaNs. This is expected behavior – if a cluster has no points, the concept of a centroid makes no sense.</p>
|
|
<div class="fragment"><div class="line">$ mlpack_kmeans -c 5 -i dataset.csv -v -o assignments.csv -C centroids.csv</div></div><!-- fragment --><h2><a class="anchor" id="cli_ex4_kmtut"></a>
|
|
Limiting the maximum number of iterations</h2>
|
|
<p>As mentioned earlier, the k-means algorithm can often fail to converge. In such a situation, it may be useful to stop the algorithm by way of limiting the maximum number of iterations. This can be done with the <code>-m</code> (<code>–max_iterations</code>) parameter, which is set to 1000 by default. If the maximum number of iterations is 0, the algorithm will run until convergence – or potentially forever. The example below sets a maximum of 250 iterations.</p>
|
|
<div class="fragment"><div class="line">$ mlpack_kmeans -c 5 -i dataset.csv -v -o assignments.csv -m 250</div></div><!-- fragment --><h2><a class="anchor" id="cli_ex6_kmtut"></a>
|
|
Using Bradley-Fayyad "refined start"</h2>
|
|
<p>The method proposed by Bradley and Fayyad in their paper "Refining initial
|
|
points for k-means clustering" is implemented in <b>mlpack</b>. This strategy samples points from the dataset and runs k-means clustering on those points multiple times, saving the resulting clusters. Then, k-means clustering is run on those clusters, yielding the original number of clusters. The centroids of those resulting clusters are used as initial centroids for k-means clustering on the entire dataset.</p>
|
|
<p>This technique generally gives better initial points than the default random partitioning, but depending on the parameters, it can take much longer. This initialization technique is enabled with the <code>-r</code> (<code>–refined_start</code>) option. The <code>-S</code> (<code>–samplings</code>) parameter controls how many samplings of the dataset are performed, and the <code>-p</code> (<code>–percentage</code>) parameter controls how much of the dataset is randomly sampled for each sampling (it must be between 0.0 and 1.0). For more information on the refined start technique, see the paper referenced in the introduction of this tutorial.</p>
|
|
<p>The example below performs k-means clustering, giving 5 clusters, using the refined start technique, sampling 10% of the dataset 25 times to produce the initial centroids.</p>
|
|
<div class="fragment"><div class="line">$ mlpack_kmeans -c 5 -i dataset.csv -v -o assignments.csv -r -S 25 -p 0.2</div></div><!-- fragment --><h2><a class="anchor" id="cli_ex7_kmtut"></a>
|
|
Using different k-means algorithms</h2>
|
|
<p>The <code>mlpack_kmeans</code> program implements six different strategies for clustering; each of these gives the exact same results, but will have different runtimes. The particular algorithm to use can be specified with the <code>-a</code> or <code>–algorithm</code> option. The choices are:</p>
|
|
<ul>
|
|
<li><code>naive:</code> the standard Lloyd iteration; takes <img class="formulaInl" alt="$O(kN)$" src="form_134.png"/> time per iteration.</li>
|
|
<li><code>pelleg-moore</code>: the 'blacklist' algorithm, which builds a kd-tree on the data. This can be fast when k is small and the dimensionality is reasonably low.</li>
|
|
<li><code>elkan:</code> Elkan's algorithm for k-means, which maintains upper and lower distance bounds between each point and each centroid. This can be very fast, but it does not scale well to the case of large N or k, and uses a lot of memory.</li>
|
|
<li><code>hamerly:</code> Hamerly's algorithm is a variant of Elkan's algorithm that handles memory usage much better and thus can operate with much larger datasets than Elkan's algorithm.</li>
|
|
<li><code>dualtree:</code> The dual-tree algorithm for k-means builds a kd-tree on both the centroids and the points in order to prune away as much work as possible. This algorithm is most effective when both N and k are large.</li>
|
|
<li><code>dualtree-covertree</code>: This is the dual-tree algorithm using cover trees instead of kd-trees. It satisfies the runtime guarantees specified in the dual-tree k-means paper.</li>
|
|
</ul>
|
|
<p>In general, the <code>naive</code> algorithm will be much slower than the others on datasets that are larger than tiny.</p>
|
|
<p>The example below uses the <code>dualtree</code> algorithm to perform k-means clustering with 5 clusters on the dataset in <code>dataset.csv</code>, using the initial centroids in <code>initial_centroids.csv</code>, saving the resulting cluster assignments to <code>assignments.csv:</code> </p>
|
|
<div class="fragment"><div class="line">$ mlpack_kmeans -i dataset.csv -c 5 -v -I initial_centroids.csv -a dualtree \</div><div class="line">> -o assignments.csv</div></div><!-- fragment --><h1><a class="anchor" id="kmeans_kmtut"></a>
|
|
The 'KMeans' class</h1>
|
|
<p>The <code>KMeans<></code> class (with default template parameters) provides a simple way to run k-means clustering using <b>mlpack</b> in C++. The default template parameters for <code>KMeans<></code> will initialize cluster assignments randomly and disallow empty clusters. When an empty cluster is encountered, the point furthest from the cluster with maximum variance is set to the centroid of the empty cluster.</p>
|
|
<h2><a class="anchor" id="kmeans_ex1_kmtut"></a>
|
|
Running k-means and getting cluster assignments</h2>
|
|
<p>The simplest way to use the <code>KMeans<></code> class is to pass in a dataset and a number of clusters, and receive the cluster assignments in return. Note that the dataset must be column-major – that is, one column corresponds to one point. See <a class="el" href="matrices.html">the matrices guide</a> for more information.</p>
|
|
<div class="fragment"><div class="line"><span class="preprocessor">#include <<a class="code" href="kmeans_8hpp.html">mlpack/methods/kmeans/kmeans.hpp</a>></span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacemlpack_1_1kmeans.html">mlpack::kmeans</a>;</div><div class="line"></div><div class="line"><span class="comment">// The dataset we are clustering.</span></div><div class="line"><span class="keyword">extern</span> arma::mat data;</div><div class="line"><span class="comment">// The number of clusters we are getting.</span></div><div class="line"><span class="keyword">extern</span> <span class="keywordtype">size_t</span> clusters;</div><div class="line"></div><div class="line"><span class="comment">// The assignments will be stored in this vector.</span></div><div class="line">arma::Row<size_t> assignments;</div><div class="line"></div><div class="line"><span class="comment">// Initialize with the default arguments.</span></div><div class="line"><a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html">KMeans<></a> k;</div><div class="line">k.<a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html#a4acb46f121570f776571ba8771fe512c">Cluster</a>(data, clusters, assignments);</div></div><!-- fragment --><p>Now, the vector <code>assignments</code> holds the cluster assignments of each point in the dataset.</p>
|
|
<h2><a class="anchor" id="kmeans_ex2_kmtut"></a>
|
|
Running k-means and getting centroids of clusters</h2>
|
|
<p>Often it is useful to not only have the cluster assignments, but the centroids of each cluster. Another overload of <code>Cluster()</code> makes this easily possible:</p>
|
|
<div class="fragment"><div class="line"><span class="preprocessor">#include <<a class="code" href="kmeans_8hpp.html">mlpack/methods/kmeans/kmeans.hpp</a>></span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacemlpack_1_1kmeans.html">mlpack::kmeans</a>;</div><div class="line"></div><div class="line"><span class="comment">// The dataset we are clustering.</span></div><div class="line"><span class="keyword">extern</span> arma::mat data;</div><div class="line"><span class="comment">// The number of clusters we are getting.</span></div><div class="line"><span class="keyword">extern</span> <span class="keywordtype">size_t</span> clusters;</div><div class="line"></div><div class="line"><span class="comment">// The assignments will be stored in this vector.</span></div><div class="line">arma::Row<size_t> assignments;</div><div class="line"><span class="comment">// The centroids will be stored in this matrix.</span></div><div class="line">arma::mat centroids;</div><div class="line"></div><div class="line"><span class="comment">// Initialize with the default arguments.</span></div><div class="line"><a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html">KMeans<></a> k;</div><div class="line">k.<a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html#a4acb46f121570f776571ba8771fe512c">Cluster</a>(data, clusters, assignments, centroids);</div></div><!-- fragment --><p>Note that the centroids matrix has columns equal to the number of clusters and rows equal to the dimensionality of the dataset. Each column represents the centroid of the according cluster – <code>centroids.col(0)</code> represents the centroid of the first cluster.</p>
|
|
<h2><a class="anchor" id="kmeans_ex3_kmtut"></a>
|
|
Limiting the maximum number of iterations</h2>
|
|
<p>The first argument to the constructor allows specification of the maximum number of iterations. This is useful because often, the k-means algorithm does not converge, and is terminated after a number of iterations. Setting this parameter to 0 indicates that the algorithm will run until convergence – note that in some cases, convergence may never happen. The default maximum number of iterations is 1000.</p>
|
|
<div class="fragment"><div class="line"><span class="comment">// The first argument is the maximum number of iterations. Here we set it to</span></div><div class="line"><span class="comment">// 500 iterations.</span></div><div class="line">KMeans<> k(500);</div></div><!-- fragment --><p>Then you can run <code>Cluster()</code> as normal.</p>
|
|
<h2><a class="anchor" id="kmeans_ex5_kmtut"></a>
|
|
Setting initial cluster assignments</h2>
|
|
<p>If you have an initial guess for the cluster assignments for each point, you can fill the assignments vector with the guess and then pass an extra boolean (initialAssignmentGuess) as true to the <code>Cluster()</code> method. Below are examples for either overload of <code>Cluster()</code>.</p>
|
|
<div class="fragment"><div class="line"><span class="preprocessor">#include <<a class="code" href="kmeans_8hpp.html">mlpack/methods/kmeans/kmeans.hpp</a>></span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacemlpack_1_1kmeans.html">mlpack::kmeans</a>;</div><div class="line"></div><div class="line"><span class="comment">// The dataset we are clustering on.</span></div><div class="line"><span class="keyword">extern</span> arma::mat dataset;</div><div class="line"><span class="comment">// The number of clusters we are obtaining.</span></div><div class="line"><span class="keyword">extern</span> <span class="keywordtype">size_t</span> clusters;</div><div class="line"></div><div class="line"><span class="comment">// A vector pre-filled with initial assignment guesses.</span></div><div class="line"><span class="keyword">extern</span> arma::Row<size_t> assignments;</div><div class="line"></div><div class="line"><a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html">KMeans<></a> k;</div><div class="line"></div><div class="line"><span class="comment">// The boolean set to true indicates that our assignments vector is filled with</span></div><div class="line"><span class="comment">// initial guesses.</span></div><div class="line">k.<a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html#a4acb46f121570f776571ba8771fe512c">Cluster</a>(dataset, clusters, assignments, <span class="keyword">true</span>);</div></div><!-- fragment --><div class="fragment"><div class="line"><span class="preprocessor">#include <<a class="code" href="kmeans_8hpp.html">mlpack/methods/kmeans/kmeans.hpp</a>></span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacemlpack_1_1kmeans.html">mlpack::kmeans</a>;</div><div class="line"></div><div class="line"><span class="comment">// The dataset we are clustering on.</span></div><div class="line"><span class="keyword">extern</span> arma::mat dataset;</div><div class="line"><span class="comment">// The number of clusters we are obtaining.</span></div><div class="line"><span class="keyword">extern</span> <span class="keywordtype">size_t</span> clusters;</div><div class="line"></div><div class="line"><span class="comment">// A vector pre-filled with initial assignment guesses.</span></div><div class="line"><span class="keyword">extern</span> arma::Row<size_t> assignments;</div><div class="line"></div><div class="line"><span class="comment">// This will hold the centroids of the finished clusters.</span></div><div class="line">arma::mat centroids;</div><div class="line"></div><div class="line"><a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html">KMeans<></a> k;</div><div class="line"></div><div class="line"><span class="comment">// The boolean set to true indicates that our assignments vector is filled with</span></div><div class="line"><span class="comment">// initial guesses.</span></div><div class="line">k.<a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html#a4acb46f121570f776571ba8771fe512c">Cluster</a>(dataset, clusters, assignments, centroids, <span class="keyword">true</span>);</div></div><!-- fragment --><dl class="section note"><dt>Note</dt><dd>If you have a heuristic or algorithm which makes initial guesses, a more elegant solution is to create a new class fulfilling the InitialPartitionPolicy template policy. See <a class="el" href="kmtutorial.html#kmeans_initial_partition_kmtut">the section about changing the initial partitioning strategy</a> for more details.</dd></dl>
|
|
<dl class="section user"><dt></dt><dd></dd></dl>
|
|
<dl class="section note"><dt>Note</dt><dd>If you set the InitialPartitionPolicy parameter to something other than the default but give an initial cluster assignment guess, the InitialPartitionPolicy will not be used to initialize the algorithm. See <a class="el" href="kmtutorial.html#kmeans_initial_partition_kmtut">the section about changing the initial partitioning strategy</a> for more details.</dd></dl>
|
|
<h2><a class="anchor" id="kmeans_ex6_kmtut"></a>
|
|
Setting initial cluster centroids</h2>
|
|
<p>An equally important option to being able to make initial cluster assignment guesses is to make initial cluster centroid guesses without having to assign each point in the dataset to an initial cluster. This is similar to the previous section, but now you must pass two extra booleans – the first (initialAssignmentGuess) as false, indicating that there are not initial cluster assignment guesses, and the second (initialCentroidGuess) as true, indicating that the centroids matrix is filled with initial centroid guesses.</p>
|
|
<p>This, of course, only works with the overload of <code>Cluster()</code> that takes a matrix to put the resulting centroids in. Below is an example.</p>
|
|
<div class="fragment"><div class="line"><span class="preprocessor">#include <<a class="code" href="kmeans_8hpp.html">mlpack/methods/kmeans/kmeans.hpp</a>></span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacemlpack_1_1kmeans.html">mlpack::kmeans</a>;</div><div class="line"></div><div class="line"><span class="comment">// The dataset we are clustering on.</span></div><div class="line"><span class="keyword">extern</span> arma::mat dataset;</div><div class="line"><span class="comment">// The number of clusters we are obtaining.</span></div><div class="line"><span class="keyword">extern</span> <span class="keywordtype">size_t</span> clusters;</div><div class="line"></div><div class="line"><span class="comment">// A matrix pre-filled with guesses for the initial cluster centroids.</span></div><div class="line"><span class="keyword">extern</span> arma::mat centroids;</div><div class="line"></div><div class="line"><span class="comment">// This will be filled with the final cluster assignments for each point.</span></div><div class="line">arma::Row<size_t> assignments;</div><div class="line"></div><div class="line"><a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html">KMeans<></a> k;</div><div class="line"></div><div class="line"><span class="comment">// Remember, the first boolean indicates that we are not giving initial</span></div><div class="line"><span class="comment">// assignment guesses, and the second boolean indicates that we are giving</span></div><div class="line"><span class="comment">// initial centroid guesses.</span></div><div class="line">k.<a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html#a4acb46f121570f776571ba8771fe512c">Cluster</a>(dataset, clusters, assignments, centroids, <span class="keyword">false</span>, <span class="keyword">true</span>);</div></div><!-- fragment --><dl class="section note"><dt>Note</dt><dd>If you have a heuristic or algorithm which makes initial guesses, a more elegant solution is to create a new class fulfilling the InitialPartitionPolicy template policy. See <a class="el" href="kmtutorial.html#kmeans_initial_partition_kmtut">the section about changing the initial partitioning strategy</a> for more details.</dd></dl>
|
|
<dl class="section user"><dt></dt><dd></dd></dl>
|
|
<dl class="section note"><dt>Note</dt><dd>If you set the InitialPartitionPolicy parameter to something other than the default but give an initial cluster centroid guess, the InitialPartitionPolicy will not be used to initialize the algorithm. See <a class="el" href="kmtutorial.html#kmeans_initial_partition_kmtut">the section about changing the initial partitioning strategy</a> for more details.</dd></dl>
|
|
<h2><a class="anchor" id="kmeans_ex7_kmtut"></a>
|
|
Running sparse k-means</h2>
|
|
<p>The <code>Cluster()</code> function can work on both sparse and dense matrices, so all of the above examples can be used with sparse matrices instead, if the fifth template parameter is modified. Below is a simple example. Note that the centroids are returned as a dense matrix, because the centroids of collections of sparse points are not generally sparse.</p>
|
|
<div class="fragment"><div class="line"><span class="comment">// The sparse dataset.</span></div><div class="line"><span class="keyword">extern</span> arma::sp_mat sparseDataset;</div><div class="line"><span class="comment">// The number of clusters.</span></div><div class="line"><span class="keyword">extern</span> <span class="keywordtype">size_t</span> clusters;</div><div class="line"></div><div class="line"><span class="comment">// The assignments will be stored in this vector.</span></div><div class="line">arma::Row<size_t> assignments;</div><div class="line"><span class="comment">// The centroids of each cluster will be stored in this sparse matrix.</span></div><div class="line">arma::sp_mat sparseCentroids;</div><div class="line"></div><div class="line"><span class="comment">// We must change the fifth (and last) template parameter.</span></div><div class="line">KMeans<<a class="code" href="namespacemlpack_1_1metric.html#a012695bde5dbf7ded7f7a180b5f3f512">metric::EuclideanDistance</a>, SampleInitialization, MaxVarianceNewCluster,</div><div class="line"> NaiveKMeans, arma::sp_mat> k;</div><div class="line">k.<a class="code" href="classmlpack_1_1kmeans_1_1KMeans.html#a4acb46f121570f776571ba8771fe512c">Cluster</a>(sparseDataset, clusters, assignments, sparseCentroids);</div></div><!-- fragment --><h1><a class="anchor" id="kmeans_template_kmtut"></a>
|
|
Template parameters for the 'KMeans' class</h1>
|
|
<p>The <code>KMeans<></code> class also takes three template parameters, which can be modified to change the behavior of the k-means algorithm. There are three template parameters:</p>
|
|
<ul>
|
|
<li><code>MetricType:</code> controls the distance metric used for clustering (by default, the squared Euclidean distance is used)</li>
|
|
<li><code>InitialPartitionPolicy:</code> the method by which initial clusters are set; by default, <a class="el" href="classmlpack_1_1kmeans_1_1SampleInitialization.html">SampleInitialization</a> is used</li>
|
|
<li><code>EmptyClusterPolicy:</code> the action taken when an empty cluster is encountered; by default, <a class="el" href="classmlpack_1_1kmeans_1_1MaxVarianceNewCluster.html">MaxVarianceNewCluster</a> is used</li>
|
|
<li><code>LloydStepType:</code> this defines the strategy used to make a single Lloyd iteration; by default this is the typical Lloyd iteration specified in <a class="el" href="classmlpack_1_1kmeans_1_1NaiveKMeans.html">NaiveKMeans</a></li>
|
|
<li><code>MatType:</code> type of data matrix to use for clustering</li>
|
|
</ul>
|
|
<p>The class is defined like below:</p>
|
|
<div class="fragment"><div class="line"><span class="keyword">template</span><</div><div class="line"> <span class="keyword">typename</span> DistanceMetric = <a class="code" href="namespacemlpack_1_1metric.html#add19cf9fc6f452a0dad6635a93451be7">mlpack::metric::SquaredEuclideanDistance</a>,</div><div class="line"> <span class="keyword">typename</span> InitialPartitionPolicy = SampleInitialization,</div><div class="line"> <span class="keyword">typename</span> EmptyClusterPolicy = MaxVarianceNewCluster,</div><div class="line"> <span class="keyword">template</span><<span class="keyword">class</span>, <span class="keyword">class</span>> <span class="keyword">class </span>LloydStepType = NaiveKMeans,</div><div class="line"> <span class="keyword">typename</span> MatType = arma::mat</div><div class="line">></div><div class="line"><span class="keyword">class </span>KMeans;</div></div><!-- fragment --><p>In the following sections, each policy is described further, with examples of how to modify them.</p>
|
|
<h2><a class="anchor" id="kmeans_metric_kmtut"></a>
|
|
Changing the distance metric used for k-means</h2>
|
|
<p>Most machine learning algorithms in <b>mlpack</b> support modifying the distance metric, and <code>KMeans<></code> is no exception. Similar to <a class="el" href="classmlpack_1_1neighbor_1_1NeighborSearch.html">NeighborSearch</a> (see <a class="el" href="nstutorial.html#metric_type_doc_nstut">the section in the NeighborSearch tutorial</a>), any class in <a class="el" href="namespacemlpack_1_1metric.html">mlpack::metric</a> can be given as an argument. The <a class="el" href="classmlpack_1_1metric_1_1LMetric.html" title="The L_p metric for arbitrary integer p, with an option to take the root. ">mlpack::metric::LMetric</a> class is a good example implementation.</p>
|
|
<p>A class fulfilling the MetricType policy must provide the following two functions:</p>
|
|
<div class="fragment"><div class="line"><span class="comment">// Empty constructor is required.</span></div><div class="line">MetricType();</div><div class="line"></div><div class="line"><span class="comment">// Computer the distance between two points.</span></div><div class="line"><span class="keyword">template</span><<span class="keyword">typename</span> VecType></div><div class="line"><span class="keywordtype">double</span> Evaluate(<span class="keyword">const</span> VecType& a, <span class="keyword">const</span> VecType& b);</div></div><!-- fragment --><p>Most of the standard metrics that could be used are stateless and therefore the <code>Evaluate()</code> method is implemented statically. However, there are metrics, such as the Mahalanobis distance (<a class="el" href="classmlpack_1_1metric_1_1MahalanobisDistance.html" title="The Mahalanobis distance, which is essentially a stretched Euclidean distance. ">mlpack::metric::MahalanobisDistance</a>), that store state. To this end, an instantiated MetricType object is stored within the <code>KMeans</code> class. The example below shows how to pass an instantiated MahalanobisDistance in the constructor.</p>
|
|
<div class="fragment"><div class="line"><span class="comment">// The initialized Mahalanobis distance.</span></div><div class="line"><span class="keyword">extern</span> <a class="code" href="classmlpack_1_1metric_1_1MahalanobisDistance.html">mlpack::metric::MahalanobisDistance</a> distance;</div><div class="line"></div><div class="line"><span class="comment">// We keep the default arguments for the maximum number of iterations, but pass</span></div><div class="line"><span class="comment">// our instantiated metric.</span></div><div class="line">KMeans<mlpack::metric::MahalanobisDistance> k(1000, distance);</div></div><!-- fragment --><dl class="section note"><dt>Note</dt><dd>While the MetricType policy only requires two methods, one of which is an empty constructor, more can always be added. <a class="el" href="classmlpack_1_1metric_1_1MahalanobisDistance.html" title="The Mahalanobis distance, which is essentially a stretched Euclidean distance. ">mlpack::metric::MahalanobisDistance</a> also has constructors with parameters, because it is a stateful metric.</dd></dl>
|
|
<h2><a class="anchor" id="kmeans_initial_partition_kmtut"></a>
|
|
Changing the initial partitioning strategy used for k-means</h2>
|
|
<p>There have been many initial cluster strategies for k-means proposed in the literature. Fortunately, the <code>KMeans<></code> class makes it very easy to implement one of these methods and plug it in without needing to modify the existing algorithm code at all.</p>
|
|
<p>By default, the <code>KMeans<></code> class uses <a class="el" href="classmlpack_1_1kmeans_1_1SampleInitialization.html">mlpack::kmeans::SampleInitialization</a>, which randomly samples points as initial centroids. However, writing a new policy is simple; it needs to only implement the following functions:</p>
|
|
<div class="fragment"><div class="line"><span class="comment">// Empty constructor is required.</span></div><div class="line">InitialPartitionPolicy();</div><div class="line"></div><div class="line"><span class="comment">// Only *one* of the following two functions is required! You should implement</span></div><div class="line"><span class="comment">// whichever you find more convenient to implement.</span></div><div class="line"></div><div class="line"><span class="comment">// This function is called to initialize the clusters and returns centroids.</span></div><div class="line"><span class="keyword">template</span><<span class="keyword">typename</span> MatType></div><div class="line"><span class="keywordtype">void</span> Cluster(MatType& data,</div><div class="line"> <span class="keyword">const</span> <span class="keywordtype">size_t</span> clusters,</div><div class="line"> arma::mat& centroids);</div><div class="line"></div><div class="line"><span class="comment">// This function is called to initialize the clusters and returns individual</span></div><div class="line"><span class="comment">// point assignments. The centroids will then be calculated from the given</span></div><div class="line"><span class="comment">// assignments.</span></div><div class="line"><span class="keyword">template</span><<span class="keyword">typename</span> MatType></div><div class="line"><span class="keywordtype">void</span> Cluster(MatType& data,</div><div class="line"> <span class="keyword">const</span> <span class="keywordtype">size_t</span> clusters,</div><div class="line"> arma::Row<size_t> assignments);</div></div><!-- fragment --><p>The templatization of the <code>Cluster()</code> function allows both dense and sparse matrices to be passed in. If the desired policy does not work with sparse (or dense) matrices, then the method can be written specifically for one type of matrix – however, be warned that if you try to use <code>KMeans</code> with that policy and the wrong type of matrix, you will get many ugly compilation errors!</p>
|
|
<div class="fragment"><div class="line"><span class="comment">// The Cluster() function specialized for dense matrices.</span></div><div class="line"><span class="keywordtype">void</span> Cluster(arma::mat& data,</div><div class="line"> <span class="keyword">const</span> <span class="keywordtype">size_t</span> clusters,</div><div class="line"> arma::Row<size_t> assignments);</div></div><!-- fragment --><p>Note that only one of the two possible <code>Cluster()</code> functions are required. This is because sometimes it is easier to express an initial partitioning policy as something that returns point assignments, and sometimes it is easier to express the policy as something that returns centroids. The KMeans<> class will use whichever of these two functions is given; if both are given, the overload that returns centroids will be preferred.</p>
|
|
<p>One alternate to the default SampleInitialization policy is the RefinedStart policy, which is an implementation of the Bradley and Fayyad approach for finding initial points detailed in "Refined initial points for k-means
|
|
clustering" and other places in this document. Another option is the RandomPartition class, which randomly assigns points to clusters, but this may not work very well for most settings. See the documentation for <a class="el" href="classmlpack_1_1kmeans_1_1RefinedStart.html" title="A refined approach for choosing initial points for k-means clustering. ">mlpack::kmeans::RefinedStart</a> and <a class="el" href="classmlpack_1_1kmeans_1_1RandomPartition.html" title="A very simple partitioner which partitions the data randomly into the number of desired clusters...">mlpack::kmeans::RandomPartition</a> for more information.</p>
|
|
<p>If the <code>Cluster()</code> method returns point assignments instead of centroids, then valid initial assignments must be returned for every point in the dataset.</p>
|
|
<p>As with the MetricType template parameter, an initialized InitialPartitionPolicy can be passed to the constructor of <code>KMeans</code> as a fourth argument.</p>
|
|
<h2><a class="anchor" id="kmeans_empty_cluster_kmtut"></a>
|
|
Changing the action taken when an empty cluster is encountered</h2>
|
|
<p>Sometimes, during clustering, a situation will arise where a cluster has no points in it. The <code>KMeans</code> class allows easy customization of the action to be taken when this occurs. By default, the point furthest from the centroid of the cluster with maximum variance is taken as the centroid of the empty cluster; this is implemented in the <a class="el" href="classmlpack_1_1kmeans_1_1MaxVarianceNewCluster.html" title="When an empty cluster is detected, this class takes the point furthest from the centroid of the clust...">mlpack::kmeans::MaxVarianceNewCluster</a> class. Another alternate choice is the <a class="el" href="classmlpack_1_1kmeans_1_1AllowEmptyClusters.html" title="Policy which allows K-Means to create empty clusters without any error being reported. ">mlpack::kmeans::AllowEmptyClusters</a> class, which simply allows empty clusters to persist.</p>
|
|
<p>A custom policy can be written and it must implement the following methods:</p>
|
|
<div class="fragment"><div class="line"><span class="comment">// Empty constructor is required.</span></div><div class="line">EmptyClusterPolicy();</div><div class="line"></div><div class="line"><span class="comment">// This function is called when an empty cluster is encountered. emptyCluster</span></div><div class="line"><span class="comment">// indicates the cluster which is empty, and then the clusterCounts and</span></div><div class="line"><span class="comment">// assignments are meant to be modified by the function. The function should</span></div><div class="line"><span class="comment">// return the number of modified points.</span></div><div class="line"><span class="keyword">template</span><<span class="keyword">typename</span> MatType></div><div class="line"><span class="keywordtype">size_t</span> EmptyCluster(<span class="keyword">const</span> MatType& data,</div><div class="line"> <span class="keyword">const</span> <span class="keywordtype">size_t</span> emptyCluster,</div><div class="line"> <span class="keyword">const</span> MatType& centroids,</div><div class="line"> arma::Col<size_t>& clusterCounts,</div><div class="line"> arma::Row<size_t>& assignments);</div></div><!-- fragment --><p>The <code>EmptyCluster()</code> function is called for each cluster that is empty at each iteration of the algorithm. As with InitialPartitionPolicy, the <code>EmptyCluster()</code> function does not need to be generalized to support both dense and sparse matrices – but usage with the wrong type of matrix will cause compilation errors.</p>
|
|
<p>Like the other template parameters to <code>KMeans</code>, EmptyClusterPolicy implementations that have state can be passed to the constructor of <code>KMeans</code> as a fifth argument. See the kmeans::KMeans documentation for further details.</p>
|
|
<h2><a class="anchor" id="kmeans_lloyd_kmtut"></a>
|
|
The LloydStepType template parameter</h2>
|
|
<p>The internal algorithm used for a single step of the k-means algorithm can easily be changed; <b>mlpack</b> implements several existing classes that satisfy the <code>LloydStepType</code> policy:</p>
|
|
<ul>
|
|
<li><a class="el" href="classmlpack_1_1kmeans_1_1NaiveKMeans.html" title="This is an implementation of a single iteration of Lloyd's algorithm for k-means. ...">mlpack::kmeans::NaiveKMeans</a></li>
|
|
<li><a class="el" href="classmlpack_1_1kmeans_1_1ElkanKMeans.html">mlpack::kmeans::ElkanKMeans</a></li>
|
|
<li><a class="el" href="classmlpack_1_1kmeans_1_1HamerlyKMeans.html">mlpack::kmeans::HamerlyKMeans</a></li>
|
|
<li><a class="el" href="classmlpack_1_1kmeans_1_1PellegMooreKMeans.html" title="An implementation of Pelleg-Moore's 'blacklist' algorithm for k-means clustering. ...">mlpack::kmeans::PellegMooreKMeans</a></li>
|
|
<li><a class="el" href="classmlpack_1_1kmeans_1_1DualTreeKMeans.html" title="An algorithm for an exact Lloyd iteration which simply uses dual-tree nearest-neighbor search to find...">mlpack::kmeans::DualTreeKMeans</a></li>
|
|
</ul>
|
|
<p>Note that the <code>LloydStepType</code> policy is itself a template template parameter, and must accept two template parameters of its own:</p>
|
|
<ul>
|
|
<li><code>MetricType:</code> the type of metric to use</li>
|
|
<li><code>MatType:</code> the type of data matrix to use</li>
|
|
</ul>
|
|
<p>The <code>LloydStepType</code> policy also mandates three functions:</p>
|
|
<ul>
|
|
<li>a constructor: <code>"LloydStepType(const MatType& dataset, MetricType&
|
|
metric);"</code> </li>
|
|
<li>an <code>Iterate()</code> function:</li>
|
|
</ul>
|
|
<div class="fragment"><div class="line"></div><div class="line"><span class="keywordtype">double</span> Iterate(<span class="keyword">const</span> arma::mat& centroids,</div><div class="line"> arma::mat& newCentroids,</div><div class="line"> arma::Col<size_t>& counts);</div></div><!-- fragment --><ul>
|
|
<li>a function to get the number of distance calculations:</li>
|
|
</ul>
|
|
<div class="fragment"><div class="line"><span class="keywordtype">size_t</span> DistanceCalculations()<span class="keyword"> const </span>{ <span class="keywordflow">return</span> distanceCalculations; }</div></div><!-- fragment --><p>Note that <code>Iterate()</code> does not need to return valid centroids if the cluster is empty. This is because <code>EmptyClusterPolicy</code> will handle the empty centroid. This behavior can be used to avoid small amounts of computation.</p>
|
|
<p>For examples, see the five aforementioned implementations of classes that satisfy the <code>LloydStepType</code> policy.</p>
|
|
<h1><a class="anchor" id="further_doc_kmtut"></a>
|
|
Further documentation</h1>
|
|
<p>For further documentation on the KMeans class, consult the <a class="el" href="classmlpack_1_1kmeans_1_1KMeans.html">complete API documentation</a>. </p>
|
|
</div></div><!-- contents -->
|
|
<!-- start footer part -->
|
|
<hr class="footer"/><address class="footer"><small>
|
|
Generated by  <a href="http://www.doxygen.org/index.html">
|
|
<img class="footer" src="doxygen.png" alt="doxygen"/>
|
|
</a> 1.8.11
|
|
</small></address>
|
|
</body>
|
|
<script type="text/javascript">
|
|
var x = document.getElementsByClassName("formulaDsp");
|
|
var i;
|
|
for (i = 0; i < x.length; i++)
|
|
{
|
|
x[i].width /= 4;
|
|
}
|
|
</script>
|
|
</html>
|