public class StreamingKMeansModel extends KMeansModel implements org.apache.spark.internal.Logging
The update algorithm uses the "mini-batch" KMeans rule, generalized to incorporate forgetfulness (i.e. decay). The update rule (for each cluster) is:
$$ \begin{align} c_{t+1} &= [(c_t * n_t * a) + (x_t * m_t)] / [n_t + m_t] \\ n_{t+1} &= n_t * a + m_t \end{align} $$
Where c_t is the previously estimated centroid for that cluster, n_t is the number of points assigned to it thus far, x_t is the centroid estimated on the current batch, and m_t is the number of points assigned to that centroid in the current batch.
The decay factor 'a' scales the contribution of the clusters as estimated thus far, by applying a as a discount weighting on the current point when evaluating new incoming data. If a=1, all batches are weighted equally. If a=0, new centroids are determined entirely by recent data. Lower values correspond to more forgetting.
Decay can optionally be specified by a half life and associated time unit. The time unit can either be a batch of data or a single data point. Considering data arrived at time t, the half life h is defined such that at time t + h the discount applied to the data from t is 0.5. The definition remains the same whether the time unit is given as batches or points.
KMeansModel.Cluster$, KMeansModel.SaveLoadV1_0$, KMeansModel.SaveLoadV2_0$
Constructor and Description |
---|
StreamingKMeansModel(Vector[] clusterCenters,
double[] clusterWeights) |
Modifier and Type | Method and Description |
---|---|
Vector[] |
clusterCenters() |
double[] |
clusterWeights() |
StreamingKMeansModel |
update(RDD<Vector> data,
double decayFactor,
String timeUnit)
Perform a k-means update on a batch of data.
|
computeCost, distanceMeasure, k, load, predict, predict, predict, save, trainingCost
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize
public StreamingKMeansModel(Vector[] clusterCenters, double[] clusterWeights)
public Vector[] clusterCenters()
clusterCenters
in class KMeansModel
public double[] clusterWeights()
public StreamingKMeansModel update(RDD<Vector> data, double decayFactor, String timeUnit)
data
- (undocumented)decayFactor
- (undocumented)timeUnit
- (undocumented)