public class BucketedRandomProjectionLSH extends Estimator<T> implements BucketedRandomProjectionLSHParams, HasSeed
BucketedRandomProjectionLSH implements Locality Sensitive Hashing functions for
Euclidean distance metrics.
The input is dense or sparse vectors, each of which represents a point in the Euclidean distance space. The output will be vectors of configurable dimension. Hash values in the same dimension are calculated by the same hash function.
References:
1. Wikipedia on Stable Distributions
2. Wang, Jingdong et al. "Hashing for similarity search: A survey." arXiv preprint arXiv:1408.2927 (2014).
| Constructor and Description |
|---|
BucketedRandomProjectionLSH() |
BucketedRandomProjectionLSH(String uid) |
| Modifier and Type | Method and Description |
|---|---|
DoubleParam |
bucketLength()
The length of each hash bucket, a larger bucket lowers the false negative rate.
|
BucketedRandomProjectionLSH |
copy(ParamMap extra)
Creates a copy of this instance with the same UID and some extra params.
|
T |
fit(Dataset<?> dataset)
Fits a model to the input data.
|
Param<String> |
inputCol()
Param for input column name.
|
static BucketedRandomProjectionLSH |
load(String path) |
IntParam |
numHashTables()
Param for the number of hash tables used in LSH OR-amplification.
|
Param<String> |
outputCol()
Param for output column name.
|
static MLReader<T> |
read() |
LongParam |
seed()
Param for random seed.
|
BucketedRandomProjectionLSH |
setBucketLength(double value) |
BucketedRandomProjectionLSH |
setInputCol(String value) |
BucketedRandomProjectionLSH |
setNumHashTables(int value) |
BucketedRandomProjectionLSH |
setOutputCol(String value) |
BucketedRandomProjectionLSH |
setSeed(long value) |
StructType |
transformSchema(StructType schema)
Check transform validity and derive the output schema from the input schema.
|
String |
uid()
An immutable unique ID for the object and its derivatives.
|
paramsequals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitgetBucketLengthclear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwntoStringgetNumHashTables, validateAndTransformSchemagetInputColgetOutputColwritesave$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitializepublic BucketedRandomProjectionLSH(String uid)
public BucketedRandomProjectionLSH()
public static BucketedRandomProjectionLSH load(String path)
public static MLReader<T> read()
public final LongParam seed()
HasSeedpublic DoubleParam bucketLength()
BucketedRandomProjectionLSHParams(max L2 norm of input vectors) / bucketLength.
If input vectors are normalized, 1-10 times of pow(numRecords, -1/inputDim) would be a reasonable value
bucketLength in interface BucketedRandomProjectionLSHParamspublic String uid()
Identifiableuid in interface Identifiablepublic BucketedRandomProjectionLSH setInputCol(String value)
public BucketedRandomProjectionLSH setOutputCol(String value)
public BucketedRandomProjectionLSH setNumHashTables(int value)
public BucketedRandomProjectionLSH setBucketLength(double value)
public BucketedRandomProjectionLSH setSeed(long value)
public StructType transformSchema(StructType schema)
PipelineStage
We check validity for interactions between parameters during transformSchema and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate().
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
transformSchema in class PipelineStageschema - (undocumented)public BucketedRandomProjectionLSH copy(ParamMap extra)
ParamsdefaultCopy().copy in interface Paramscopy in class Estimator<BucketedRandomProjectionLSHModel>extra - (undocumented)public T fit(Dataset<?> dataset)
Estimatorpublic final Param<String> inputCol()
HasInputColinputCol in interface HasInputColpublic final IntParam numHashTables()
LSHParamsLSH OR-amplification can be used to reduce the false negative rate. Higher values for this param lead to a reduced false negative rate, at the expense of added computational complexity.
numHashTables in interface LSHParamspublic final Param<String> outputCol()
HasOutputColoutputCol in interface HasOutputCol