jsat.text.wordweighting

Class WordWeighting

java.lang.Object

jsat.math.IndexFunction

jsat.text.wordweighting.WordWeighting

All Implemented Interfaces:

Serializable, Function

Direct Known Subclasses:

BinaryWordPresent, OkapiBM25, TfIdf, WordCount

public abstract class WordWeighting
extends IndexFunction

WordWeighting is an index function specifically mean for modifying the values of a vectors used for a bag-of-words representation of text data.
Some Word weighting schemes may need information about the document collection as a whole before constructing the weightings, and this class provides the facilities for this to be done in a standardized manner.

Author:

Edward Raff

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
WordWeighting()

Method Summary

Modifier and Type	Method and Description
abstract void	applyTo(Vec vec) The implementation may want to pre compute come values based on the vector it is about to be applied to.
abstract void	setWeight(List<? extends Vec> allDocuments, List<Integer> df) Prepares the word weighting to be performed on a data set.

Methods inherited from class jsat.math.IndexFunction

f, f, indexFunc

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WordWeighting

public WordWeighting()

Method Detail

setWeight

public abstract void setWeight(List<? extends Vec> allDocuments,
                               List<Integer> df)

Prepares the word weighting to be performed on a data set. This should be called once before being applied to any vectors. Different WordWeightings may require different amounts of computation to set up.

Parameters:

allDocuments - the list of all vectors that make up the set of documents. The word vectors should be unmodified, containing the value of how many times a word appeared in the document for each index.

df - a list mapping each integer index of a word to how many times that word occurred in total

applyTo

public abstract void applyTo(Vec vec)

The implementation may want to pre compute come values based on the vector it is about to be applied to. This should be called in place of Vec.applyIndexFunction(jsat.math.IndexFunction) . The vector should be in a bag-of-words form where each index value indicates how many times the word for that index occurred in the document represented by the vector.

Parameters:

vec - the vector to set up for and then alter by invoking Vec.applyIndexFunction(jsat.math.IndexFunction) on