jsat.text.wordweighting

Class TfIdf

java.lang.Object

jsat.math.IndexFunction

jsat.text.wordweighting.WordWeighting

jsat.text.wordweighting.TfIdf

All Implemented Interfaces:

Serializable, Function

public class TfIdf
extends WordWeighting

Applies Term Frequency Inverse Document Frequency (TF IDF) weighting to the word vectors.

Author:

Edward Raff

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TfIdf.TermFrequencyWeight

Constructor Summary

Constructors

Constructor and Description
TfIdf() Creates a new TF-IDF document weighting scheme that uses LOG weighting for term frequency.
TfIdf(TfIdf.TermFrequencyWeight tfWeighting) Creates a new TF-IDF document weighting scheme that uses the specified term frequency weighting

Method Summary

Modifier and Type	Method and Description
void	applyTo(Vec vec) The implementation may want to pre compute come values based on the vector it is about to be applied to.
double	indexFunc(double value, int index) An index function, meant to be applied to vectors where the value to be computed may vary based on the position in the vector of the value.
void	setWeight(List<? extends Vec> allDocuments, List<Integer> df) Prepares the word weighting to be performed on a data set.

Methods inherited from class jsat.math.IndexFunction

f, f

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TfIdf

public TfIdf()

Creates a new TF-IDF document weighting scheme that uses LOG weighting for term frequency.

TfIdf

public TfIdf(TfIdf.TermFrequencyWeight tfWeighting)

Creates a new TF-IDF document weighting scheme that uses the specified term frequency weighting

Parameters:

tfWeighting - the weighting method to use for the term frequency (tf) component

Method Detail

setWeight

public void setWeight(List<? extends Vec> allDocuments,
                      List<Integer> df)

Description copied from class: WordWeighting

Prepares the word weighting to be performed on a data set. This should be called once before being applied to any vectors. Different WordWeightings may require different amounts of computation to set up.

Specified by:

setWeight in class WordWeighting

Parameters:

allDocuments - the list of all vectors that make up the set of documents. The word vectors should be unmodified, containing the value of how many times a word appeared in the document for each index.

df - a list mapping each integer index of a word to how many times that word occurred in total

indexFunc

public double indexFunc(double value,
                        int index)

Description copied from class: IndexFunction

An index function, meant to be applied to vectors where the value to be computed may vary based on the position in the vector of the value.

Specified by:

indexFunc in class IndexFunction

Parameters:

value - the value at the specified index

index - the index the value is from

Returns:

the computed result. If a negative index was given, the function should return 0.0 if indexFunc(0,indx) would return zero for all valid indices. If this is not the case, any non zero value should be returned.

applyTo

public void applyTo(Vec vec)

Description copied from class: WordWeighting

The implementation may want to pre compute come values based on the vector it is about to be applied to. This should be called in place of Vec.applyIndexFunction(jsat.math.IndexFunction) . The vector should be in a bag-of-words form where each index value indicates how many times the word for that index occurred in the document represented by the vector.

Specified by:

applyTo in class WordWeighting

Parameters:

vec - the vector to set up for and then alter by invoking Vec.applyIndexFunction(jsat.math.IndexFunction) on