jsat.text

Class BasicTextVectorCreator

java.lang.Object

jsat.text.BasicTextVectorCreator

All Implemented Interfaces:

Serializable, TextVectorCreator

public class BasicTextVectorCreator
extends Object
implements TextVectorCreator

Creates new text vectors from a dictionary of known tokens and a word weighting scheme.

This object is generally intended to be constructed by a TextDataLoader, though can be used if you know all the words you will need (and can initialize the WordWeighting) before creating this object.

Author:

Edward Raff

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
BasicTextVectorCreator(Tokenizer tokenizer, Map<String,Integer> wordIndex, WordWeighting weighting) Creates a new basic text vector creator

Method Summary

Modifier and Type	Method and Description
Vec	newText(String text) Converts the given input text into a vector representation.
Vec	newText(String input, StringBuilder workSpace, List<String> storageSpace) Converts the given input text into a vector representation

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BasicTextVectorCreator

public BasicTextVectorCreator(Tokenizer tokenizer,
                              Map<String,Integer> wordIndex,
                              WordWeighting weighting)

Creates a new basic text vector creator

Parameters:

tokenizer - the tokenizer to apply to incoming strings

wordIndex - the map of each known word to its index, the size of the map indicating the maximum (exclusive) index

weighting - the weighting process to apply to each loaded document. This should have already been initialized, or be stateless.

Method Detail

newText

public Vec newText(String text)

Description copied from interface: TextVectorCreator

Converts the given input text into a vector representation.

Specified by:

newText in interface TextVectorCreator

Parameters:

text - the input string

Returns:

a vector representation

newText

public Vec newText(String input,
                   StringBuilder workSpace,
                   List<String> storageSpace)

Description copied from interface: TextVectorCreator

Converts the given input text into a vector representation

Specified by:

newText in interface TextVectorCreator

Parameters:

input - the input string

workSpace - an already allocated (but empty) string builder than can be used as a temporary work space.

storageSpace - an already allocated (but empty) list to place the tokens into

Returns:

a vector representation