jsat.text.tokenizer

Class StemmingTokenizer

java.lang.Object

jsat.text.tokenizer.StemmingTokenizer

All Implemented Interfaces:

Serializable, Tokenizer

public class StemmingTokenizer
extends Object
implements Tokenizer

Author:

Edward Raff

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
StemmingTokenizer(Stemmer stemmer, Tokenizer baseTokenizer)

Method Summary

Modifier and Type	Method and Description
List<String>	tokenize(String input) Breaks the input string into a series of tokens that may be used as features for a classifier.
void	tokenize(String input, StringBuilder workSpace, List<String> storageSpace) Breaks the input string into a series of tokens that may be used as features for a classifier.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StemmingTokenizer

public StemmingTokenizer(Stemmer stemmer,
                         Tokenizer baseTokenizer)

Method Detail

tokenize

public List<String> tokenize(String input)

Description copied from interface: Tokenizer

Breaks the input string into a series of tokens that may be used as features for a classifier. The returned tokens must be either new string objects or interned strings. If a token is returned that is backed by the original document, memory may get leaked by processes consuming the token.
This method should be thread safe

Specified by:

tokenize in interface Tokenizer

Parameters:

input - the string to tokenize

Returns:

an already allocated list to place the tokens into

tokenize

public void tokenize(String input,
                     StringBuilder workSpace,
                     List<String> storageSpace)

Description copied from interface: Tokenizer

Breaks the input string into a series of tokens that may be used as features for a classifier. The returned tokens must be either new string objects or interned strings. If a token is returned that is backed by the original document, memory may get leaked by processes consuming the token.
This method should be thread safe

Specified by:

tokenize in interface Tokenizer

Parameters:

input - the string to tokenize

workSpace - an already allocated (but empty) string builder than can be used as a temporary work space.

storageSpace - an already allocated (but empty) list to place the tokens into