jsat.text.tokenizer

Interface Tokenizer

All Superinterfaces:

Serializable

All Known Implementing Classes:

NaiveTokenizer, NGramTokenizer, StemmingTokenizer, StopWordTokenizer

public interface Tokenizer
extends Serializable

Interface for taking the text of a document and breaking it up into features. For example "This doc" might become "this" and "doc"

Author:

Edward Raff

Method Summary

Modifier and Type	Method and Description
List<String>	tokenize(String input) Breaks the input string into a series of tokens that may be used as features for a classifier.
void	tokenize(String input, StringBuilder workSpace, List<String> storageSpace) Breaks the input string into a series of tokens that may be used as features for a classifier.

Method Detail

tokenize

List<String> tokenize(String input)

Breaks the input string into a series of tokens that may be used as features for a classifier. The returned tokens must be either new string objects or interned strings. If a token is returned that is backed by the original document, memory may get leaked by processes consuming the token.
This method should be thread safe

Parameters:

input - the string to tokenize

Returns:

an already allocated list to place the tokens into

tokenize

void tokenize(String input,
              StringBuilder workSpace,
              List<String> storageSpace)

Breaks the input string into a series of tokens that may be used as features for a classifier. The returned tokens must be either new string objects or interned strings. If a token is returned that is backed by the original document, memory may get leaked by processes consuming the token.
This method should be thread safe

Parameters:

input - the string to tokenize

workSpace - an already allocated (but empty) string builder than can be used as a temporary work space.

storageSpace - an already allocated (but empty) list to place the tokens into