jsat.text.tokenizer

Class NGramTokenizer

java.lang.Object

jsat.text.tokenizer.NGramTokenizer

All Implemented Interfaces:

Serializable, Tokenizer

public class NGramTokenizer
extends Object
implements Tokenizer

This tokenizer creates n-grams, which are sequences of tokens combined into their own larger token. For example, "the dog barked" could be a 3-gram. If all sub n-grams are also being generated, the returned set would contain the 1-grams "the", "dog", and "barked", the 2-grams "the dog" and "dog barked", and the aforementioned 3-gram. For this to work, this tokenizer assumes the base tokenizer returns tokens in the order they were seen.
Note that n-grams can significantly increase the number of unique tokens, and n-grams are inherently rarer than the 1-grams they are generated from.

Author:

Edward Raff

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
NGramTokenizer(int n, Tokenizer base, boolean allSubN) Creates a new n-gramer

Method Summary

Modifier and Type	Method and Description
List<String>	tokenize(String input) Breaks the input string into a series of tokens that may be used as features for a classifier.
void	tokenize(String input, StringBuilder workSpace, List<String> storageSpace) Breaks the input string into a series of tokens that may be used as features for a classifier.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NGramTokenizer

public NGramTokenizer(int n,
                      Tokenizer base,
                      boolean allSubN)

Creates a new n-gramer

Parameters:

n - the length of the ngrams. While it should be greater than 1, 1 is still a valid input.

base - the base tokenizer to create n-grams from

allSubN - true to generate all sub n-grams, false to only return the n-grams specified

Method Detail

tokenize

public List<String> tokenize(String input)

Description copied from interface: Tokenizer

Breaks the input string into a series of tokens that may be used as features for a classifier. The returned tokens must be either new string objects or interned strings. If a token is returned that is backed by the original document, memory may get leaked by processes consuming the token.
This method should be thread safe

Specified by:

tokenize in interface Tokenizer

Parameters:

input - the string to tokenize

Returns:

an already allocated list to place the tokens into

tokenize

public void tokenize(String input,
                     StringBuilder workSpace,
                     List<String> storageSpace)

Description copied from interface: Tokenizer

Breaks the input string into a series of tokens that may be used as features for a classifier. The returned tokens must be either new string objects or interned strings. If a token is returned that is backed by the original document, memory may get leaked by processes consuming the token.
This method should be thread safe

Specified by:

tokenize in interface Tokenizer

Parameters:

input - the string to tokenize

workSpace - an already allocated (but empty) string builder than can be used as a temporary work space.

storageSpace - an already allocated (but empty) list to place the tokens into