jsat.classifiers

Class ClassificationModelEvaluation

java.lang.Object

jsat.classifiers.ClassificationModelEvaluation

public class ClassificationModelEvaluation
extends Object

Provides a mechanism to quickly perform an evaluation of a model on a data set. This can be done with cross validation or with a testing set.

Author:

Edward Raff

Constructor Summary

Constructors

Constructor and Description
ClassificationModelEvaluation(Classifier classifier, ClassificationDataSet dataSet) Constructs a new object that can perform evaluations on the model.
ClassificationModelEvaluation(Classifier classifier, ClassificationDataSet dataSet, ExecutorService threadpool) Constructs a new object that can perform evaluations on the model.

Method Summary

Modifier and Type	Method and Description
void	addScorer(ClassificationScore scorer) Adds a new score object that will be used as part of the evaluation when calling evaluateCrossValidation(int, java.util.Random) or evaluateTestSet(jsat.classifiers.ClassificationDataSet).
boolean	doseStoreResults()
void	evaluateCrossValidation(int folds) Performs an evaluation of the classifier using the training data set.
void	evaluateCrossValidation(int folds, Random rand) Performs an evaluation of the classifier using the training data set.
void	evaluateCrossValidation(List<ClassificationDataSet> lcds) Performs an evaluation of the classifier using the training data set, where the folds of the training data set are provided by the user.
void	evaluateCrossValidation(List<ClassificationDataSet> lcds, List<ClassificationDataSet> trainCombinations) Note: Most people should never need to call this method.
void	evaluateTestSet(ClassificationDataSet testSet) Performs an evaluation of the classifier using the initial data set to train, and testing on the given data set.
Classifier	getClassifier() Returns the classifier that was original given for evaluation.
double[][]	getConfusionMatrix()
double	getCorrectWeights() Returns the total value of the weights for data points that were classified correctly.
double	getErrorRate() Computes the weighted error rate of the classifier.
OnLineStatistics	getErrorRateStats() Returns the object that keeps track of the error on individual evaluations.
Classifier[]	getKeptModels() Returns the models that were kept after the last evaluation.
double[]	getPointWeights() If keepPredictions(boolean) was set, this method will return the array storing the weights for each of the points that were classified
CategoricalResults[]	getPredictions() If keepPredictions(boolean) was set, this method will return the array storing the predictions made by the classifier during evaluation.
OnLineStatistics	getScoreStats(ClassificationScore score) Gets the statistics associated with the given score.
double	getSumOfWeights() Returns the total value of the weights for all data points that were tested against
long	getTotalClassificationTime() Returns the total number of milliseconds spent performing classification on the testing set.
long	getTotalTrainingTime() Returns the total number of milliseconds spent training the classifier.
int[]	getTruths() If keepPredictions(boolean) was set, this method will return the array storing the target classes that should have been predicted during evaluation.
boolean	isKeepModels() This will keep the models trained when evaluating the model.
void	keepPredictions(boolean keepPredictions) Indicates whether or not the predictions made during evaluation should be stored with the expected value for retrieval later.
void	prettyPrintClassificationScores() Prints out the classification information in a convenient format.
void	prettyPrintConfusionMatrix() Assuming that we are on the start of a new line, the confusion matrix will be pretty printed to System.out
void	setDataTransformProcess(DataTransformProcess dtp) Sets the data transform process to use when performing cross validation.
void	setKeepModels(boolean keepModels) Set this to true in order to keep the trained models after evaluation.
void	setWarmModels(Classifier... warmModels) Sets the models that will be used for warm starting training.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ClassificationModelEvaluation

public ClassificationModelEvaluation(Classifier classifier,
                                     ClassificationDataSet dataSet)

Constructs a new object that can perform evaluations on the model. The model will not be trained until evaluation time.

Parameters:

classifier - the model to train and evaluate

dataSet - the training data set.

ClassificationModelEvaluation

public ClassificationModelEvaluation(Classifier classifier,
                                     ClassificationDataSet dataSet,
                                     ExecutorService threadpool)

Constructs a new object that can perform evaluations on the model. The model will not be trained until evaluation time.

Parameters:

classifier - the model to train and evaluate

dataSet - the training data set.

threadpool - the source of threads for parallel training. If set to null, training will be done using the Classifier.trainC(jsat.classifiers.ClassificationDataSet) method.

Method Detail

setKeepModels

public void setKeepModels(boolean keepModels)

Set this to true in order to keep the trained models after evaluation. They can then be retrieved used the getKeptModels() methods. The default value is false.

Parameters:

keepModels - true to keep the trained models after evaluation, false to discard them.

isKeepModels

public boolean isKeepModels()

This will keep the models trained when evaluating the model. The models can be obtained after an evaluation from getKeptModels().

Returns:

true if trained models will be kept after evaluation.

getKeptModels

public Classifier[] getKeptModels()

Returns the models that were kept after the last evaluation. null will be returned instead if isKeepModels() returns false, which is the default.

Returns:

the models that were kept after the last evaluation. Or null if if models are not being kept.

setWarmModels

public void setWarmModels(Classifier... warmModels)

Sets the models that will be used for warm starting training. If using cross-validation, the number of models given should match the number of folds. If using a test set, only one model should be given.

Parameters:

warmModels - the models to use for warm start training

setDataTransformProcess

public void setDataTransformProcess(DataTransformProcess dtp)

Sets the data transform process to use when performing cross validation. By default, no transforms are applied

Parameters:

dtp - the transformation process to clone for use during evaluation

evaluateCrossValidation

public void evaluateCrossValidation(int folds)

Performs an evaluation of the classifier using the training data set. The evaluation is done by performing cross validation.

Parameters:

folds - the number of folds for cross validation

Throws:

UntrainedModelException - if the number of folds given is less than 2

evaluateCrossValidation

public void evaluateCrossValidation(int folds,
                                    Random rand)

Performs an evaluation of the classifier using the training data set. The evaluation is done by performing cross validation.

Parameters:

folds - the number of folds for cross validation

rand - the source of randomness for generating the cross validation sets

Throws:

UntrainedModelException - if the number of folds given is less than 2

evaluateCrossValidation

public void evaluateCrossValidation(List<ClassificationDataSet> lcds)

Performs an evaluation of the classifier using the training data set, where the folds of the training data set are provided by the user. The folds do not need to be the same sizes, though it is assumed that they are all approximately the same size. It is the caller's responsibility to ensure that the folds are only from the original training data set.

This method exists so that the user can provide very specific folds if they so desire. This can be useful when there is known bias in the data set, such as when caused by duplicate data point values. The caller can then manually make sure duplicate values all occur in the same fold to avoid over-estimating the accuracy of the model.

Parameters:

lcds - the training data set already split into folds

evaluateCrossValidation

public void evaluateCrossValidation(List<ClassificationDataSet> lcds,
                                    List<ClassificationDataSet> trainCombinations)

Note: Most people should never need to call this method. Make sure you understand what you are doing before you do.

Performs an evaluation of the classifier using the training data set, where the folds of the training data set, and their combinations, are provided by the user. The folds do not need to be the same sizes, though it is assumed that they are all approximately the same size - and the the training combination corresponding to each index will be the sum of the folds in the other indices. It is the caller's responsibility to ensure that the folds are only from the original training data set.

This method exists so that the user can provide very specific folds if they so desire, and when the same folds will be used multiple times. Doing so allows the algorithms called to take advantage of any potential caching of results based on the data set and avoid all possible excessive memory movement. (For example, DataSet.getNumericColumns() may get re-used and benefit from its caching)
The same behavior of this method can be obtained by calling evaluateCrossValidation(java.util.List).

Parameters:

lcds - training data set already split into folds

trainCombinations - each index contains the training data sans the data stored in the fold associated with that index

evaluateTestSet

public void evaluateTestSet(ClassificationDataSet testSet)

Performs an evaluation of the classifier using the initial data set to train, and testing on the given data set.

Parameters:

testSet - the data set to perform testing on

addScorer

public void addScorer(ClassificationScore scorer)

Adds a new score object that will be used as part of the evaluation when calling evaluateCrossValidation(int, java.util.Random) or evaluateTestSet(jsat.classifiers.ClassificationDataSet). The statistics for the given score are reset on every call, and the mean / standard deviation comes from multiple folds in cross validation.

The score statistics can be obtained from getScoreStats(ClassificationScore) after one of the evaluation methods have been called.

Parameters:

scorer - the score method to keep track of.

getScoreStats

public OnLineStatistics getScoreStats(ClassificationScore score)

Gets the statistics associated with the given score. If the score is not currently in the model evaluation null will be returned. The object passed in does not need to be the exact same object passed to addScorer(ClassificationScore), it only needs to be equal to the object.

Parameters:

score - the score type to get the result statistics

Returns:

the result statistics for the given score, or null if the score is not in th evaluation set

keepPredictions

public void keepPredictions(boolean keepPredictions)

Indicates whether or not the predictions made during evaluation should be stored with the expected value for retrieval later.

Parameters:

keepPredictions - true if space should be allocated to store the predictions made

doseStoreResults

public boolean doseStoreResults()

Returns:

true if the predictions are being stored

getPredictions

public CategoricalResults[] getPredictions()

If keepPredictions(boolean) was set, this method will return the array storing the predictions made by the classifier during evaluation. These results may not be in the same order as the data set they came from, but the order is paired with getTruths()

Returns:

the array of predictions, or null

getTruths

public int[] getTruths()

If keepPredictions(boolean) was set, this method will return the array storing the target classes that should have been predicted during evaluation. These results may not be in the same order as the data set they came from, but the order is paired with getPredictions()

Returns:

the array of target class values, or null

getPointWeights

public double[] getPointWeights()

If keepPredictions(boolean) was set, this method will return the array storing the weights for each of the points that were classified

Returns:

the array of data point weights, or null

getConfusionMatrix

public double[][] getConfusionMatrix()

prettyPrintConfusionMatrix

public void prettyPrintConfusionMatrix()

Assuming that we are on the start of a new line, the confusion matrix will be pretty printed to System.out

prettyPrintClassificationScores

public void prettyPrintClassificationScores()

Prints out the classification information in a convenient format. If no additional scores were added via the addScorer(ClassificationScore) method, nothing will be printed.

getCorrectWeights

public double getCorrectWeights()

Returns the total value of the weights for data points that were classified correctly.

Returns:

the total value of the weights for data points that were classified correctly.

getSumOfWeights

public double getSumOfWeights()

Returns the total value of the weights for all data points that were tested against

Returns:

the total value of the weights for all data points that were tested against

getErrorRate

public double getErrorRate()

Computes the weighted error rate of the classifier. If all weights of the data points tested were equal, then the value returned is also the percent of data points that the classifier erred on.

Returns:

the weighted error rate of the classifier.

getErrorRateStats

public OnLineStatistics getErrorRateStats()

Returns the object that keeps track of the error on individual evaluations. If cross-validation was used, it is the statistics for the errors of each fold. If not, it is for each time evaluateTestSet(jsat.classifiers.ClassificationDataSet) was called.

Returns:

the statistics for the error of all evaluation sets

getTotalTrainingTime

public long getTotalTrainingTime()

Returns the total number of milliseconds spent training the classifier.

Returns:

the total number of milliseconds spent training the classifier.

getTotalClassificationTime

public long getTotalClassificationTime()

Returns the total number of milliseconds spent performing classification on the testing set.

Returns:

the total number of milliseconds spent performing classification on the testing set.

getClassifier

public Classifier getClassifier()

Returns the classifier that was original given for evaluation.

Returns:

the classifier that was original given for evaluation.