deepensemble.metrics

Metrics¶

Base Class Metrics¶

class deepensemble.metrics.basemetrics.BaseMetrics(model)[source]¶

Base class generate metrics.

Parameters:

model : Model

Model.

Attributes

model	(Model or EnsembleModel) Handle of model.
_error	(dict[list[DataPlot]]) Data plot error.
_cost	(dict[list[DataPlot]]) Data plot cost.
_costs	(dict[dict[list[DataPlot]]]) Data plot error.
_scores	(dict[dict[list[DataPlot]]]) Data plot scores.
_y_true	(list[numpy.array]) List of array with target samples.
_y_pred	(list[numpy.array]) List of array with output or diversity of model.

append_data(data, epoch, type_set_data)[source]¶

Append metrics data.

Parameters:

data : list or array

List of data.

epoch : int

Current epoch of training when was called this method.

type_set_data : str

This string means what kind of data is passed: train or test.

Returns:

int

Returns index of last item saved from data list.

append_metric(metric)[source]¶

Adds metric of another metric model.

Parameters:

metric : BaseMetrics

Metric of another model.

append_prediction(_input, _target, append_last_pred=False)[source]¶

Add a sample of diversity and target for generating metrics.

Parameters:

_input : numpy.array

Input sample.

_target : numpy.array

Target sample.

append_last_pred : bool

This flag indicates that the current diversity is append in the last saved diversity.

Returns:

float

Return model score (classifier: accuracy, regressor: MSE).

get_cost(type_set_data)[source]¶

Getter total cost.

Parameters:

type_set_data : str

This string means what kind of data is passed: train or test.

Returns:

list[]

Returns cost list.

get_costs(type_set_data)[source]¶

Getter costs.

Parameters:

type_set_data : str

This string means what kind of data is passed: train or test.

Returns:

dict

Returns costs dictionary.

get_model()[source]¶

Getter model.

Returns:

Model

Returns current model in metric.

get_score_prediction(_target, _prediction)[source]¶

Get score the diversity.

Parameters:

_target : numpy.array

Target sample.

_prediction : numpy.array

Prediction of model.

Returns:

float

Returns diversity od model, in case of classifier model return accuracy and in case of regressor model return mean square error.

get_scores(type_set_data)[source]¶

Getter scores.

Parameters:

type_set_data : str

This string means what kind of data is passed: train or test.

Returns:

dict

Returns scores dictionary.

plot_cost(max_epoch, title='Cost', log_xscale=False, log_yscale=False)[source]¶

Generate cost plot.

Parameters:

max_epoch : int

Number of epoch of training.

title : str

Plot title of training cost.

log_xscale : bool

Flag for show plot x-axis in logarithmic scale.

log_yscale : bool

Flag for show plot y-axis in logarithmic scale.

plot_costs(max_epoch, title='Cost', log_xscale=False, log_yscale=False)[source]¶

Generate costs plot.

Parameters:

max_epoch : int

Number of epoch of training.

title : str

Plot title of cost.

log_xscale : bool

Flag for show plot x-axis in logarithmic scale.

log_yscale : bool

Flag for show plot y-axis in logarithmic scale.

plot_scores(max_epoch, title='Score', log_xscale=False, log_yscale=False)[source]¶

Generate training score plot.

Parameters:

max_epoch : int

Number of epoch of training.

title : str, “Train score” by default

Plot title of training score.

log_xscale : bool

Flag for show plot x-axis in logarithmic scale.

log_yscale : bool

Flag for show plot y-axis in logarithmic scale.

reset()[source]¶: Reset metrics.

class deepensemble.metrics.basemetrics.EnsembleMetrics(model)[source]¶

Class for generate different metrics for ensemble models.

Parameters:

model : EnsembleModel

Ensemble Model.

Attributes

_models_metric	(dict[BaseMetrics]) Dictionary of models metrics.
_y_true_per_model	(list[numpy.array]) Array for saving target of sample.
_y_pred_per_model	(dict[numpy.array]) Dictionary for saving diversity of ensemble models.

append_data(data, epoch, type_set_data)[source]¶

Append metrics data of ensemble.

Parameters:

data : list or array

List of data.

epoch : int

Current epoch of training when was called this method.

type_set_data : str

This string means what kind of data is passed: train or test.

Returns:

int

Returns index of last item saved from data list.

append_metric(metric)[source]¶

Adds metric of another metric model.

Parameters:

metric : EnsembleMetrics or BaseMetrics

Metric of another model.

append_prediction(_input, _target, append_last_pred=False)[source]¶

Append diversity (using for generate reports).

Parameters:

_input : numpy.array

Input sample.

_target : numpy.array

Target sample.

append_last_pred : bool

This flag indicates that the current diversity is append in the last saved diversity.

Returns:

float

Returns the score diversity.

append_prediction_per_model(_input, _target, append_last_pred=False)[source]¶

Append diversity for each model in ensemble.

Parameters:

_input : numpy.array

Input sample.

_target : numpy.array

Target sample.

append_last_pred : bool

This flag indicates that the current diversity is append in the last saved diversity.

Returns:

None

get_models_metric()[source]¶

Gets Ensemble models.

Returns:

list[]

Returns list of Ensemble models.

plot_costs(max_epoch, name=None, title='Cost', log_xscale=False, log_yscale=False)[source]¶

Generate costs plot.

Parameters:

max_epoch : int

Number of epoch of training.

name : str

Name plot.

title : str

Plot title of cost.

log_xscale : bool

Flag for show plot x-axis in logarithmic scale.

log_yscale : bool

Flag for show plot y-axis in logarithmic scale.

class deepensemble.metrics.basemetrics.FactoryMetrics[source]¶

Factory Metrics

static get_metric(_model)[source]¶

Parameters:	_model : Model

Classifier Metrics¶

class deepensemble.metrics.classifiermetrics.ClassifierMetrics(model)[source]¶

Class for generate different metrics for classifier models.

Parameters:

model : Model

Model.

classification_report(name_report=None)[source]¶

Generate a classification report (wrapper classification_report scikit)

Returns:	None

plot_confusion_matrix(**kwargs)[source]¶

Generate Confusion Matrix plot.

Note

Show Confusion Matrix plot.

Parameters:	kwargs

plot_confusion_matrix_prediction(y_true, y_pred, title=None, cmap=<matplotlib.colors.LinearSegmentedColormap object at 0x7f9b52b814e0>)[source]¶

Generate Confusion Matrix plot.

Note

Show Confusion Matrix plot.

Parameters:

y_true : numpy.array

Target sample.

y_pred : numpy.array

Prediction.

title : str

Plot title.

cmap : plt.cm

Plot Color.

static plot_matrix(ax, cm, labels, title, cmap=<matplotlib.colors.LinearSegmentedColormap object at 0x7f9b52b814e0>)[source]¶

Plot Matrix.

Parameters:

ax

Handler plot.

cm : numpy.array

Matrix for plotting, the matrix must be square.

labels : list[str]

List of labels for each row and cols in Matrix.

title : str

Title of plot.

cmap

Colors of plot.

Raises:

ValueError

If the matrix is not square.

class deepensemble.metrics.classifiermetrics.EnsembleClassifierMetrics(model)[source]¶

Class for generate different metrics for ensemble classifier models.

Parameters:

model : EnsembleModel

Ensemble Model.

diversity_report()[source]¶

Generate diversity report of ensemble model.

Returns:	None

plot_diversity(input_test, target_test, prefix=None)[source]¶

Generate a plot for different diversity metrics.

Parameters:

input_test : numpy.array

Input sample.

target_test : numpy.array

Target sample.

prefix : str

This string is add as prefix a each label the figures.

Returns:

list

Returns a list of tuple where each tuple has the figure and a label that identify the figure.

static statistic_metric_non_pairwise(metric, list_target, list_classifiers)[source]¶

Parameters:

metric

Diversity metric.

list_target : list or array

List of target.

list_classifiers : list or array

Predictions of classifiers.

Returns:

tuple

Returns average between two metric models.

static statistic_metric_pairwise(metric, list_target, list_c1, list_c2)[source]¶

Parameters:

metric

Diversity metric.

list_target : list or array

List of target.

list_c1 : list or array

Prediction first model.

list_c2 : list or array

Prediction second model.

Returns:

tuple

Returns average between two metric models.

Regression Metrics¶

class deepensemble.metrics.regressionmetrics.RegressionMetrics(model)[source]¶

Class for generate different metrics for regressor models.

Parameters:

model : Model

Model.

class deepensemble.metrics.regressionmetrics.EnsembleMetrics(model)[source]¶

Class for generate different metrics for ensemble models.

Parameters:

model : EnsembleModel

Ensemble Model.

Attributes

_models_metric	(dict[BaseMetrics]) Dictionary of models metrics.
_y_true_per_model	(list[numpy.array]) Array for saving target of sample.
_y_pred_per_model	(dict[numpy.array]) Dictionary for saving diversity of ensemble models.

append_data(data, epoch, type_set_data)[source]¶

Append metrics data of ensemble.

Parameters:

data : list or array

List of data.

epoch : int

Current epoch of training when was called this method.

type_set_data : str

This string means what kind of data is passed: train or test.

Returns:

int

Returns index of last item saved from data list.

append_metric(metric)[source]¶

Adds metric of another metric model.

Parameters:

metric : EnsembleMetrics or BaseMetrics

Metric of another model.

append_prediction(_input, _target, append_last_pred=False)[source]¶

Append diversity (using for generate reports).

Parameters:

_input : numpy.array

Input sample.

_target : numpy.array

Target sample.

append_last_pred : bool

This flag indicates that the current diversity is append in the last saved diversity.

Returns:

float

Returns the score diversity.

append_prediction_per_model(_input, _target, append_last_pred=False)[source]¶

Append diversity for each model in ensemble.

Parameters:

_input : numpy.array

Input sample.

_target : numpy.array

Target sample.

append_last_pred : bool

This flag indicates that the current diversity is append in the last saved diversity.

Returns:

None

get_models_metric()[source]¶

Gets Ensemble models.

Returns:

list[]

Returns list of Ensemble models.

plot_costs(max_epoch, name=None, title='Cost', log_xscale=False, log_yscale=False)[source]¶

Generate costs plot.

Parameters:

max_epoch : int

Number of epoch of training.

name : str

Name plot.

title : str

Plot title of cost.

log_xscale : bool

Flag for show plot x-axis in logarithmic scale.

log_yscale : bool

Flag for show plot y-axis in logarithmic scale.

Diversity¶

Utils¶

deepensemble.metrics.diversitymetrics.oracle(y, c)[source]¶

Compare between two arrays that represent the target and the output of a classifier.

Note

The input arrays must be same shape and type (numpy.array).

Parameters:

y : numpy.array

Target sample.

c : numpy.array

Output model for comparing with target.

Returns:

numpy.array

return an array with 1 and 0 by each elements of the input arrays.

References

[R99]

Ludmila I. Kuncheva (2004), pp 298: Combining Pattern Classifiers Methods and Algorithms A Wiley-Interscience publication, ISBN 0-471-21078-1 (cloth).

deepensemble.metrics.diversitymetrics.contingency_table(y, c1, c2)[source]¶

Compute de contingency table.

Note

The input arrays must be same shape and type (numpy.array).

Parameters:

y : numpy.array

Target sample.

c1 : numpy.array

Output of the first classifier.

c2 : numpy.array

Output of the second classifier.

Returns:

tuple

Returns a tuple with the elements of contingency table between two classifiers.

References

[R1111]

Zhi-Hua Zhou. (2012), pp 105: Ensemble Methods Foundations and Algorithms Chapman & Hall/CRC Machine Learning & Pattern Recognition Series.

Pairwise metrics¶

deepensemble.metrics.diversitymetrics.correlation_coefficient(y, c1, c2)[source]¶

Correlation between two binary vectors in this case between the output of the classifiers.

Parameters:

y : numpy.array

Target sample.

c1 : numpy.array

Output of the first classifier.

c2 : numpy.array

Output of the second classifier.

Returns:

float

Return the correlation coefficient between the output of the classifiers ‘c1’ and ‘c2’.

References

[R1313]

Zhi-Hua Zhou. (2012), pp 105: Ensemble Methods Foundations and Algorithms Chapman & Hall/CRC Machine Learning & Pattern Recognition Series.

deepensemble.metrics.diversitymetrics.disagreement_measure(y, c1, c2)[source]¶

Measure is equal to the probability that the two classifiers will disagree on their decisions.

Parameters:

y : numpy.array

Target sample.

c1 : numpy.array

Output of the first classifier.

c2 : numpy.array

Output of the second classifier.

Returns:

float

Return the disagreement measure between the classifiers ‘c1’ and ‘c2’, takes value in the range of [0, 1].

References

[R1516]

Skalak, D. (1996): The sources of increased accuracy for two proposed boosting algorithms. In Proc. American Association for Artificial Intelligence, AAAI-96, Integrating Multiple Learned Models Workshop.

[R1616]

Ludmila I. Kuncheva (2004), pp 300: Combining Pattern Classifiers Methods and Algorithms A Wiley-Interscience publication, ISBN 0-471-21078-1 (cloth).

deepensemble.metrics.diversitymetrics.double_fault_measure(y, c1, c2)[source]¶

Measure is equal to the probability that both classifiers being wrong.

Parameters:

y : numpy.array

Target sample.

c1 : numpy.array

Output of the first classifier.

c2 : numpy.array

Output of the second classifier.

Returns:

float

Return the double fault measure between the classifiers ‘c1’ and ‘c2’, takes value in the range of [0, 1].

References

[R1919]

Ludmila I. Kuncheva (2004), pp 301: Combining Pattern Classifiers Methods and Algorithms A Wiley-Interscience publication, ISBN 0-471-21078-1 (cloth).

deepensemble.metrics.diversitymetrics.kappa_statistic(y, c1, c2)[source]¶

Kappa-Statistic.

Parameters:

y : numpy.array

Target sample.

c1 : numpy.array

Output of the first classifier.

c2 : numpy.array

Output of the second classifier.

Returns:

float

Return the Kappa-Statistic (kp) measure between the classifiers ‘c1’ and ‘c2’.

kp = 1 if the two classifiers totally agree.

kp = 0 if the two classifiers agree by chance.

kp < 0 is a rare case where the agreement is even less than what is expected by chance.

References

[R2121]

Zhi-Hua Zhou. (2012), pp 105: Ensemble Methods Foundations and Algorithms Chapman & Hall/CRC Machine Learning & Pattern Recognition Series.

deepensemble.metrics.diversitymetrics.q_statistic(y, c1, c2)[source]¶

Q-Statistic.

Parameters:

y : numpy.array

Target sample.

c1 : numpy.array

Output of the first classifier.

c2 : numpy.array

Output of the second classifier.

Returns:

float

Return the Q-Statistic measure between the classifiers ‘c1’ and ‘c2’. Q-Statistic takes value in the range of [-1, 1]:

is zero if ‘c1’ and ‘c2’ are independent.

is positive if ‘c1’ and ‘c2’ make similar predictions.

is negative if ‘c1’ and ‘c2’ make different predictions.

References

[R2323]

Zhi-Hua Zhou. (2012), pp 105: Ensemble Methods Foundations and Algorithms Chapman & Hall/CRC Machine Learning & Pattern Recognition Series.

Non Pairwise metrics¶

deepensemble.metrics.diversitymetrics.kohavi_wolpert_variance(y, classifiers)[source]¶

Kohavi-Wolpert Variance.

Parameters:

y : numpy.array

Target sample.

classifiers : numpy.array

Output of the classifiers.

Returns:

float

Return the Kohavi-Wolpert Variance.

References

[R2525]

Zhi-Hua Zhou. (2012), pp 107: Ensemble Methods Foundations and Algorithms Chapman & Hall/CRC Machine Learning & Pattern Recognition Series.

deepensemble.metrics.diversitymetrics.interrater_agreement(y, classifiers)[source]¶

Interrater agreement.

Parameters:

y : numpy.array

Target sample.

classifiers : numpy.array

Output of the classifiers.

Returns:

float

Return the Interrater agreement.

References

[R2727]

Zhi-Hua Zhou. (2012), pp 107: Ensemble Methods Foundations and Algorithms Chapman & Hall/CRC Machine Learning & Pattern Recognition Series.

deepensemble.metrics.diversitymetrics.entropy_cc(y, classifiers)[source]¶

Entropy Cuningham and Carney [2000].

Parameters:

y : numpy.array

Target sample.

classifiers : numpy.array

Output of the classifiers.

Returns:

float

Return the Entropy Cuningham and Carney.

References

[R2929]

Zhi-Hua Zhou. (2012), pp 107: Ensemble Methods Foundations and Algorithms Chapman & Hall/CRC Machine Learning & Pattern Recognition Series.

deepensemble.metrics.diversitymetrics.entropy_sk(y, classifiers)[source]¶

Entropy Shipp and Kuncheva [2002].

Parameters:

y : numpy.array

Target sample.

classifiers : numpy.array

Output of the classifiers.

Returns:

float

Return the Entropy Shipp and Kuncheva.

References

[R3131]

Kuncheva, Ludmila I. (2004), pp 301: Combining pattern classifiers: methods and algorithms. John Wiley & Sons, 2004.

deepensemble.metrics.diversitymetrics.coincident_failure(y, classifiers)[source]¶

Coincident Failure (Partridge and Krzanowski, 1997).

Parameters:

y : numpy.array

Target sample.

classifiers : numpy.array

Output of the classifiers.

Returns:

float

Return the Coincident Failure.

References

[R3333]

Zhi-Hua Zhou. (2012), pp 109: Ensemble Methods Foundations and Algorithms Chapman & Hall/CRC Machine Learning & Pattern Recognition Series.

deepensemble.metrics.diversitymetrics.difficulty(y, classifiers)[source]¶

Difficulty.

Parameters:

y : numpy.array

Target sample.

classifiers : numpy.array

Output of the classifiers.

Returns:

float

Return the Generalized Diversity.

References

[R3535]

Zhi-Hua Zhou. (2012), pp 108: Ensemble Methods Foundations and Algorithms Chapman & Hall/CRC Machine Learning & Pattern Recognition Series.

deepensemble.metrics.diversitymetrics.generalized_diversity(y, classifiers)[source]¶

Generalized Diversity .

Parameters:

y : numpy.array

Target sample.

classifiers : numpy.array

Output of the classifiers.

Returns:

float

Return the Generalized Diversity.

References

[R3737]

Zhi-Hua Zhou. (2012), pp 108: Ensemble Methods Foundations and Algorithms Chapman & Hall/CRC Machine Learning & Pattern Recognition Series.