deepensemble.utils

Utils Functions

Cost Functions

deepensemble.utils.cost_functions.mse(model, _input, _target)

Compute MSE error between target and output diversity.

Parameters:

model : Model

Model for generating output for compare with target sample.

_input : theano.tensor.matrix

Input sample.

_target : theano.tensor.matrix

Target sample.

Returns:

theano.tensor.matrix

Return MSE error.

deepensemble.utils.cost_functions.mcc(model, _input, _target, s=None, kernel=<function ITLFunctions.kernel_gauss at 0x7f9b55a3a7b8>)

Compute the MCC.

Parameters:

model : Model

Model for generating output for compare with target sample.

_input : theano.tensor.matrix

Input sample.

_target : theano.tensor.matrix

Target sample.

s : float

Size of Kernel.

kernel : callable

Kernel for compute MCC.

Returns:

theano.tensor.matrix

Return MCC.

deepensemble.utils.cost_functions.mee(model, _input, _target, s=None, kernel=<function ITLFunctions.kernel_gauss at 0x7f9b55a3a7b8>)

Compute the MEE.

Parameters:

model : Model

Model for generating output for compare with target sample.

_input : theano.tensor.matrix

Input sample.

_target : theano.tensor.matrix

Target sample.

s : float

Size of Kernel.

kernel : callable

Kernel for compute MEE.

Returns:

theano.tensor.matrix

Return MEE.

deepensemble.utils.cost_functions.cross_entropy(model, _input, _target)

Compute Cross Entropy between target and output diversity.

Parameters:

model : Model

Model for generating output for compare with target sample.

_input : theano.tensor.matrix

Input sample.

_target : theano.tensor.matrix

Target sample.

Returns:

theano.tensor.matrix

Return Cross Entropy.

deepensemble.utils.cost_functions.neg_log_likelihood(model, _input, _target)

Compute the negative means of errors between target and output diversity

Parameters:

model : Model

Model for generating output for compare with target sample.

_input : theano.tensor.matrix

Input sample.

_target : theano.tensor.matrix

Target sample.

Returns:

theano.tensor.matrix

Return negative logarithm likelihood.

deepensemble.utils.cost_functions.kullback_leibler(model, _input, _target)

Kullback Leilbler divergence.

Parameters:

model : Model

Model for generating output for compare with target sample.

_input : theano.tensor.matrix

Input sample.

_target : theano.tensor.matrix

Target sample.

Returns:

theano.tensor.matrix

Return Kullback Leilbler divergence.

deepensemble.utils.cost_functions.kullback_leibler_generalized(model, _input, _target)

Kullback Leilbler generalized divergence.

Parameters:

model : Model

Model for generating output for compare with target sample.

_input : theano.tensor.matrix

Input sample.

_target : theano.tensor.matrix

Target sample.

Returns:

theano.tensor.matrix

Return Kullback Leilbler generalized divergence.

deepensemble.utils.cost_functions.itakura_saito(model, _input, _target)

Itakura Saito distance.

Parameters:

model : Model

Model for generating output for compare with target sample.

_input : theano.tensor.matrix

Input sample.

_target : theano.tensor.matrix

Target sample.

Returns:

theano.tensor.matrix

Return Itakura Saito divergence.

deepensemble.utils.cost_functions.dummy_cost(model, _input, _target)

Dummy cost function, this function only return zeros for each elements in _target.

Parameters:

model : Model

Model.

_input : theano.tensor.matrix

Input Sample

_target : theano.tensor.matrix

Target Sample.

Returns:

theano.tensor.matrix

Returns only zeros for each elements in _target.

Cost Function for Ensemble Models

deepensemble.utils.cost_functions.neg_corr(model, _input, _target, ensemble, lamb=0.5)

Compute the Negative Correlation in Ensemble.

Parameters:

model : theano.tensor.matrix

Current model that one would want to calculate the cost.

_input : theano.tensor.matrix

Input sample.

_target : theano.tensor.matrix

Target sample.

ensemble : EnsembleModel

Ensemble.

lamb : float, 0.5 by default

Ratio negative correlation.

Returns:

theano.tensor.matrix

Return Negative Correlation.

Regularizer Functions

deepensemble.utils.regularizer_functions.L1(model, lamb, batch_reg_ratio)

Compute regularization L1.

Parameters:

model : Model

Model for generating output for compare with target sample.

batch_reg_ratio : float

Ratio batch.

lamb : float

Ratio regularization L1.

Returns:

float

Return regularization L1.

deepensemble.utils.regularizer_functions.L2(model, lamb, batch_reg_ratio)

Compute regularization square L2.

Parameters:

model : Model

Model for generating output for compare with target sample.

batch_reg_ratio : float

Ratio batch.

lamb : float

Ratio regularization L2.

Returns:

float

Return regularization square L2.

Update Functions

deepensemble.utils.update_functions.sgd(cost_function, params, learning_rate=0.1)

Stochastic Gradient Descent (SGD).

Parameters:

cost_function : theano.function

Function cost.

params : theano.share

List of params model.

learning_rate : float, 0.1 by default

The learning rate controlling the size of update steps.

Returns:

OrderedDict

A dictionary mapping each parameter to its update expression.

deepensemble.utils.update_functions.sgd_momentum(cost_function, params, learning_rate=0.1, momentum_rate=0.9)

Stochastic Gradient Descent (SGD) updates with momentum.

Parameters:

cost_function : theano.function

Function cost.

params : theano.share

List of params model.

learning_rate : float, 0.1 by default

The learning rate controlling the size of update steps.

momentum_rate : float, 0.9 by default

The Momentum rate smoothing over more update steps.

Returns:

OrderedDict

A dictionary mapping each parameter to its update expression.

deepensemble.utils.update_functions.adadelta(cost_function, params, learning_rate=0.1, rho=0.95, fudge_factor=1e-06)

Adadelta updates

Parameters:

cost_function : theano.function

Function cost.

params : theano.share

List of params model.

learning_rate : float, 0.1 by default

The learning rate controlling the size of update steps.

rho : float, 0.95 by default

Squared gradient moving average decay factor.

fudge_factor : float, 1e-6 by default

Small value added for numerical stability.

Returns:

OrderedDict

A dictionary mapping each parameter to its update expression.

References

[R3939]

Zeiler, M. D. (2012): ADADELTA: An Adaptive Learning Rate Method. arXiv Preprint arXiv:1212.5701.

deepensemble.utils.update_functions.adagrad(cost_function, params, learning_rate=0.1, epsilon=1e-06)

Adagrad updates.

Parameters:

cost_function : theano.function

Function cost.

params : theano.share

List of params model.

learning_rate : float, 0.1 by default

The learning rate controlling the size of update steps.

epsilon : float, 1e-6 by default

Small value added for numerical stability.

Returns:

OrderedDict

A dictionary mapping each parameter to its update expression.

References

[R4142]

Duchi, J., Hazan, E., & Singer, Y. (2011): Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121-2159.

[R4242]

Chris Dyer: Notes on AdaGrad. http://www.ark.cs.cmu.edu/cdyer/adagrad.pdf

deepensemble.utils.update_functions.dummy_update(cost_function, params)

Dummy update function.

Parameters:

cost_function : callable

Cost function.

params : dict

Parameters of cost function.

Returns:

OrderedDict

Returns an empty dictionary.

Score Functions

deepensemble.utils.score_functions.score_accuracy(_input, _output, _target, model)

Accuracy score in a classifier models.

Parameters:

_input : theano.tensor.matrix

Input sample.

_output : theano.tensor.matrix

Output sample.

_target : theano.tensor.matrix

Target sample.

model : Model

Model.

Returns:

theano.tensor.matrix

Returns accuracy in a classifier models.

deepensemble.utils.score_functions.score_ensemble_ambiguity(_input, _output, _target, model)

Score ambiguity for Ensemble.

Parameters:

_input : theano.tensor.matrix

Input sample.

_output : theano.tensor.matrix

Output sample.

_target : theano.tensor.matrix

Target sample.

model : Model

Model.

Returns:

float

Returns a score ambiguity.

deepensemble.utils.score_functions.score_rms(_input, _output, _target, model)

Gets Root Mean Square like score in a regressor model.

Parameters:

_input : theano.tensor.matrix

Input sample.

_output : theano.tensor.matrix

Output sample.

_target : theano.tensor.matrix

Target sample.

model : Model

Model.

Returns:

theano.tensor.matrix

Returns Root Mean Square.

deepensemble.utils.score_functions.score_silverman(_input, _output, _target, model)

Score Silverman.

Parameters:

_input : theano.tensor.matrix

Input sample.

_output : theano.tensor.matrix

Output sample.

_target : theano.tensor.matrix

Target sample.

model : Model

Model.

Returns:

float

Returns size kernel with Silverman Rule.

deepensemble.utils.score_functions.dummy_score(_input, _output, _target, model)

Dummy score function, this function only return zeros for each elements in _target.

Parameters:

_input : theano.tensor.matrix

Input Sample.

_output : theano.tensor.matrix

Output model.

_target : theano.tensor.matrix

Target Sample.

model : Model

Model.

Returns:

theano.tensor.matrix

Returns only zeros for each elements in _target.

Logger

class deepensemble.utils.logger.Logger

Class for controlling log and console messages.

Attributes

log_activate	(bool, True by default) Flag for print or not text in console.
tic	(list[]) Save init value timer.
buffer	(str) Buffer for save all print text.

get_buffer()

Gets buffer.

Returns:

str

Returns the buffer.

log(message='', **kwargs)

Print message in console, also the message is saved in the buffer.

Parameters:

message : str

String to show in console.

kwargs

log_disable()

Disable print on console.

log_enable()

Enable print on console.

progressbar(it, prefix='', postfix='', end='', size=20)

Show a progressbar (it is necessary called for increment counter).

Parameters:

it : iterator or list[]

Range of values that progressbar uses.

prefix : str

This string is displayed before of progressbar.

postfix : str

This string is displayed after of progressbar.

end : str

This string will be displayed when the count of progressbar ends.

size : int

Size of progressbar.

progressbar_training(max_epoch, model)

Show a progressbar (it is necessary called for increment counter).

Parameters:

max_epoch : int

Max epoch or count of progress bar.

model : Model

Model used of training.

Returns:

iterator

Returns a iterator each time is called.

progressbar_training2(it, model)

Show a progressbar (it is necessary called for increment counter).

Parameters:

it

Iterator for progressbar.

model : Model

Model used of training.

Returns:

iterator

Returns a iterator each time is called.

push_buffer(message, end='\n', **kwargs)

Push message in buffer.

Parameters:

message : str

Message.

end : str

This string is concatenate in the end of message.

kwargs

reset()

Reset intern parameters.

save_buffer(filename)

Save the buffer in a file.

Parameters:

filename : str

Name of file.

start_measure_time(message='', **kwargs)

Start timer, is possible show a message.

Parameters:

message : str

String to show in console.

kwargs

stop_measure_time(message='', **kwargs)

Stop timer, also show a message with the time elapsed since it called the start_measure_time function.

Parameters:

message : str

Message that it want to print in console.

kwargs

write(message='', write_buf=False, **kwargs)

Write message in console and the buffer.

Parameters:

message : str

Message.

write_buf : bool

Flag for indicate if the message is copied in buffer.

kwargs

Utils Classifiers

deepensemble.utils.utils_classifiers.get_index_label_classes(_output, is_binary_classification=False)

Gets index labels.

Parameters:

_output : theano.function

Output of classifier model.

is_binary_classification : bool

This flag means that model is for binary classification.

Returns:

numpy.array[int]

Returns index of labels from output model.

deepensemble.utils.utils_classifiers.translate_binary_target(_target, target_labels)

Gets a vector with binary classes: +1 or -1.

Parameters:

_target : numpy.array

Target sample.

target_labels : list

Target labels.

Returns:

numpy.array

Returns the ‘_target’ translated according to target labels, also each class is encoding as: +1 and -1.

deepensemble.utils.utils_classifiers.translate_output(_output, n_classes, is_binary_classification=False)

Gets matrix with one hot encoding where the 1 represent index of class.

Parameters:

_output : theano.tensor.matrix

Output sample.

n_classes : int

Number of classes (or size of one hot encoding rows)

is_binary_classification : bool

This flag means that model is for binary classification.

Returns:

theano.tensor.matrix

Returns one hot encoding.

deepensemble.utils.utils_classifiers.translate_target(_target, target_labels)

For each example you get a vector indicating the “index” from the vector labels class, this vector has all its elements in zero except the element of the position equals to “index” that it is 1.

Parameters:

_target : numpy.array

Target sample.

target_labels : list[]

Target labels.

Returns:

numpy.array

Returns the ‘_target’ translated according to target labels.

Utils Data Bases

deepensemble.utils.utils_data_bases.load_data(db_name, classes_labels=None, normalize=True, data_home='data')

Load data from mldata.org.

Parameters:

db_name : str

Name of data base.

classes_labels : list[]

Labels of classes.

normalize : bool

Flag for indicate if it necessary to normalize the data.

data_home : str

String with path of data bases directory.

Returns:

tuple

Returns a tuple with data as follow: (input data, target data, labels classes, name data base, description data base, list with feature names)

deepensemble.utils.utils_data_bases.load_data_iris(normalize=True)

Load data Iris.

Returns:

tuple

Returns a tuple with data as follow: (input data, target data, labels classes, name data base)

Utils Functions

class deepensemble.utils.utils_functions.ActivationFunctions

Static class with common useful activation functions.

static elu(x)

Exponential Linear Unit function.

\[\begin{split}\varphi(x) = (x > 0) ? x : e^x - 1\end{split}\]

Parameters:

x : float

Input sample.

Returns:

float

Returns the output of the elu function.

static linear(x)

Linear function.

\[\varphi(x) = x\]

Parameters:

x : float

Input sample.

Returns:

float

The same value passed as input.

static relu(x, alpha=0)

Relu function

\[\varphi(x) = \max(alpha * x, x)\]

Parameters:

x : float

Input sample.

alpha : float

The scale parameter.

Returns:

float

Returns the output of the relu function.

static sigmoid(x)

Sigmoid function.

\[\varphi(x) = \frac{1}{1 + e^{-x}}\]

Parameters:

x : float

Input sample.

Returns:

float

Returns the output of the sigmoid function.

static softmax(x, alpha=1.0)

Softmax function.

\[\varphi(\mathbf{x'=\alpha \cdot x})_j = \frac{e^{\mathbf{x'}_j}}{\sum_{k=1}^K e^{\mathbf{x'}_k}}\]

Parameters:

x : float

Input sample.

alpha : float

The scale parameter.

Returns:

function

Returns the output of the softmax function.

static softplus(x)

Softplus function

\[\varphi(x) = \log(1 + e^x)\]

Parameters:

x : float

Input sample.

Returns:

float

Returns the output of the softplus function.

static tanh(x, alpha=1.0, beta=1.0)

Tangent Hyperbolic function

\[\varphi(x) = \tanh(alpha * x) * beta\]

Parameters:

x : float

Input sample.

alpha : float

The scale parameter.

beta : float

The scale parameter.

Returns:

float

Returns the output of the tanh function.

class deepensemble.utils.utils_functions.DiversityFunctions

Static class with useful diversity functions (Ensembles diversity).

static ambiguity(_input, model, ensemble)

Ambiguity of a model and its Ensemble.

Parameters:

_input : numpy.array or theano.matrix

Input sample.

model : Model

Model.

ensemble : EnsembleModel

Ensemble.

Returns:

theano.tensor

Returns ambiguity.

static bias(_input, ensemble, _target)

Bias among outputs of models in Ensemble.

Parameters:

_input : numpy.array or theano.matrix

Input sample.

ensemble : EnsembleModel

Ensemble.

_target : numpy.array or theano.matrix

Target sample.

Returns:

theano.tensor

Returns bias.

static mean_ambiguity(_input, model, ensemble)

Mean ambiguity of a model and its Ensemble.

Parameters:

_input : numpy.array or theano.matrix

Input sample.

model : Model

Model.

ensemble : EnsembleModel

Ensemble.

Returns:

theano.tensor

Returns mean ambiguity.

static variance(_input, ensemble)

Variance among outputs of models in Ensemble.

Parameters:

_input : numpy.array or theano.matrix

Input sample.

ensemble : EnsembleModel

Ensemble.

Returns:

theano.tensor

Returns variance.

class deepensemble.utils.utils_functions.ITLFunctions

Static class with useful ITL (Information Theoretic Learning) functions.

static cross_entropy(px, py)

Cross entropy.

Parameters:

px : float

Probability of random variable X.

py : float

Probability of random variable Y.

Returns:

float

Returns cross entropy.

static entropy(px)

Entropy.

Parameters:

px : float

Probability of random variable X.

Returns:

float

Returns entropy.

static error_rate(y, t)

Error Rates for classification models.

Assume that vector y and t have the same shape and length.

Parameters:

y : numpy.array or theano.matrix

Array with Predictions.

t : numpy.array or theano.matrix

Array with Targets.

Returns:

theano.tensor.scalar

Returns a float with error rates.

static get_diff(X)

Compute difference among each element in each set.

Parameters:

X : list

List of sets.

Returns:

list

Returns a list with the differences of each set.

static get_prod(x, y)

Compute difference among each element of x and y sets.

Parameters:

x

Set 1.

y

Set 2.

Returns:

list

Returns a list with the differences of each set.

static information_potential(x, kernel, s)

Information Potential.

Parameters:

x : theano.tensor.matrix

Input data.

kernel : callable

Kernel function.

s : float

Size of kernel.

Returns:

theano.tensor.matrix

Returns Information Potential.

static kernel_gauss(x, y, s)

Gaussian Kernel.

Parameters:

x : theano.tensor.matrix

Input data.

y : theano.tensor.matrix

Input data.

s : float

Deviation standard.

Returns:

theano.tensor.matrix

Returns Gaussian Kernel.

static kernel_gauss_numpy(x, y, s)

Gaussian Kernel.

Parameters:

x : theano.tensor.matrix

Input data.

y : theano.tensor.matrix

Input data.

——-

s : float

Deviation standard.

Returns:

theano.tensor.matrix

Returns Gaussian Kernel.

static kernel_laplace(x, y, s)

Gaussian Kernel.

Parameters:

x : theano.tensor.matrix

Input data.

y : theano.tensor.matrix

Input data.

s : float

Deviation standard.

Returns:

theano.tensor.matrix

Returns Gaussian Kernel.

static mutual_information(px1, px2, px1x2, eps=1e-06)

Mutual Information.

Parameters:

px1 : theano.tensor.matrix

Probability of random variable X1.

px2 : theano.tensor.matrix

Probability of random variable X2.

px1x2 : theano.tensor.matrix

Joint Probability between X1 and X2.

eps : float

Constant for avoiding numerical errors.

Returns:

float

Returns Mutual Information.

static mutual_information_parzen(x, y, s)

Mutual Information estimate with Parzen Windows.

Parameters:

x : theano.tensor.matrix

The first input data.

y : theano.tensor.matrix

The second input data.

s : float

Deviation standard.

Returns:

theano.tensor.scalar

Returns mutual information

static norm(x, s)

Normal.

Parameters:

x : theano.tensor.matrix

Input data.

s : float

Deviation standard.

Returns:

theano.tensor.matrix

Returns normal.

static silverman(x)

Silverman

Parameters:

x : theano.tensor.matrix

Input data.

Returns:

theano.tensor.scalar

Returns a size kernel computed with Silverman Rule.

Utils Models

Utils Testing

deepensemble.utils.utils_test.plot_hist_train_test(train_means, test_means, ylabel, title, labels)

Parameters:

train_means

test_means

ylabel

title

labels

deepensemble.utils.utils_test.plot_hist_train_test2(train_means, train_std, test_means, test_std, ylabel, title, labels)

Parameters:

train_means

train_std

test_means

test_std

ylabel

title

labels

deepensemble.utils.utils_test.plot_scores_classifications(models, input_train, target_train, input_test, target_test, classes_labels)

Parameters:

models

input_train

target_train

input_test

target_test

classes_labels

deepensemble.utils.utils_test.make_dirs(_dir)

Makes directories.

Parameters:

_dir : str

Path of directory.

Serializable

class deepensemble.utils.serializable.Serializable(data=None)

Class for save an load data in a object.

load(filename)

Load model from file.

Parameters:

filename : str

Path of file where recovery data of model.

save(filename)

Save data to file.

Parameters:

filename : str

Path of file where storage data of model.