* Fri Jul 26 2019 Todd R <email@example.com>
- Update to Version 0.21.2
* Fix: Fixed a bug in cross_decomposition.CCA improving numerical
stability when Y is close to zero..
* Fix: Fixed a bug in metrics.euclidean_distances where a part of the
distance matrix was left un-instanciated for suffiently large float32
datasets (regression introduced in 0.21)..
* Fix: Fixed a bug in preprocessing.OneHotEncoder where the new
drop parameter was not reflected in get_feature_names..
* Fix: Fixed a bug where min_max_axis would fail on 32-bit systems
for certain large inputs. This affects preprocessing.MaxAbsScaler,
preprocessing.normalize and preprocessing.LabelBinarizer..
- Update to Version 0.21.1
* Fix: Fixed a bug in metrics.pairwise_distances where it would raise
AttributeError for boolean metrics when X had a boolean dtype and
Y == None..
* Fix: Fixed two bugs in metrics.pairwise_distances when
n_jobs > 1. First it used to return a distance matrix with same dtype as
input, even for integer dtype. Then the diagonal was not zeros for euclidean
metric when Y is X..
* Fix: Fixed a bug in neighbors.KernelDensity which could not be
restored from a pickle if sample_weight had been used..
- Update to Version 0.21.0
+ Changed models
The following estimators and functions, when fit with the same data and
parameters, may produce different models from the previous version. This often
occurs due to changes in the modelling logic (bug fixes or enhancements), or in
random sampling procedures.
* discriminant_analysis.LinearDiscriminantAnalysis for multiclass
* discriminant_analysis.LinearDiscriminantAnalysis with 'eigen'
* linear_model.BayesianRidge |Fix|
* Decision trees and derived ensembles when both max_depth and
max_leaf_nodes are set. |Fix|
* linear_model.LogisticRegression and
linear_model.LogisticRegressionCV with 'saga' solver. |Fix|
* ensemble.GradientBoostingClassifier |Fix|
* neural_network.MLPClassifier |Fix|
* svm.SVC.decision_function and
* linear_model.SGDClassifier and any derived classifiers. |Fix|
* Any model using the linear_model.sag.sag_solver function with a 0
seed, including linear_model.LogisticRegression,
and linear_model.RidgeCV with 'sag' solver. |Fix|
* linear_model.RidgeCV when using generalized cross-validation
with sparse inputs. |Fix|
Details are listed in the changelog below.
(While we are trying to better inform users by providing this information, we
cannot assure that this list is complete.)
+ Known Major Bugs
* The default max_iter for linear_model.LogisticRegression is too
small for many solvers given the default tol. In particular, we
accidentally changed the default max_iter for the liblinear solver from
1000 to 100 iterations in released in version 0.16.
In a future release we hope to choose better default max_iter and tol
heuristically depending on the solver.
+ Support for Python 3.4 and below has been officially dropped.
* API: The R2 score used when calling score on a regressor will use
multioutput='uniform_average' from version 0.23 to keep consistent with
metrics.r2_score. This will influence the score method of all
the multioutput regressors (except for
* Enhancement: Added support to bin the data passed into
calibration.calibration_curve by quantiles instead of uniformly
between 0 and 1..
* Enhancement: Allow n-dimensional arrays as input for
* MajorFeature: A new clustering algorithm: cluster.OPTICS: an
algoritm related to cluster.DBSCAN, that has hyperparameters easier
to set and that scales better,
* Fix: Fixed a bug where cluster.Birch could occasionally raise an
* Fix: Fixed a bug in cluster.KMeans where empty clusters weren't
correctly relocated when using sample weights..
* API: The n_components_ attribute in cluster.AgglomerativeClustering
and cluster.FeatureAgglomeration has been renamed to
* Enhancement: cluster.AgglomerativeClustering and
cluster.FeatureAgglomeration now accept a distance_threshold
parameter which can be used to find the clusters instead of n_clusters.
* API: compose.ColumnTransformer is no longer an experimental
* Fix: Added support for 64-bit group IDs and pointers in SVMLight files..
* Fix: datasets.load_sample_images returns images with a deterministic
* Enhancement: decomposition.KernelPCA now has deterministic output
(resolved sign ambiguity in eigenvalue decomposition of the kernel matrix)..
* Fix: Fixed a bug in decomposition.KernelPCA, fit().transform()
now produces the correct output (the same as fit_transform()) in case
of non-removed zero eigenvalues (remove_zero_eig=False).
fit_inverse_transform was also accelerated by using the same trick as
fit_transform to compute the transform of X.
* Fix: Fixed a bug in decomposition.NMF where init = 'nndsvd',
init = 'nndsvda', and init = 'nndsvdar' are allowed when
n_components < n_features instead of
n_components <= min(n_samples, n_features).
* API: The default value of the init argument in
decomposition.non_negative_factorization will change from
random to None in version 0.23 to make it consistent with
decomposition.NMF. A FutureWarning is raised when
the default value is used..
* Enhancement: discriminant_analysis.LinearDiscriminantAnalysis now
preserves float32 and float64 dtypes.
* Fix: A ChangedBehaviourWarning is now raised when
discriminant_analysis.LinearDiscriminantAnalysis is given as
parameter n_components > min(n_features, n_classes - 1), and
n_components is changed to min(n_features, n_classes - 1) if so.
Previously the change was made, but silently..
* Fix: Fixed a bug in discriminant_analysis.LinearDiscriminantAnalysis
where the predicted probabilities would be incorrectly computed in the
* Fix: Fixed a bug in discriminant_analysis.LinearDiscriminantAnalysis
where the predicted probabilities would be incorrectly computed with eigen
* Fix: Fixed a bug in dummy.DummyClassifier where the
predict_proba method was returning int32 array instead of
float64 for the stratified strategy..
* Fix: Fixed a bug in dummy.DummyClassifier where it was throwing a
dimension mismatch error in prediction time if a column vector y with
shape=(n, 1) was given at fit time.
* MajorFeature: Add two new implementations of
gradient boosting trees: ensemble.HistGradientBoostingClassifier
and ensemble.HistGradientBoostingRegressor. The implementation of
these estimators is inspired by
LightGBM and can be orders of
magnitude faster than ensemble.GradientBoostingRegressor and
ensemble.GradientBoostingClassifier when the number of samples is
larger than tens of thousands of samples. The API of these new estimators
is slightly different, and some of the features from
ensemble.GradientBoostingRegressor are not yet supported.
These new estimators are experimental, which means that their results or
their API might change without any deprecation cycle. To use them, you
need to explicitly import enable_hist_gradient_boosting::
>>> # explicitly require this experimental feature
>>> from sklearn.experimental import enable_hist_gradient_boosting # noqa
>>> # now you can import normally from sklearn.ensemble
>>> from sklearn.ensemble import HistGradientBoostingClassifier.
* Feature: Add ensemble.VotingRegressor
which provides an equivalent of ensemble.VotingClassifier
for regression problems.
* Efficiency: Make ensemble.IsolationForest prefer threads over
processes when running with n_jobs > 1 as the underlying decision tree
fit calls do release the GIL. This changes reduces memory usage and
* Efficiency: Make ensemble.IsolationForest more memory efficient
by avoiding keeping in memory each tree prediction..
* Efficiency: ensemble.IsolationForest now uses chunks of data at
prediction step, thus capping the memory usage..
* Efficiency: sklearn.ensemble.GradientBoostingClassifier and
sklearn.ensemble.GradientBoostingRegressor now keep the
input y as float64 to avoid it being copied internally by trees..
* Enhancement: Minimized the validation of X in
ensemble.AdaBoostClassifier and ensemble.AdaBoostRegressor.
* Enhancement: ensemble.IsolationForest now exposes warm_start
parameter, allowing iterative addition of trees to an isolation
* Fix: The values of feature_importances_ in all random forest based
> sum up to 1
> all the single node trees in feature importance calculation are ignored
> in case all trees have only one single node (i.e. a root node),
feature importances will be an array of all zeros.
* Fix: Fixed a bug in ensemble.GradientBoostingClassifier and
ensemble.GradientBoostingRegressor, which didn't support
scikit-learn estimators as the initial estimator. Also added support of
initial estimator which does not support sample weights. and.
* Fix: Fixed the output of the average path length computed in
ensemble.IsolationForest when the input is either 0, 1 or 2.
* Fix: Fixed a bug in ensemble.GradientBoostingClassifier where
the gradients would be incorrectly computed in multiclass classification
* Fix: Fixed a bug in ensemble.GradientBoostingClassifier where
validation sets for early stopping were not sampled with stratification..
* Fix: Fixed a bug in ensemble.GradientBoostingClassifier where
the default initial prediction of a multiclass classifier would predict the
classes priors instead of the log of the priors..
* Fix: Fixed a bug in ensemble.RandomForestClassifier where the
predict method would error for multiclass multioutput forests models
if any targets were strings..
* Fix: Fixed a bug in ensemble.gradient_boosting.LossFunction and
ensemble.gradient_boosting.LeastSquaresError where the default
value of learning_rate in update_terminal_regions is not consistent
with the document and the caller functions. Note however that directly using
these loss functions is deprecated..
* Fix: ensemble.partial_dependence (and consequently the new
version sklearn.inspection.partial_dependence) now takes sample
weights into account for the partial dependence computation when the
gradient boosting model has been trained with sample weights..
* API: ensemble.partial_dependence and
ensemble.plot_partial_dependence are now deprecated in favor of
* Fix: ensemble.VotingClassifier and
ensemble.VotingRegressor were failing during fit in one
of the estimators was set to None and sample_weight was not None..
* API: ensemble.VotingClassifier and
ensemble.VotingRegressor accept 'drop' to disable an estimator
in addition to None to be consistent with other estimators (i.e.,
pipeline.FeatureUnion and compose.ColumnTransformer)..
* API: Deprecated externals.six since we have dropped support for
* Fix: If input='file' or input='filename', and a callable is given as
the analyzer, sklearn.feature_extraction.text.HashingVectorizer,
sklearn.feature_extraction.text.CountVectorizer now read the data
from the file(s) and then pass it to the given analyzer, instead of
passing the file name(s) or the file object(s) to the analyzer..
* MajorFeature: Added impute.IterativeImputer, which is a strategy
for imputing missing values by modeling each feature with missing values as a
function of other features in a round-robin fashion.
The API of IterativeImputer is experimental and subject to change without any
deprecation cycle. To use them, you need to explicitly import
>>> from sklearn.experimental import enable_iterative_imputer # noqa
>>> # now you can import normally from sklearn.impute
>>> from sklearn.impute import IterativeImputer
* Feature: The impute.SimpleImputer and
impute.IterativeImputer have a new parameter 'add_indicator',
which simply stacks a impute.MissingIndicator transform into the
output of the imputer's transform. That allows a predictive estimator to
account for missingness.
* Fix: In impute.MissingIndicator avoid implicit densification by
raising an exception if input is sparse add missing_values property
is set to 0..
* Fix: Fixed two bugs in impute.MissingIndicator. First, when
X is sparse, all the non-zero non missing values used to become
explicit False in the transformed data. Then, when
features='missing-only', all features used to be kept if there were no
missing values at all..
* Feature: Partial dependence plots
(inspection.plot_partial_dependence) are now supported for
any regressor or classifier (provided that they have a predict_proba
* Feature: Allow different dtypes (such as float32) in
* Enhancement: linear_model.Ridge now preserves float32 and
* Feature: linear_model.LogisticRegression and
linear_model.LogisticRegressionCV now support Elastic-Net penalty,
with the 'saga' solver..
* Feature: Added linear_model.lars_path_gram, which is
linear_model.lars_path in the sufficient stats mode, allowing
users to compute linear_model.lars_path without providing
X and y..
* Efficiency: linear_model.make_dataset now preserves
float32 and float64 dtypes, reducing memory consumption in stochastic
gradient, SAG and SAGA solvers.
* Enhancement: linear_model.LogisticRegression now supports an
unregularized objective when penalty='none' is passed. This is
equivalent to setting C=np.inf with l2 regularization. Not supported
by the liblinear solver..
* Enhancement: sparse_cg solver in linear_model.Ridge
now supports fitting the intercept (i.e. fit_intercept=True) when
inputs are sparse..
* Enhancement: The coordinate descent solver used in Lasso, ElasticNet,
etc. now issues a ConvergenceWarning when it completes without meeting the
* Fix: Fixed a bug in linear_model.LogisticRegression and
linear_model.LogisticRegressionCV with 'saga' solver, where the
weights would not be correctly updated in some cases..
* Fix: Fixed the posterior mean, posterior covariance and returned
regularization parameters in linear_model.BayesianRidge. The
posterior mean and the posterior covariance were not the ones computed
with the last update of the regularization parameters and the returned
regularization parameters were not the final ones. Also fixed the formula of
the log marginal likelihood used to compute the score when
* Fix: Fixed a bug in linear_model.LassoLarsIC, where user input
copy_X=False at instance creation would be overridden by default
parameter value copy_X=True in fit.
* Fix: Fixed a bug in linear_model.LinearRegression that
was not returning the same coeffecients and intercepts with
fit_intercept=True in sparse and dense case.
* Fix: Fixed a bug in linear_model.HuberRegressor that was
broken when X was of dtype bool..
* Fix: Fixed a performance issue of saga and sag solvers when called
in a joblib.Parallel setting with n_jobs > 1 and
backend="threading", causing them to perform worse than in the sequential
* Fix: Fixed a bug in
linear_model.stochastic_gradient.BaseSGDClassifier that was not
deterministic when trained in a multi-class setting on several threads..
* Fix: Fixed bug in linear_model.ridge_regression,
caused unhandled exception for arguments return_intercept=True and
solver=auto (default) or any other solver different from sag.
* Fix: linear_model.ridge_regression will now raise an exception
if return_intercept=True and solver is different from sag. Previously,
only warning was issued.
* Fix: linear_model.ridge_regression will choose sparse_cg
solver for sparse inputs when solver=auto and sample_weight
is provided (previously cholesky solver was selected).
* API: The use of linear_model.lars_path with X=None
while passing Gram is deprecated in version 0.21 and will be removed
in version 0.23. Use linear_model.lars_path_gram instead..
* API: linear_model.logistic_regression_path is deprecated
in version 0.21 and will be removed in version 0.23..
* Fix: linear_model.RidgeCV with generalized cross-validation
now correctly fits an intercept when fit_intercept=True and the design
matrix is sparse.
* Efficiency: Make manifold.tsne.trustworthiness use an inverted index
instead of an np.where lookup to find the rank of neighbors in the input
space. This improves efficiency in particular when computed with
lots of neighbors and/or small datasets..
* Feature: Added the metrics.max_error metric and a corresponding
'max_error' scorer for single output regression..
* Feature: Add metrics.multilabel_confusion_matrix, which calculates a
confusion matrix with true positive, false positive, false negative and true
negative counts for each class. This facilitates the calculation of set-wise
metrics such as recall, specificity, fall out and miss rate.
* Feature: metrics.jaccard_score has been added to calculate the
Jaccard coefficient as an evaluation metric for binary, multilabel and
multiclass tasks, with an interface analogous to metrics.f1_score.
* Feature: Added metrics.pairwise.haversine_distances which can be
accessed with metric='pairwise' through metrics.pairwise_distances
and estimators. (Haversine distance was previously available for nearest
* Efficiency: Faster metrics.pairwise_distances with n_jobs
> 1 by using a thread-based backend, instead of process-based backends.
* Efficiency: The pairwise manhattan distances with sparse input now uses the
BLAS shipped with scipy instead of the bundled BLAS.
* Enhancement: Use label accuracy instead of micro-average on
metrics.classification_report to avoid confusion. micro-average is
only shown for multi-label or multi-class with a subset of classes because
it is otherwise identical to accuracy.
* Enhancement: Added beta parameter to
metrics.v_measure_score to configure the
tradeoff between homogeneity and completeness.
* Fix: The metric metrics.r2_score is degenerate with a single sample
and now it returns NaN and raises exceptions.UndefinedMetricWarning..
* Fix: Fixed a bug where metrics.brier_score_loss will sometimes
return incorrect result when there's only one class in y_true..
* Fix: Fixed a bug in metrics.label_ranking_average_precision_score
where sample_weight wasn't taken into account for samples with degenerate
* API: The parameter labels in metrics.hamming_loss is deprecated
in version 0.21 and will be removed in version 0.23.
* Fix: The function metrics.pairwise.euclidean_distances, and
therefore several estimators with metric='euclidean', suffered from
numerical precision issues with float32 features. Precision has been
increased at the cost of a small drop of performance.
* API: metrics.jaccard_similarity_score is deprecated in favour of
the more consistent metrics.jaccard_score. The former behavior for
binary and multiclass targets is broken..
* Fix: Fixed a bug in mixture.BaseMixture and therefore on estimators
based on it, i.e. mixture.GaussianMixture and
mixture.BayesianGaussianMixture, where fit_predict and
fit.predict were not equivalent..
* Feature: Classes ~model_selection.GridSearchCV and
~model_selection.RandomizedSearchCV now allow for refit=callable
to add flexibility in identifying the best estimator.
* Enhancement: Classes ~model_selection.GridSearchCV,
~model_selection.RandomizedSearchCV, and methods
~model_selection.cross_validate, now print train scores when
return_train_scores is True and verbose > 2. For
~model_selection.validation_curve only the latter is required.
* Enhancement: Some CV splitter classes and
model_selection.train_test_split now raise ValueError when the
resulting training set is empty..
* Fix: Fixed a bug where model_selection.StratifiedKFold
shuffles each class's samples with the same random_state,
making shuffle=True ineffective..
* Fix: Added ability for model_selection.cross_val_predict to handle
multi-label (and multioutput-multiclass) targets with predict_proba-type
* Fix: Fixed an issue in ~model_selection.cross_val_predict where
method="predict_proba" returned always 0.0 when one of the classes was
excluded in a cross-validation fold.
* Fix: Fixed an issue in multiclass.OneVsOneClassifier.decision_function
where the decision_function value of a given sample was different depending on
whether the decision_function was evaluated on the sample alone or on a batch
containing this same sample due to the scaling used in decision_function..
* Fix: Fixed a bug in multioutput.MultiOutputClassifier where the
predict_proba method incorrectly checked for predict_proba attribute in
the estimator object.
* MajorFeature: Added neighbors.NeighborhoodComponentsAnalysis for
metric learning, which implements the Neighborhood Components Analysis
* API: Methods in neighbors.NearestNeighbors :
now raise NotFittedError, rather than AttributeError,
when called before fit.
* Fix: Fixed a bug in neural_network.MLPClassifier and
neural_network.MLPRegressor where the option shuffle=False
was being ignored..
* Fix: Fixed a bug in neural_network.MLPClassifier where
validation sets for early stopping were not sampled with stratification. In
the multilabel case however, splits are still not stratified..
* Feature: pipeline.Pipeline can now use indexing notation (e.g.
my_pipeline[0:-1]) to extract a subsequence of steps as another Pipeline
instance. A Pipeline can also be indexed directly to extract a particular
step (e.g. my_pipeline['svc']), rather than accessing named_steps..
* Feature: Added optional parameter verbose in pipeline.Pipeline,
compose.ColumnTransformer and pipeline.FeatureUnion
and corresponding make_ helpers for showing progress and timing of
* Enhancement: pipeline.Pipeline now supports using 'passthrough'
as a transformer, with the same effect as None..
* Enhancement: pipeline.Pipeline implements __len__ and
therefore len(pipeline) returns the number of steps in the pipeline..
* Feature: preprocessing.OneHotEncoder now supports dropping one
feature per category with a new drop parameter..
* Efficiency: preprocessing.OneHotEncoder and
preprocessing.OrdinalEncoder now handle pandas DataFrames more
* Efficiency: Make preprocessing.MultiLabelBinarizer cache class
mappings instead of calculating it every time on the fly.
* Efficiency: preprocessing.PolynomialFeatures now supports
compressed sparse row (CSR) matrices as input for degrees 2 and 3. This is
typically much faster than the dense case as it scales with matrix density
and expansion degree (on the order of density^degree), and is much, much
faster than the compressed sparse column (CSC) case..
* Efficiency: Speed improvement in preprocessing.PolynomialFeatures,
in the dense case. Also added a new parameter order which controls output
order for further speed performances..
* Fix: Fixed the calculation overflow when using a float16 dtype with
* Fix: Fixed a bug in preprocessing.QuantileTransformer and
preprocessing.quantile_transform to force n_quantiles to be at most
equal to n_samples. Values of n_quantiles larger than n_samples were either
useless or resulting in a wrong approximation of the cumulative distribution
* API: The default value of copy in preprocessing.quantile_transform
will change from False to True in 0.23 in order to make it more consistent
with the default copy values of other functions in
preprocessing and prevent unexpected side effects by modifying
the value of X inplace..
* Fix: Fixed an issue in svm.SVC.decision_function when
decision_function_shape='ovr'. The decision_function value of a given
sample was different depending on whether the decision_function was evaluated
on the sample alone or on a batch containing this same sample due to the
scaling used in decision_function..
* Feature: Decision Trees can now be plotted with matplotlib using
tree.plot_tree without relying on the dot library,
removing a hard-to-install dependency..
* Feature: Decision Trees can now be exported in a human readable
textual format using tree.export_text.
* Feature: get_n_leaves() and get_depth() have been added to
tree.BaseDecisionTree and consequently all estimators based
on it, including tree.DecisionTreeClassifier,
* Fix: Trees and forests did not previously predict multi-output
classification targets with string labels, despite accepting them in fit..
* Fix: Fixed an issue with tree.BaseDecisionTree
and consequently all estimators based
on it, including tree.DecisionTreeClassifier,
and tree.ExtraTreeRegressor, where they used to exceed the given
max_depth by 1 while expanding the tree if max_leaf_nodes and
max_depth were both specified by the user. Please note that this also
affects all ensemble methods using decision trees..
* Feature: utils.resample now accepts a stratify parameter for
sampling according to class distributions..
* API: Deprecated warn_on_dtype parameter from utils.check_array
and utils.check_X_y. Added explicit warning for dtype conversion
in check_pairwise_arrays if the metric being passed is a
pairwise boolean metric..
+ Multiple modules
* MajorFeature: The __repr__() method of all estimators (used when calling
print(estimator)) has been entirely re-written, building on Python's
pretty printing standard library. All parameters are printed by default,
but this can be altered with the print_changed_only option in
* MajorFeature: Add estimators tags: these are annotations of estimators
that allow programmatic inspection of their capabilities, such as sparse
matrix support, supported output types and supported methods. Estimator
tags also determine the tests that are run on an estimator when
check_estimator is called.
* Efficiency: Memory copies are avoided when casting arrays to a different
dtype in multiple estimators..
* Fix: Fixed a bug in the implementation of the our_rand_r
helper function that was not behaving consistently across platforms.
* Enhancement: Joblib is no longer vendored in scikit-learn, and becomes a
dependency. Minimal supported version is joblib 0.11, however using
version >= 0.13 is strongly recommended..
+ Changes to estimator checks
These changes mostly affect library developers.
* Add check_fit_idempotent to
~utils.estimator_checks.check_estimator, which checks that
when fit is called twice with the same data, the ouput of
predict, predict_proba, transform, and decision_function does not
* Many checks can now be disabled or configured with estimator_tags..