ValueError in sparse_array.astype when read-only with unsorted indices [scipy issue]

[StevenLOL]

To reproduce:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report
from sklearn.multiclass import OneVsRestClassifier

from sklearn.pipeline import Pipeline
from sklearn.svm import SVC, LinearSVC
from sklearn.datasets import fetch_20newsgroups

data_train = fetch_20newsgroups(subset='train')
data_test = fetch_20newsgroups(subset='test')




clf=OneVsRestClassifier(estimator=SVC(),n_jobs=-1)  ##Error when calling fit
#clf=OneVsRestClassifier(estimator=SVC(),n_jobs=1) ##NO Error if set to 1



pipeLine=Pipeline([('tfidf',TfidfVectorizer(min_df=10)),
    ('clf',clf)])

trainx=data_train.data
trainy=data_train.target
evalx=data_test.data
evaly=data_test.target
pipeLine.fit(trainx,trainy)

predictValue=pipeLine.predict(evalx)

print classification_report(evaly,predictValue)

Output:

ValueError: UPDATEIFCOPY base is read-only

Linux-4.2.0-19-generic-x86_64-with-Ubuntu-15.10-wily
('Python', '2.7.10 (default, Oct 14 2015, 16:09:02) \n[GCC 5.2.1 20151010]')
('NumPy', '1.10.4')
('SciPy', '0.17.0')
('Scikit-Learn', '0.18.dev0')

[BiaDarkia]

The traceback points to sklearn/externals/joblib/parallel.py as the origin of the error. I am not sure, whether this error should occur, but I will have a look into why exactly it is raised.

[raghavrv]

Related to #5956

Ping : @lesteve @ogrisel

[lesteve]

Another related, maybe more central, issue is #5481. IIRC estimators should avoid in-place modification of X, e.g. X -= X.mean(). cc @arthurmensch.

More explanation: when the input data is big enough, joblib use memmapping by default. This allows to share the input data across workers instead of having a copy of the input data in each worker. See this for more details. The memmap is opened in read-only mode because of possible data corruption if different workers write into the same data.

If you have access to the joblib.Parallel object, you can use max_nbytes=None to disable memmaping. Whether it will be faster than just doing n_jobs=1 depends on your particular use case I reckon. From a few cases I looked at, it looks like you generally don't have access to the underlying joblib.Parallel object when you create your estimator, so that doesn't really help in most cases.

[raghavrv]

Thanks for the lucid explanation Loic!

[BiaDarkia]

Would it be possible to add an optional parameter to estimators that sets max_nbytes=None when accessing joblib.Parallel when set to True? It shouldn't be too hard to implement and fix the issue.

[lesteve]

A more proper fix would to modify sklearn.svm.base.BaseLibSVM._sparse_fit so that it doesn't modify X in place.

Looking at this problem more closely I found a work-around in case it is useful:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report
from sklearn.multiclass import OneVsRestClassifier

from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
from sklearn.datasets import fetch_20newsgroups

data_train = fetch_20newsgroups(subset='train')
data_test = fetch_20newsgroups(subset='test')

clf = OneVsRestClassifier(estimator=SVC(), n_jobs=-1)


class MyTfidfVectorizer(TfidfVectorizer):
    def fit_transform(self, X, y):
        result = super(MyTfidfVectorizer, self).fit_transform(X, y)
        result.sort_indices()
        return result

pipeLine = Pipeline([('tfidf', MyTfidfVectorizer(min_df=10)),
                     ('clf', clf)])

trainx = data_train.data
trainy = data_train.target
evalx = data_test.data
evaly = data_test.target
pipeLine.fit(trainx, trainy)

predictValue = pipeLine.predict(evalx)

print(classification_report(evaly, predictValue))

This is based on the fact that the in-place modification only happens when the output of the TfidfVectorizer doesn't have its indices sorted.

[amueller]

IIRC estimators should avoid in-place modification of X, e.g. X -= X.mean()

There should be a copy parameter to control this.

[amueller]

@lesteve close as duplicate of #5481?

[lesteve]

It' does seem very very similar to #5481 althoug this one needs sparse matrices to be triggered IIRC.

[amueller]

Ok. We should add a common test anyhow that covers dense and sparse. But alright, let's leave this open to make sure we cover this case.

[armancohan]

Getting the same error.

[ryandolen]

Any updates here? Still getting the same error.

[lesteve]

No updates I am afraid. You are more welcome to submit a PR for this issue or use the work-around suggested in #6614 (comment).

[lesteve]

For the record I opened scipy/scipy#8678.

[csvankhede]

I got the similar Issue.

clf = OneVsRestClassifier(LogisticRegression(penalty = 'l2'),n_jobs = -1)
clf.fit(x_train_multilabel,y_train)
pred_y = clf.predict(x_test_multilabel)

Here when I changed the code as below it worked.

clf.fit(x_train_multilabel.copy(),y_train)

[rth]

Thanks @csvankhede ! Yes, it's a scipy bug as mentionned above. Adding a workaround for all code that uses sparse_array.astype in scikit-learn would probably be difficult.

Just for the record, would,

x_train_multilabel.sort_indices()
clf.fit(x_train_multilabel, y_train)

also work in your case?

[emsi]

This is based on the fact that the in-place modification only happens when the output of the TfidfVectorizer doesn't have its indices sorted.

Adding dummy sorting estimator to pipeline is even simpler workaround :)

class SortVectorizer():
    def fit(self, X, y=None):
        return self

    def transform(self, X, y=None):
        X.sort_indices()
        return X

[damienlancry]

Thanks for the workaround, I am no longer getting the error "ValueError: UPDATEIFCOPY base is read-only"
However I am now getting

TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The code to reproduce is the following:

lr_kwargs = dict(penalty="l1", random_state=42, solver="liblinear", fit_intercept=False)
ovrlr=OneVsRestClassifier(LogisticRegression(**lr_kwargs), n_jobs=-1)
X.sort_indices()
ovrlr.fit(X, Y)

Where X is a (1375308, 80614) scipy sparse matrix obtained with a CountVectorizer() and Y is a (1375308, 157) binarized label matrix.

I am using a machine with 72 CPU cores and 252 GB of RAM. I previously ran the same code with n_jobs=1 and it took 4hours56min to run so I was expecting a 72x speedup. It indeed crashes after 5 minutes which suggests that the training was almost finished.

Apologies if this is unrelated, or if I am late to the party.

EDIT: I used n_job=30 and it works, the job was crashing simply because the parallel workers were exhausting the ram.

[glemaitre]

I am closing this issue since it should be solved by installing the future SciPy release since scipy/scipy#18192 has been merged