All tokens are out of vocabulary in 2.0

[bittlingmayer]

vector is fine, but the is_oov bit is off.

>>> en = spacy.load('en_core_web_sm')
>>> x = en('This is a tessssst')
>>> [w.is_oov for w in x]
[True, True, True, True]
>>> en("welder")[0].is_oov
True

• Python version: 3.5.2
• Platform: Darwin-16.6.0-x86_64-i386-64bit
• spaCy version: 2.0.0a0

[bittlingmayer]

Related:

#315 - Some clients use is_oov as a spellchecker.

#1191 - 1.0 is the opposite, is_oov is False but vector is all 0s.

[honnibal]

Thanks. I didn't update the vocab data before exporting the model. .prob is broken too I think.

You can fix these by writing to the attributes:

lex = nlp.vocab[u'dog']
lex.prob = -3.4
lex.is_oov = False
doc = nlp(u'dog')
print(doc[0].prob, doc[0].is_oov)

[bittlingmayer]

If I understand the suggestion correctly, it won't work in this case, I can't write to the attributes because it is precisely their correct values that I need.

(For now I am just using Gensim with fastText-pretrained models for this functionality.)

[honnibal]

@bittlingmayer Sorry I missed this reply. I meant that you might get the values from the v1 model, and import them. Once you've saved out the model, the correct values will be there. The stable models will have the correct vocab data.

This is an issue with the model files rather than a bug in the codebase, so I'll relabel this.

[ines]

Merging this with #1457, as it'll be part of the same fix!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.