numpy.ndarray size changed, may indicate binary incompatibility

[pingsutw]

🐛 Bug Description

/opt/conda/lib/python3.7/site-packages/qlib/data/data.py in <module>
     18 from multiprocessing import Pool
     19 
---> 20 from .cache import H
     21 from ..config import C
     22 from .ops import Operators

/opt/conda/lib/python3.7/site-packages/qlib/data/cache.py in <module>
     34 from .base import Feature
     35 
---> 36 from .ops import Operators
     37 
     38 

/opt/conda/lib/python3.7/site-packages/qlib/data/ops.py in <module>
     17 
     18 try:
---> 19     from ._libs.rolling import rolling_slope, rolling_rsquare, rolling_resi
     20     from ._libs.expanding import expanding_slope, expanding_rsquare, expanding_resi
     21 except ImportError:

/opt/conda/lib/python3.7/site-packages/qlib/data/_libs/rolling.pyx in init qlib.data._libs.rolling()

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

To Reproduce

Steps to reproduce the behavior:

1. conda create --name test python=3.7
2. pip install numpy cython pyqlib
3. Run this example (https://github.com/microsoft/qlib/blob/main/examples/workflow_by_code.ipynb) in jupyter notebook

Expected Behavior

Should successfully run this example

Screenshot

Environment

Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.

• Qlib version: 0.6.3
• numpy version: 1.20.0
• cython version: 0.29.21
• Python version: 3.7.7
• OS ( Linux):
• Commit number (optional, please provide it if you are using the dev version):

Additional Notes

[you-n-g]

This may be related to the recent upgrading of numpy.
We are investigating this issue.
Thanks for your report

[Derek-Wds]

@pingsutw Hi, thanks for the report. This error may due to the latest numpy release in the version 1.20.0. We recommend to install the lower version of numpy==1.19.5. You can use the following command to install Qlib:

pip install numpy==1.19.5
pip install pyqlib --ignore-installed numpy

We will try to see what is updated and make Qlib to be compatible with numpy 1.20.0 in the future.

[PayneJoe]

@pingsutw Hi, thanks for the report. This error may due to the latest numpy release in the version 1.20.0. We recommend to use the lower version of numpy==1.19.5. You can use the following command to install Qlib:

pip install numpy==1.19.5
pip install pyqlib --ignore-installed numpy

We will try to see what is updated and make Qlib to be compatible with numpy 1.20.0 in the future.

Same error still exists after reinstalling numpy package, while the numpy/qlib installations seem abnormal.

Looking in indexes: https://mirrors.aliyun.com/pypi/simple Collecting numpy==1.19.5 Using cached https://mirrors.aliyun.com/pypi/packages/66/d7/3b133b17e185f14137bc8afe7a41daf1f31556900f10238312a5ae9c7345/numpy-1.19.5-cp38-cp38-manylinux2010_x86_64.whl (14.9 MB) Installing collected packages: numpy ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. torchvision 0.8.1+cu110 requires pillow>=4.1.1, which is not installed. torch 1.7.0+cu110 requires dataclasses, which is not installed. Successfully installed numpy-1.19.5

Do we have to reinstall all downstream packages (qlib/torch/...) after reinstalling numpy?

[pingsutw]

Thanks @you-n-g and @Derek-Wds for the help.
I found that if I use pyqlib==0.6.1 with numpy==1.19.5, I can successfully run the example.
However, using pyqlib==0.6.3, I will get the same error.
cc @PayneJoe, Maybe you could try to downgrade your pyqlib

[PayneJoe]

Thanks @you-n-g and @Derek-Wds for the help.
I found that if I use pyqlib==0.6.1 with numpy==1.19.5, I can successfully run the example.
However, using pyqlib==0.6.3, I will get the same error.
cc @PayneJoe, Maybe you could try to downgrade your pyqlib

The error disappears! thanks, @pingsutw

In addition, few dnn models released within latest version may not work well. For one case, TableNet do not exists
within pyqlib==0.6.1 and numpy==1.19.5, while the upon error occurs within pyqlib==0.6.3 and numpy==1.19.5.

@Derek-Wds maybe you should re-confirm whether the newest dnn models work well.

[Derek-Wds]

Hi @pingsutw @PayneJoe , thanks for the report and sorry for any inconvenience this may bring to you. We are working on it now to make sure everything will work fine. Please stay tuned and we will update ASAP!

Thanks!

[Derek-Wds]

Hi, there is a recent merge #258 about this issue, please now try to reinstall qlib to see if any errors occurs. Thanks!

[PayneJoe]

It worked!

[qiuwei]

Got the same issue with a fresh install of the latest version(0.7.2.99) from pypi.

numpy == 1.19.5 
python == 3.7.1
OS == Ubuntu 20.04

EDIT:
Strange enough, upgrading numpy to 1.20 fixes the issue.