"AttributeError: np.float_ was removed in the NumPy 2.0 release" causing CI job failures during installation step

[lbianchi-lbl]

• See e.g. https://github.com/watertap-org/watertap/actions/runs/9553673592/job/26333049284
• I've been trying to reproduce this locally with a fresh environment, without success
• This is especially puzzling since the installed version of NumPy in the CI environment is 1.26.4

   Traceback (most recent call last):
    File "/usr/share/miniconda/envs/watertap-dev/bin/idaes", line 5, in <module>
      from idaes.commands.base import command_base
    File "/usr/share/miniconda/envs/watertap-dev/lib/python3.10/site-packages/idaes/commands/__init__.py", line 37, in <module>
      finder.find_spec(module_name).loader.load_module(module_name)
    File "/usr/share/miniconda/envs/watertap-dev/lib/python3.10/site-packages/idaes/commands/env_info.py", line 23, in <module>
      from idaes.core.util.env_info import EnvironmentInfo
    File "/usr/share/miniconda/envs/watertap-dev/lib/python3.10/site-packages/idaes/core/__init__.py", line 16, in <module>
      from .base.process_base import ProcessBlockData, useDefault, MaterialFlowBasis
    File "/usr/share/miniconda/envs/watertap-dev/lib/python3.10/site-packages/idaes/core/base/process_base.py", line 27, in <module>
      from pandas import DataFrame
    File "/usr/share/miniconda/envs/watertap-dev/lib/python3.10/site-packages/pandas/__init__.py", line 14, in <module>
      __import__(_dependency)
    File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
    File "/usr/share/miniconda/envs/watertap-dev/lib/python3.10/site-packages/pyomo/common/dependencies.py", line 475, in exec_module
      deferred.resolve()
    File "/usr/share/miniconda/envs/watertap-dev/lib/python3.10/site-packages/pyomo/common/dependencies.py", line 334, in resolve
      self._module, self._available = _perform_import(
    File "/usr/share/miniconda/envs/watertap-dev/lib/python3.10/site-packages/pyomo/common/dependencies.py", line 768, in _perform_import
      callback(module, True)
    File "/usr/share/miniconda/envs/watertap-dev/lib/python3.10/site-packages/pyomo/common/dependencies.py", line 1002, in _finalize_numpy
      _floats = [np.float_, np.float16, np.float32, np.float64]
    File "/usr/share/miniconda/envs/watertap-dev/lib/python3.10/site-packages/numpy/__init__.py", line 397, in __getattr__
      raise AttributeError(
  AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead.. Did you mean: 'float16'?

[lbianchi-lbl]

As it turns out, it looks like this "NumPy 2.0" error was actually introduced in NumPy 1.24.0: https://stackoverflow.com/questions/74844262/how-can-i-solve-error-module-numpy-has-no-attribute-float-in-python and https://numpy.org/doc/stable/release/1.24.0-notes.html#expired-deprecations.

It's still not clear to me why this is happening only now (or why I'm unable to reproduce the error locally), since the latest relevant commit in Pyomo for that line is from 4 months ago (https://github.com/Pyomo/pyomo/blame/7bbe3a7849f2511876118ba66132c6e7fd1dd344/pyomo/common/dependencies.py#L1002), and NumPy 1.24.0 is from Dec 2022 (!): https://pypi.org/project/numpy/1.24.0/.

EDIT: the latest version of pandas, that also shows up in the stack trace, is 2.2.2, released in April 2024: https://pypi.org/project/pandas/2.2.2/.

[lbianchi-lbl]

This might potentially be related to Pyomo/pyomo#3290, but really it's just based on the error message and the affected lines in pyomo.common.dependencies. There's still too many unclear things for me to make even an educated guess of what could be the actual cause.

[lbianchi-lbl]

OK, I might have figured out what's happening: numpy is getting installed both by Conda and pip.

As far as I understand, the Conda installation happens because of cyipopt, and we see here that the version being installed is 2.0.0:

This makes sense as there is no constraint in the conda install command that prevents this from happening. My guess is that the reason why adding conda install cyipopt in #1431 wasn't causing the CI to fail in the same way is that NumPy 2.0.0 hadn't been released yet at the time #1431 was merged.

I don't really understand the details of why both conda list and pip list listed the installed version of 1.26.4 (from pip) instead of 2.0.0 (from Conda), but it seems clear that this is a case of "Conda-pip interference" that we have managed to avoid until now (perhaps we've just been lucky).

For a short term fix, I think adding a NumPy version constraint when installing CyIpopt (e.g. conda install -c conda-forge cyipopt numpy<2) should work, but we need to figure out a way to make the Conda-installed CyIpopt "play nice" with the rest of our dev environment (where Python dependencies are managed by pip) until the idaes-ext CyIpopt is available.