Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mamba segfault #1049

Closed
datapythonista opened this issue Jun 30, 2021 · 15 comments
Closed

Mamba segfault #1049

datapythonista opened this issue Jun 30, 2021 · 15 comments

Comments

@datapythonista
Copy link

Looks like Mamba started to randomly segfault in the Ibis CI recently, I guess since 0.14.1 was released.

From our CI:

1565 Segmentation fault mamba env update -n base --file=environment.yml

I didn't check in detail the first couple of times I saw it failing, but this last time seems it's in a Windows build with this version:

mamba conda-forge/win-64::mamba-0.14.1-py38hdd88130_0

I guess the other times it was also in the Windows builds, but not sure.

Let me know if I can provide further information. The file we're running is this environment.yml, to which we add the Python version dynamically (the last fail is Python 3.7, unsure about the previous).

@datapythonista
Copy link
Author

Probably related, we've been having Connection reset by peer from conda-forge servers frequently. Maybe that's what's causing the segfault, when it happens from mamba and not conda.

@wolfv
Copy link
Member

wolfv commented Jul 1, 2021

Hey, thanks for letting me know!

Was it Happening for the first time yesterday? In that case it could be related to a libsolv patch that I published. I'll have a look!

@wolfv
Copy link
Member

wolfv commented Jul 1, 2021

And sorry for the trouble!

@wolfv
Copy link
Member

wolfv commented Jul 1, 2021

I tried to reproduce using this file:

name: ibis-dev
channels:
  - nodefaults
  - conda-forge
dependencies:
  # Ibis hard dependencies
  - multipledispatch
  - numpy
  - pandas
  - pytz
  - regex
  - toolz
  - cached_property # for 3.7 compat, functools.cached_property is for >=3.8
  - setuptools

  # Ibis soft dependencies
  - sqlalchemy
  - python-graphviz

  # Dev tools
  - asv
  - black=19.10b0
  - click # few scripts in ci/
  - flake8
  - flake8-comprehensions # used by flake8, linting of unnecessary comprehensions
  - isort
  - jinja2<3 # feedstock
  - mypy
  - plumbum # few scripts in ci/ and dev/
  - pydocstyle
  - pytest
  - pytest-cov
  - pytest-mock

  # Release
  - twine
  - wheel
  - conda-smithy
  - conda-build # feedstock
  - ruamel.yaml # feedstock
  - pygit2 # dev/genrelease.py

  # Docs
  - pip
  - pip:
    - pysuerga
  - ipython
  - jupyter
  - matplotlib
  - nbconvert
  - nbsphinx
  - nomkl
  - semantic_version=2.6 # ibis-project/ibis/issues/2027
  - sphinx<4.0.0 # ibis-project/ibis/issues/2771
  - sphinx-releases
  - sphinx_rtd_theme
  - python=3.7
  - pyarrow
  - pytables

But so far I haven't been successful ...

One thing to note is that in the CI you are installing with

mamba env update -n base --file env.yml

So that you're installing these new packages into the base environment. In general I would probably do it differetnly (since this way the python vesrion of hte base env probably needs to be changed). Would be cleaner to make a new env.

@datapythonista
Copy link
Author

Was it Happening for the first time yesterday?

Sorry for the delay. I was checking, and seems like it was, but we didn't have much activity in our CI the days before, so can't really tell which of the two releases caused it. My bet is that this is failing when conda servers return the connection reset by peer. And that happens in waves from time to time. Not sure if that would be easy to emulate. I can try to have a look next week if you think that can be useful.

Thanks a lot for the help, and also for the advice. I thought that reusing the base environment it'd probably save a bit of time of installing packages already there. But it probably makes more sense to make sure our dependencies are installed in a clean environment. I'll have a look.

@wolfv
Copy link
Member

wolfv commented Jul 2, 2021

No worries. Is it still happening?

@datapythonista
Copy link
Author

No worries. Is it still happening?

We only had a CI run since I opened the issue, and it didn't fail with the segfault. I'm going to force few more and see.

@wolfv
Copy link
Member

wolfv commented Jul 2, 2021

Thanks!!

@wolfv
Copy link
Member

wolfv commented Jul 5, 2021

Hey @datapythonista did you have a chance to trigger a re-run? Did this happen again?

@datapythonista
Copy link
Author

Sorry for the delay. I kept trying, and I couldn't get it failing until now, but I just had a build with the segfault: https://github.com/ibis-project/ibis/pull/2828/checks?check_run_id=3004472824#step:3:477

@wolfv
Copy link
Member

wolfv commented Jul 7, 2021

I just published a new libsolv version that fixes at least one issue with the previous implementation of this PR: openSUSE/libsolv#457

I hope this also helps here but I am not sure, still, where exactly the issue lies.

@wolfv
Copy link
Member

wolfv commented Jul 8, 2021

I checked today with valgrind, and only found two relatively minor things:

@wolfv
Copy link
Member

wolfv commented Jul 8, 2021

hey @datapythonista I had a bit more of a look and let the workflow run 2 times in my fork, and indeed also got a segfault: https://github.com/wolfv/ibis/runs/3017295242

However, I also noticed that you're mixing anaconda and conda-forge in the base env (specifically the zlib package is kept from the anaconda channel). That could be a source of problems. I am trying to run it right now with strict repo priority to see if that improves things.

However, in general I think your current strategy is not optimal: you install things in the base env which also changes the python version (your base env is first 3.8, then you change it to 3.7).

I think it would be quite a bit cleaner to either use the provision-with-micromamba action or the setup-miniconda action (which let's you select mamba as well).

You could do the modifications to the env yaml in a simple step right before executing one of these two actions (just append the python pin in the run:... step).

@wolfv
Copy link
Member

wolfv commented Jul 8, 2021

hey @datapythonista I finally found it: a50a034 (#457)

😅

Sorry for this issue, 100% my fault. I'll push another update to the libsolv conda package.

And many thanks for reporting!

@wolfv wolfv closed this as completed Jul 8, 2021
@datapythonista
Copy link
Author

Amazing, thanks a lot for the fix, all the advice, and for mamba itself (it's saving me loads of time, we have problems with the conda solver all the time until we switched).

I created an issue for your suggestions on how to install our environment, will be taking care of them soon: ibis-project/ibis#2846

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants