CI is broken, Windows-WSL (and some other Linux) users need some extra systems packages

[pgunn]

Very recently, we've seen dependencies on OpenGL system packages turning up in both Github-CI and for Ubuntu-on-WSL users. The former is broken, the latter is fixable with some extra steps.

The dependency is coming in through OpenCV, but it's not their fault; unfortunately they recently changed dependencies to Qt6, which for some reason needs libEGL. This is tracked upstream here:

conda-forge/opencv-feedstock#401

The suggested solution - depending on headless build variants of opencv, won't work for us (because I think we use opengl to play things occasionally).

We either need:
A) To convince opencv to not build against Qt6
B) To convince the Qt6 developers to remove dependencies on OpenGL
C) To somehow handle getting WSL users to install some additional system depencies (annoying but not hard) and to get Github's workers to install opengl libraries (no idea what's involved, may or may not be possible) or to have a different yaml file for CI (really don't want to do this)

Investigating all of these.

[hmaarrfk]

why not add a constraint to use the opencv build for qt5.

instead of depending on:

opencv =*=headless*

in your tests, you can install

opencv =*=qt5*

for now, this will work as you transition to qt6 (eventually you will have to).

I also notice that your github CIs are using ubuntu. Are you able to add the following to your setup instructions:

sudo apt install -y \
        libglapi-mesa \
        libegl-mesa0 \
        libegl1 \
        libopengl0 \
        libgl1-mesa-glx

I was meaning to document the ubuntu dependencies but we just go to the "minimal" set this morning on our own CI.

[pgunn]

This is a good short-term solution (I'm likely to do it), although in the longer term I suspect opencv will eventually stop doing builds for qt5 and I'm worried this solution just kicks the can down the road. In 2 years do you think opencv will have qt5 builds?

[pgunn]

@hmaarrfk Looks like if I do that I need to have separate dependencies based on OS; osx-64 doesn't have any packages that match that build string.

[hmaarrfk]

does your application actually use opencv's highGUI?

I find:

CaImAn/caiman/source_extraction/cnmf/estimates.py

Line 700 in 83ce6ce

cv2.imshow('frame', frame.astype('uint8'))

but it seems that most of your code uses matplotlib.

I strongly suspect that many mature pieces of software like yours likely has been transitionning away from opencv's GUI since it is more barebones than Matplotlib (in my opinion).

[pgunn]

@hmaarrfk
It doesn't use OpenCV much to display things. I don't think we're wed to it long-term.

[hmaarrfk]

@hmaarrfk Looks like if I do that I need to have separate dependencies based on OS; osx-64 doesn't have any packages that match that build string.

Given that your application "works" on OSX, I strongly suspect you aren't using OpenCV's high GUI from conda-forge since we never built it for OSX ^_^.

I personally think that the OpenGL issues are CI-only (sorry for this). I believe most GUI users have the opengl dependencies installed.
I believe the cloud / data center users will appreciate the ability to install opencv without qt on their servers (saving 100s of MB of dependencies).

If you are sure you cannot install system dependencies to:
https://github.com/flatironinstitute/CaImAn/blob/main/.github/workflows/run_tests.yml#L24

[hmaarrfk]

If you are sure you cannot install system dependencies to:
https://github.com/flatironinstitute/CaImAn/blob/main/.github/workflows/run_tests.yml#L24

then I would agree this is a real problem for many users on github actions, but I think you should be able to add the sudo apt install lines I suggested to "move toward qt6".

[hmaarrfk]

FWIW, I'm using Qt6 on both Ubuntu 20.04 and Ubuntu 22.04, so I think that will hit most users of new installations in 2024. I have not personally tested Ubuntu 18.04.

[pgunn]

@hmaarrfk
We also got opencv import failures for WSL users (because the WSL Ubuntu doesn't come with OpenGL drivers; they're not hard to install though).

[pgunn]

@hmaarrfk Example: #1308

[hmaarrfk]

We also got opencv import failures for WSL users

Yeah.... WSL is difficult. I've never used it myself. Your users must already be following something to install the minimal. For about 5 years on conda-forge we've required users to install some system dependency for opencv.

conda-forge/opencv-feedstock@813fd05#diff-e36d294fb902698a9e5c5b991fd921efb1cfb09609fc4db4f94d72759c9e011a

I agree that the increase is annoying for end users of CIs (and what I didn't expected is WSL). However, perhaps updating the instructions to include the 5 packages I listed will help too?

[hmaarrfk]

also this conversation might give some context on other maintainers having limited time to support qt6.
conda-forge/qt-main-feedstock#208 (comment)

BTW, if you need the OSX qt5 builds, please say so on conda-forge, i'm happy to expand the build matrix (but I would also build out the qt6 ones at that point). I was just trying to save some CIs.... they already count about 90 different builds....

[pgunn]

@hmaarrfk It's doable. I really wish we could get the qt6 people not to include troublesome dependencies, but there's only so much convincing we can do. If we have to we'll just add more steps to our docs and hope our users usually will follow them. Software maintenance is often like this.

I'll test the github actions/CI stuff soon.

[hmaarrfk]

Software maintenance is often like this.

yeah, i delayed building out qt6 for a while. But on Ubuntu I've seen some subtle benefits so I decided to see if it could help others.

Many distros have stopped shipping Qt5, so if developers can't even "try" qt6, then they won't ever be able to transition their packages.

BTW, I'm not sure if this kind of package would be helpful for you:
conda-forge/staged-recipes#25688

I'm scared of using a heavy handed approach to just create new packages without others expressing interest in it.

[pgunn]

No worries; I'll talk to the qt6 packagers, and now that our CI is fixed, I'll have to add some additional docs for WSL users. Every extra step makes things a little harder for support, but we're all in this big compromise between a lot of values and efforts and I don't expect we'll always win. Thanks for the help and the chat!

(it's also useful to know that we're allowed to use sudo to install extra systems packages with github's workers; I wasn't aware that was permitted)

[hmaarrfk]

(Ps. I also help package qt6 but it’s really good to hear from users on what to focus our attention on!!)