Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

split outputs, build with rattler-build, flang 19 #98

Merged
merged 19 commits into from
Dec 12, 2024

Conversation

minrk
Copy link
Member

@minrk minrk commented Nov 25, 2024

use rattler because this is a lot of outputs and it saves a lot of time for the many env setup/teardown transactions over conda-build

adds suitesparse-mongoose as an additional component and unskips spex on windows, now that the required gmp is available there.

Further splitting not done, that could be, and I think could be done later without major consequence:

closes #100
closes #99

@conda-forge-admin

This comment was marked as outdated.

use rattler because this is a lot of outputs and it takes forever with conda-build

adds mongoose as an additional component
@minrk
Copy link
Member Author

minrk commented Nov 26, 2024

lint error is conda-forge/conda-smithy#2165 already fixed by conda-forge/conda-smithy#2166

@minrk
Copy link
Member Author

minrk commented Nov 26, 2024

This actually runs all the builds independently in sequence. A faster approach would be to keep the single 'build and install everything' and use files. Downside of files: we need to keep track of which files go with which package, and know about every added file for each release. The approach in this PR is a lot easier to keep correct, at the expense of some time (still only 5 minutes on linux-64 thanks to the switch to rattler-build).

@xhochy
Copy link
Member

xhochy commented Nov 26, 2024

At this point you could also think of splitting it into libpkg and libpkg-devel. If you do it at a later point, you can no longer use libpkg as the package containing only the runtime library as this would probably break existing packages (too much).

@minrk
Copy link
Member Author

minrk commented Nov 26, 2024

@xhochy that's a good point. I did think of that and figured the cost of the dev files wasn't big enough to justify the recipe complexity. It's a little hard with this multi-package layout where multiple builds have to run unless I go with a single build and pure files splitting since I don't think I can run a bunch of builds and do files splitting from several different build steps.

One way to do the splitting that I think would be maintainable is to switch to a single build and do files splitting where:

  • lib* packages include only libs (a simple glob for each)
  • a single suitesparse-devel includes all the headers/cmake files, making the file-include patterns tractable (include/, lib/cmake/ etc.)

so that a package depending on e.g. umfpack would have host:

- libumfpack # lib, run_exports
- suitesparse-devel # headers, etc., no run_exports

Do you think that's worth it?

Some numbers for the fully installed suitesparse to help inform the choice, which should be unchanged (plus the addition of suitesparse-mongoose):

  • total instal size: 5.4MB, 159 files
  • libs: 4.1MB, 51 files (2/3 are symlinks)
  • include: 800kB, 19 files
  • cmake, pkg-config: 500kB, 89 files

so the devel package would reduce the number of files for a full suitesparse install by 2/3 and the size by only 20%.

The single-build would also have to wait for conda-forge/conda-smithy#2057 to resolve because rattler-build currently has the build cache behind an --experimental flag we can't yet pass.

@xhochy
Copy link
Member

xhochy commented Nov 26, 2024

@xhochy that's a good point. I did think of that and figured the cost of the dev files wasn't big enough to justify the recipe complexity.

That is something where I don't have an opinion on as I am not a user of this package, but I stumbled on this case of libpkg vs libpkg-devel several times in the last days and wanted to raise it again, so that if you would have long-term plans for it, that you don't "fall into the same trap":

@minrk
Copy link
Member Author

minrk commented Nov 26, 2024

Since most packages don't split -devel on conda-forge, I've mostly been doing it considering the size/quantity of devel files compared to both the library itself, and the size/frequency of a typical env that depends on it. 1MB of headers in an env that's unlikely to be less that 300MB due to other dependencies doesn't seem entirely worth it.

I also don't think splitting further libumfpack into libumfpack-devel down the line is likely to break anything, as it would only mean the next build depending on libumfpack would need to add -devel, but runtime would only see an install-size savings without any breakage. Or is there something I'm missing there?

required gmp is now available, so give it a go
gives slightly clearer if/else for single dependency category (cmake generator)
@minrk minrk marked this pull request as ready for review November 26, 2024 11:39
@minrk
Copy link
Member Author

minrk commented Nov 26, 2024

@conda-forge-admin please lint

recipe/recipe.yaml Outdated Show resolved Hide resolved
recipe/recipe.yaml Outdated Show resolved Hide resolved
@conda-forge-admin
Copy link
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/recipe.yaml) and found it was in an excellent condition.

@minrk
Copy link
Member Author

minrk commented Dec 4, 2024

This feels like a big migration to do by myself, so I'd like some approvals before merging. But at this point, I'm not sure who from core and/or @conda-forge/suitesparse should weigh in, but any 👍s would be appreciated.

Copy link
Member

@xhochy xhochy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just some questions for my own understanding.

recipe/recipe.yaml Show resolved Hide resolved
recipe/recipe.yaml Outdated Show resolved Hide resolved
recipe/recipe.yaml Show resolved Hide resolved
Copy link
Member

@xhochy xhochy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for clarifying things!

Copy link
Member

@h-vetinari h-vetinari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just some minor questions

recipe/recipe.yaml Show resolved Hide resolved
recipe/build-one.bat Outdated Show resolved Hide resolved
recipe/build-one.sh Outdated Show resolved Hide resolved
minrk and others added 4 commits December 9, 2024 09:34
instead of SOVERSION
TL;DR: We are trying to switch our Fortran compilers on windows to flang.
This is not 100% guaranteed to work, but should be fine in the majority of cases.

The new LLVM-based flang has become mature enough that it should be possible to
broadly switch over our Fortran compilers on windows to it (until now we only had
an ancient pre-LLVM flang 5, or alternatively the GCC-based `m2w64_fortran`).

As such, this PR attempts to homogenize any use of `m2w64_fortran` and other `m2w64_*`
compilers to our default stack (which would then be MSVC + flang on windows), with
the exception of feedstocks for R-packages, which stay on the `m2w64_` compilers.

Recipes that have hard-coded expectations about the name of the fortran compiler
will need to adjust to use `%FC%` or `flang-new` for the compiler name. Similarly,
you may need to change the linker to `%LD%` or use `lld-link`.

It is also possible that you run into compilation errors due to differences in
compiler behaviour, bugs or as-yet unimplemented features. In case of compilation
errors due to stricter default language standards, you should be able to fix things
by passing `-std=legacy` to `FFLAGS`.

If you have problems with this PR, feel free to ping the @c-f/flang-activation team.
In case you have convinced yourself that flang really is not ready yet to be used to
compile a given feedstock, you may also close this migrator PR.

# Conflicts:
#	recipe/meta.yaml
@minrk minrk changed the title split outputs, build with rattler-build split outputs, build with rattler-build, flang 19 Dec 9, 2024
@minrk minrk merged commit a289ee4 into conda-forge:main Dec 12, 2024
9 checks passed
@minrk minrk deleted the split-rattler branch December 12, 2024 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants