Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dotnet: implement autoPatchcilHook #373107

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

GGG-KILLER
Copy link
Contributor

@GGG-KILLER GGG-KILLER commented Jan 12, 2025

Adds a new tool called patchcil which aims to be the CIL (.NET's intermediate language) equivalent of patchelf.

Also implements a new hook called autoPatchcilHook which uses the patchcil tool to rewrite all static (read: [DllImport]-based) native library uses to point to the correct location in the nix store.

Finally, rewrites avalonia-ilspy to use autoPatchcilHook as a (rather extreme, as it uses quite a few Xorg libraries, GTK 3 and OpenGL) test case.

cc: @NixOS/dotnet @js6pak @AngryAnt @emilytrau

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@GGG-KILLER GGG-KILLER changed the title Implement autoPatchcilHook dotnet: implement autoPatchcilHook Jan 12, 2025
@github-actions github-actions bot added 8.has: documentation This PR adds or changes documentation 6.topic: dotnet Language: .NET labels Jan 12, 2025
@nix-owners nix-owners bot requested review from emilytrau and AngryAnt January 12, 2025 03:50
@GGG-KILLER
Copy link
Contributor Author

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 373107


x86_64-linux

✅ 92 packages built:
  • ArchiSteamFarm
  • alttpr-opentracker
  • am2rlauncher
  • autoPatchcilHook
  • avalonia-ilspy
  • azure-artifacts-credprovider
  • azure-functions-core-tools
  • bicep
  • blendfarm
  • btcpayserver
  • btcpayserver-altcoins
  • cavalier
  • celeste64
  • certdump
  • clps2c-compiler
  • csharp-ls
  • csharpier
  • csharprepl
  • cyclonedx-cli
  • dafny
  • denaro
  • dependency-track
  • depotdownloader
  • discordchatexporter-cli
  • discordchatexporter-desktop
  • dotnet-ef
  • dotnet-outdated
  • dotnet-repl
  • fable
  • famistudio
  • fantomas
  • formula
  • fsautocomplete
  • galaxy-buds-client
  • garnet
  • gh-gei
  • git-credential-manager
  • github-runner
  • gitversion
  • ilspycmd
  • imewlconverter
  • inklecate
  • jackett
  • jellyfin
  • juniper
  • kavita
  • knossosnet
  • kryptor
  • libation
  • lubelogger
  • lumafly
  • marksman
  • mesen
  • mqttmultimeter
  • msbuild-structured-log-viewer
  • msgraph-cli
  • nbxplorer
  • netcoredbg
  • nexusmods-app
  • nexusmods-app-unfree
  • nixpkgs-manual
  • omnisharp-roslyn
  • opentabletdriver
  • openutau
  • osu-lazer
  • parabolic
  • patchcil
  • pbm
  • pinta
  • ps3-disc-dumper
  • pupdate
  • recyclarr
  • retrospy
  • ryujinx
  • ryujinx-greemdev
  • scarab
  • seq-cli
  • skeditor
  • slskd
  • smtp4dev
  • space-station-14-launcher
  • tagger
  • technitium-dns-server
  • technitium-dns-server-library
  • tone
  • torrentstream
  • upgrade-assistant
  • v2rayn
  • vrcadvert
  • wiseunpacker
  • xivlauncher
  • yafc-ce

@GGG-KILLER
Copy link
Contributor Author

GGG-KILLER commented Jan 25, 2025

A new change that I made with patchcil 0.2.1 is that now it uses Native AOT for a 200% speed improvement (albeit at the cost of a 5x size increase from 704K to 3.8M).

v0.2.0:

  Time (mean ± σ):      3.580 s ±  0.069 s    [User: 0.959 s, System: 2.956 s]
  Range (min … max):    3.480 s …  3.846 s    50 runs

v0.2.1 (No AOT):

  Time (mean ± σ):     807.8 ms ±  36.0 ms    [User: 562.5 ms, System: 275.0 ms]
  Range (min … max):   755.7 ms … 913.4 ms    50 runs

v0.2.1 (AOT):

  Time (mean ± σ):     413.4 ms ±  16.6 ms    [User: 170.4 ms, System: 240.3 ms]
  Range (min … max):   389.8 ms … 478.4 ms    50 runs

@GGG-KILLER
Copy link
Contributor Author

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 373107


x86_64-linux

⏩ 14 packages marked as broken and skipped:
  • beatsabermodmanager
  • boogie
  • dotnetPackages.Boogie
  • eddie
  • eventstore
  • msbuild
  • naps2
  • networkminer
  • openra
  • pablodraw
  • roslyn
  • sonarr
  • spotlight-downloader
  • wasabibackend
✅ 110 packages built:
  • ArchiSteamFarm
  • alttpr-opentracker
  • am2rlauncher
  • autoPatchcilHook (dotnetCorePackages.autoPatchcilHook)
  • avalonia-ilspy
  • azure-artifacts-credprovider
  • azure-functions-core-tools
  • bicep
  • blendfarm
  • bms-to-osu
  • btcpayserver
  • btcpayserver-altcoins
  • cavalier
  • celeste64
  • certdump
  • clps2c-compiler
  • csharp-ls
  • csharpier
  • csharprepl
  • cyclonedx-cli
  • dafny
  • denaro
  • dependency-track
  • depotdownloader
  • discordchatexporter-cli
  • discordchatexporter-desktop
  • dotnet-ef
  • dotnet-outdated
  • dotnet-repl
  • fable
  • famistudio
  • fantomas
  • formula
  • fsautocomplete
  • galaxy-buds-client
  • garnet
  • gh-gei
  • git-credential-manager
  • github-runner
  • gitversion
  • ilspycmd
  • imewlconverter
  • inklecate
  • jackett
  • jellyfin
  • juniper
  • kavita
  • knossosnet
  • kryptor
  • libation
  • lubelogger
  • lumafly
  • marksman
  • mesen
  • mqttmultimeter
  • msbuild-structured-log-viewer
  • msgraph-cli
  • nbxplorer
  • netcoredbg
  • nexusmods-app
  • nexusmods-app-unfree
  • nixpkgs-manual
  • omnisharp-roslyn
  • opentabletdriver
  • openutau
  • osu-lazer
  • parabolic
  • patchcil
  • pbm
  • pinta
  • ps3-disc-dumper
  • pupdate
  • recyclarr
  • retrospy
  • roslyn-ls
  • ryubing
  • ryujinx
  • scarab
  • seq-cli
  • skeditor
  • slskd
  • smtp4dev
  • space-station-14-launcher
  • tagger
  • technitium-dns-server
  • technitium-dns-server-library
  • tests.dotnet.final-attrs.check-output
  • tests.dotnet.final-attrs.output-matches-const
  • tests.dotnet.final-attrs.override-has-no-effect
  • tests.dotnet.final-attrs.override-modifies-output
  • tests.dotnet.nuget-deps.derivation
  • tests.dotnet.nuget-deps.json-file
  • tests.dotnet.nuget-deps.list
  • tests.dotnet.nuget-deps.nix-file
  • tests.dotnet.nuget-deps.null
  • tests.dotnet.project-references
  • tests.dotnet.structured-attrs.check-output
  • tests.dotnet.structured-attrs.no-structured-attrs
  • tests.dotnet.use-dotnet-from-env.fallback
  • tests.dotnet.use-dotnet-from-env.use-dotnet-path-env
  • tests.dotnet.use-dotnet-from-env.use-dotnet-root-env
  • tests.dotnet.use-dotnet-from-env.without-fallback
  • tone
  • torrentstream
  • upgrade-assistant
  • v2rayn
  • vrcadvert
  • wiseunpacker
  • xivlauncher
  • yafc-ce

@corngood
Copy link
Contributor

I'd like to get this merged, but I'd prefer to remove:

avalonia-ilspy: use autoPatchcilHook

I don't think it sets a great example, and it's not coming from a maintainer.

@GGG-KILLER
Copy link
Contributor Author

GGG-KILLER commented Jan 29, 2025

avalonia-ilspy: use autoPatchcilHook

I don't think it sets a great example, and it's not coming from a maintainer.

What are your apprehensions regarding it? Why would you say it's not a great example?

Also, speaking of maintainers, @AngryAnt is one of the package's maintainers and didn't seem to have any feedback on the changes in avalonia-ilspy itself.

@corngood
Copy link
Contributor

What are your apprehensions regarding it? Why would you say it's not a great example?

It's way more verbose. Transitive dependencies could require changes to be made in an update.

Keep in mind that the package should work with this change:

diff --git a/pkgs/applications/misc/avalonia-ilspy/default.nix b/pkgs/applications/misc/avalonia-ilspy/default.nix
index 69aeb11e2bbd..b6bedeb8821f 100644
--- a/pkgs/applications/misc/avalonia-ilspy/default.nix
+++ b/pkgs/applications/misc/avalonia-ilspy/default.nix
@@ -4,15 +4,6 @@
   fetchFromGitHub,
   buildDotnetModule,
   dotnetCorePackages,
-  libX11,
-  libICE,
-  libSM,
-  libXi,
-  libXcursor,
-  libXext,
-  libXrandr,
-  fontconfig,
-  glew,
   makeDesktopItem,
   copyDesktopItems,
   icoutils,
@@ -52,25 +43,6 @@ buildDotnetModule rec {
       autoSignDarwinBinariesHook
     ];
 
-  buildInputs = [
-    # Dependencies of nuget packages w/ native binaries
-    (lib.getLib stdenv.cc.cc)
-    fontconfig
-  ];
-
-  runtimeDeps = [
-    # Avalonia UI
-    libX11
-    libICE
-    libSM
-    libXi
-    libXcursor
-    libXext
-    libXrandr
-    fontconfig
-    glew
-  ];
-
   postInstall =
     ''
       icotool --icon -x ILSpy/ILSpy.ico

If I was the maintainer, I wouldn't want this extra burden. At least not without a clear benefit.

@GGG-KILLER
Copy link
Contributor Author

GGG-KILLER commented Jan 29, 2025

What are your apprehensions regarding it? Why would you say it's not a great example?

It's way more verbose. Transitive dependencies could require changes to be made in an update.

Yes, since the transitive dependencies aren't packaged in nixpkgs that's how things work in nixpkgs.

Just look at native programs (or even Java programs) that have transitive dependencies packaged together with them and it'll be exactly the same, autoPatchelfHook modifying transitive dependencies and all.

Keep in mind that the package should work with this change:
[...]

Totally disagree, there's no mention at all of an avalonia package which would be the build/patching for avalonia to work in nixpkgs.
Things don't just magically work in nixpkgs, they need to be patched.

A better example would be (and then avalonia would be gone from deps.json):

diff --git a/pkgs/applications/misc/avalonia-ilspy/default.nix b/pkgs/applications/misc/avalonia-ilspy/default.nix
index 69aeb11e2bbd..b6bedeb8821f 100644
--- a/pkgs/applications/misc/avalonia-ilspy/default.nix
+++ b/pkgs/applications/misc/avalonia-ilspy/default.nix
@@ -4,15 +4,6 @@
   fetchFromGitHub,
   buildDotnetModule,
   dotnetCorePackages,
+  dotnetPackages,
-  libX11,
-  libICE,
-  libSM,
-  libXi,
-  libXcursor,
-  libXext,
-  libXrandr,
-  fontconfig,
-  glew,
   makeDesktopItem,
   copyDesktopItems,
   icoutils,
@@ -52,25 +43,6 @@ buildDotnetModule rec {
       autoSignDarwinBinariesHook
     ];
 
   buildInputs = [
+    dotnetPackages.avalonia
-    # Dependencies of nuget packages w/ native binaries
-    (lib.getLib stdenv.cc.cc)
-    fontconfig
   ];
-
-  runtimeDeps = [
-    # Avalonia UI
-    libX11
-    libICE
-    libSM
-    libXi
-    libXcursor
-    libXext
-    libXrandr
-    fontconfig
-    glew
-  ];
-
   postInstall =
     ''
       icotool --icon -x ILSpy/ILSpy.ico

But we aren't there yet so this is needed.

If I was the maintainer, I wouldn't want this extra burden. At least not without a clear benefit.

The maintainer hasn't voiced against it so I don't see any problems.

@corngood
Copy link
Contributor

I think it's a worse packaging experience than before, even if you ignore overrides. I think we've both made our points on this, and we're just talking in circles now.

The rest of this PR is fine, and I would love to replace runtime deps for avalonia with it, but I still think overrides is the place to do it.

@GGG-KILLER
Copy link
Contributor Author

GGG-KILLER commented Jan 29, 2025

I think it's a worse packaging experience than before, even if you ignore overrides. I think we've both made our points on this, and we're just talking in circles now.

It's not a worse packaging experiece, it just removes the magic that shouldn't be there to start with (as nothing in nixpkgs does the same nor should).
You simply want a commodity that doesn't nor should exist in nixpkgs. If people want easier packaging of avalonia apps they should package avalonia too and then add it as a dependency. Magic working behind the scenes by buildDotnetModule is entirely wrong both in principle and practice.

The rest of this PR is fine, and I would love to replace runtime deps for avalonia with it, but I still think overrides is the place to do it.

Okay, I'll just bite the bullet since overrides will be nuked after the nuget deps rework PR I'm planning anyways.

@corngood
Copy link
Contributor

It's not a worse packaging experiece, it just removes the magic that shouldn't be there to start with (as nothing in nixpkgs does the same nor should).

I fundamentally disagree with you on this, so I would really like some feedback from package maintainers.

There's a derivation which outputs something like share/nuget/packages/avalonia.x11/[version]/.../Avalonia.X11.dll. That can be e.g.:

  • output of avalonia
  • output of fetchNupkg for 'avalonia.x11'

In either of these cases I think the dll should be patched in store. This makes it quite simple to switch between them if necessary.

I'm not against changing fetchNupkgs to make it less 'magic', but I don't want to lose the ability to abstract it.

Also I've mentioned it above, but if you leave patching until the fixup phase of the leaf package, those assemblies will not work in build/check, etc.

@GGG-KILLER
Copy link
Contributor Author

GGG-KILLER commented Jan 29, 2025

It's not a worse packaging experiece, it just removes the magic that shouldn't be there to start with (as nothing in nixpkgs does the same nor should).

I fundamentally disagree with you on this, so I would really like some feedback from package maintainers.

There's a derivation which outputs something like share/nuget/packages/avalonia.x11/[version]/.../Avalonia.X11.dll. That can be e.g.:

  • output of avalonia
  • output of fetchNupkg for 'avalonia.x11'

Were it option number 1 I'd be perfectly fine with it because it'd be explicit that you're requesting another package from nixpkgs (with its own build script and patches to work in nix world as it is expected whenever you add a dependency to another package) so it having its own modifications is expected.
However the 2nd option doing anything other than just fetching the nuget package goes totally against the concept of fetchers. Imagine you do fetchurl and because it notices that there's a markdown file in the contents it then proceeds to automatically include a markdown to html renderer and converts them to html, that's not expected, it should just fetch the file like you told it to. They don't even extract compressed files, that's done in unpackPhase by mkDerivation.

In either of these cases I think the dll should be patched in store. This makes it quite simple to switch between them if necessary.

No, it should only be done in the first, because the 2nd is a fetcher so it should just do that. It is a fetcher so it should only fetch the package, no extra implicit magic to make things "easier".

I'm not against changing fetchNupkgs to make it less 'magic', but I don't want to lose the ability to abstract it.

There's no "abstracting" to be done here. If you want extra processing then you use something else (like a nixpkgs package that wraps avalonia.x11 to patch it, some setup hook like autoPatchcil, add the libraries to runtimeDeps or have some reusable function that deals with avalonia somehow).
But a fetcher shouldn't do that nor should buildDotnetModule. They should be agnostic and not care about what they're building/fetching, be it a console app, asp.net core app or even a .NET Framework WinForms app.

Also I've mentioned it above, but if you leave patching until the fixup phase of the leaf package, those assemblies will not work in build/check, etc.

Yes, however it isn't needed in every case, and if users need to do patching beforehand, they can do so in the postConfigurePhase or somewhere similar (and if they aren't, then we should fix buildDotnetModule so they're able to do so, or do that thing I mentioned similar to mkYarnDeps so they can add a fixupPhase to it).
Eagerly patching everything might end up with us wasting build time patching things that might not even be required, let's say a nuget package comes with a tool or test-only dependency, we'd be wasting our time and generating warnings for something that isn't even used. And I know in this case you'd say that the maintainer shouldn't have to worry about transitive dependencies, but that only applies for things that are already packaged in nixpkgs (which fetchNupkgs doesn't count as, it needs to be an actual mkDerivation or buildDotnetModule derivation). If they're using something like deps.json, then it's equivalent to adding them to the srcs of the mkDerivation, so you have to know about them as they aren't properly packaged.

EDIT:
Also, speaking of Avalonia, it's not some 9 headed hydra that's impossible to know how to package, there's plenty of examples of it being packaged in nixpkgs so it's not like a maintainer would have to figure it out from scratch.
They could just simply look at another package that includes it and copy what they do or we can start packaging commonly used nuget packages already adjusted to work with nix (like you already started with one of the avalonia versions) and they just add it to the buildInputs. I don't see the need for an overrides system when we can already use existing methods that all of nixpkgs uses. It was a good stopgap solution while patchcil didn't exist but now I don't see a need for it anymore.

@corngood
Copy link
Contributor

There's no "abstracting" to be done here.

The abstraction was done in #339953. That concept got approval and upvotes from a bunch of devs. Nobody brought up any concerns.

I think this is getting off-topic for this PR, but it's an important discussion, so we should have it somewhere.

@GGG-KILLER
Copy link
Contributor Author

GGG-KILLER commented Jan 29, 2025

There's no "abstracting" to be done here.

The abstraction was done in #339953. That concept got approval and upvotes from a bunch of devs. Nobody brought up any concerns.

I know, and I was one of the people that approved it. What I meant was that the ideal scenario would be for no abstraction to exist here.
I admit that I approved without properly understanding it nor nix as a whole as I was just a "noob" maintainer and doing one off updates without understanding nixpkgs philosophy and other things properly. I now know enough to realize it is not the ideal way of doing it, but I don't think it was a mistake as we didn't have the tools nor spare effort to properly implement it.

I think this is getting off-topic for this PR, but it's an important discussion, so we should have it somewhere.

I agree, also I apologize if I'm getting too long-winded or ended up sounding rude, I ended up getting a bit heated but mostly because I was frustrated at my own incapability at getting my point across. I think I did a good job on the previous comment so I'll copy it to another issue so we can move this there.

I went ahead and removed the troubling commit, so everything should be fine to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants