Staging pipeline should sign binaries in Mac archives

[mmitche]

The mac .pkg files have signed binaries in them today (and the pkg files are notarized). But we stopped short of repacking the osx archives with those binaries. This should be resolved, for 3.1+

Epic completion check boxes:

Walk scenario:

• Location for Apple signing process is plainly linked and summarized in documentation near our signing automation.
• Signing capabilities of dotnet-release repository include the ability to unpack, notarize-sign, and re-pack tar.gz and .pkg file formats

Current status: This PR should address both of these, I am currently continuing to add to the readme to be stored in the eng folder adjacent to the signing scripts.

Run scenario: (pending feasibility)

• [-] SignTool package is capable of loading on an OSX agent and performing the tasks of the signing python scripts mentioned above (cut - Microbuild signing tasks are much slower than ESRP for mac signing /notarization and provide less feedback, so depending on Microbuild to do this indirectly adds hours to the process.
• [-] OSX notary-signing python scripts stored in dotnet/arcade and automatically flowed via existing dependency-flow channels (cut - don't see demand)

[mmitche]

I took a look at this to determine what should be done here. There are two potential places to put this code:

• In the mac notarization code that exists at https://dev.azure.com/dnceng/internal/_git/dotnet-release?path=%2Feng%2Fpipeline%2Ftools%2Fosx_pkg_operations.py
• In signtool

I believe signtool is the right long-term location for all signing and notarization. However, right now this still needs to work with 2.1, 3.1, and 5.0 and there are things to work out if we wanted to do that:

1. Signtool runs on Windows and invokes ESRP via Microbuild. The tooling necessary for pkg notarization and some signing isn't available on Windows. Adding the entitlements, for instance, requires the codesign tool.
2. 2.1, 3.1 and 5.0's staging pipelines don't use signtool, though we could alter them to invoke it, passing in a constructed signing manifest file that includes mac assets.

Based on this, it might be best to leave the mac signing in osx_pkg_operations.py for now, and augment it to unpack and deal with the tar.gz archives. It would be best if this was done by a person who has a mac though, as it's rather difficult to alter the code without being able to run it.

/cc @richlander

[mmitche]

/cc @markwilkie

[markwilkie]

Which epic do you think this belong in @mmitche ?

[MattGal]

[Async Triage] I got nothin'

[mmitche]

Which epic do you think this belong in @mmitche ?

Probably its own? It bigger than a one off item, though not huge.

[jakubstilec]

[Async Triage] How urgent is this? What would happen if we didn't implement it soon?

[mmitche]

I think it's blocking some scenarios right now (Offinline install of Education Bundle) /cc @mavasani

[MattGal]

[Async Triage] if one or more user users can confirm blocked-ness, this is within scope of FR. (@mavasani ?)

[riarenas]

Probably its own? It bigger than a one off item, though not huge.

This makes me think it's probably not a great fit for FR. It seems like it might fit the urgency, but not the scope.

[richlander]

I consider this issue "blocking". Obviously, it depends on what blocking truly means, but "unreasonable pain" doesn't really cover it. The lack of signing is blocking certain important scenarios.

[markwilkie]

@mmitche - what's the cost here do you think?

[mmitche]

Here's what I wrote in another mail:

I think the following needs to happen for this to be done completely:

1. Affects 6.0 only - The staging pipeline needs to be altered so that it signs and notarizes after post-build signing runs. This means that 6.0 will not regress signing of other files in the mac zip/tar.gz files. Today the zips are getting signed (the MSIL binaries)
2. Affects 6.0 only - Post build signing needs to handle .tar.gz files. Otherwise the zip and tar.gz mac files will have different signed MSIL binaries
3. Affects all releases – Alter the mac notarization script to handle the appropriate .tar.gz files and zip files in addition to the .pkg files.

I think #1 is < 1 week (rollout not included), #2 is a few days (it’s just adding a new archive type to handle), and #3 is probably ~2 weeks, maybe a little more. Conservatively, we should add on some time for unknowns there. The mac notarization script is the hardest part. It’s centered around pkg files right now. It also needs to be done by someone with access to a mac, since it only runs on a mac.

I’d say it’s about a month of work, calendar time. All 3 aspects can be parallelized though.

[mmitche]

Temp workarounds:

You might be able to manually deconstruct a signed pkg (or maybe just install the product?), pull out the files you need and replace them in the .tar.gz and .zip files. Seems possible to hack up at least. Definitely fraught though, as you would want to ensure that those altered archives get replaced everywhere.

I don’t have a Mac to test this out (a Mac is required for all these signing operations), so this is largely off the top of my head.

[richlander]

or maybe just install the product?

The problem is that you cannot. .NET 5 x64 and .NET 6 Arm64 cannot co-exist on an Apple Silicon machine, installed globally with our installers. That's why this is a blocking issue. I have to sudo rm -rf /usr/local/share/dotnet before I install .NET 6 after installing .NET 5 and vice versa.

On the deconstruction idea, are the files within the pkg signed, or just the pkg?

[mmitche]

@richlander I meant to install it as a one-off on an isolated machine to extract the binaries, the update them elsewhere.

I seems like there are multiple issues around the mac signing that are blocking different scenarios? The ones I've heard primarily about are:

• The hosts and other native binaries no longer work on apple silicon without notarization. It used to be that this was fine as long as:
  • You installed via a PKG, which was signed OR
  • You unzipped/untarred the archive, which didn't have issues.
• Other folks are unzipping the archives and repackaging the binaries into other archives/installers. Those are unsigned, and causing issues.

[jakubstilec]

[Async Triage]: Is it worth to create a mini epic for this?

[mmitche]

Yep I think so.

[jakubstilec]

[Async Triage]: Converted to epic per @mmitche, FYI @markwilkie.

[MattGal]

Moved remaining issue to future work, closing this epic.