Tracking issue for i686 msvc CI issues with latest Windows 2022 runner image 20250224.5.0+

[jieyouxu]

This is a tracking issue for CI compilation errors related to missing SSE2 types encountered when trying to use latest Windows 2022 runner image 20250224.5.0+ on the i686-msvc-{1,2} and dist-i686-msvc jobs. Key difference for the image version is the Windows 11 SDK version bump from 10.0.22621.0 -> 10.0.26100.0 that modified UCRT headers (see below).

Context

Zulip thread: https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra/topic/spurious.20.28.3F.29.20i686.20msvc.20errors.

cc <= 1.2.15 emits /arch:IA32 for clang-cl on x86 arch Windows targets, which in conjunction with GitHub Runner Image versions Windows 2022 20250224.5.0+ seems to cause a bunch of failures related to compilation errors failing to using SSE2 types without enabling SSE2 via /arch:SSE2. msvc seems to silently accept this¹ and happily emit SSE2 instruments but clang-cl will error.

As Fulgen pointed out (thanks!) on the zulip thread:

The UCRT headers are, which is where they differ (10.0.22621.0/ucrt/wchar.h doesn't have the vectorization, 10.0.26100.0/ucrt/wchar.h does)

Current mitigation attempts

• On the cc side: clang-cl: use /arch:SSE2 for x86 target arch cc-rs#1425
• (In progress): Use Windows 2019 for 32-bit MSVC CI jobs #137732
  • Not going to work. A ui test failure. Old msvc toolchain has old cdb that will fail debuginfo tests.
• (In progress) use large runners since they haven't been updated yet: Fix 32-bit MSVC CI #137749
  • To buy time, but going to break again once github updates large runners to use new image.
• [do not merge] fix 32-bit MSVC CI [alt version] #137754
• Downgrade 32-bit Windows SDK to 10.0.22621.0 in CI #137766
• The large runners were updated already, trying a different approach in Remove latest Windows SDK from 32-bit CI #137753.

Work needed to unblock using latest Windows 2022 runner image 20250224.5.0+ again

Warning

These steps are advisory and likely incomplete, we may find that we need to transitively bump cc versions in more deps.

• Fix cc to not emit /arch:IA32, but instead either omit it or explicitly set /arch:SSE2 for clang-cl on x86: clang-cl: use /arch:SSE2 for x86 target arch cc-rs#1425
  • Release a new patch version of cc: https://github.com/rust-lang/cc-rs/releases/tag/cc-v1.2.16
• Have cargo bump its cc version to the new version: Bump cc to 1.2.16 to fix x86 windows jobs in rust-lang/rust CI cargo#15245
• Sync cargo submodule that contains the bumped cc: Update cargo #137791.
• Bump rustc_{codegen_ssa,llvm} cc to =1.2.16: Bump rustc_{codegen_ssa,llvm} cc to 1.2.16 to fix x86 Windows jobs on newest Windows SDK #137788.
• Revert remove-new-SDK workaround, and confirm that we can successfully run i686-msvc-{1,2} and dist-i686-msvc, including the compiler crates and cargo: Revert "Remove Win SDK 10.0.26100.0 from CI" #137849.

Footnotes

1. https://developercommunity.visualstudio.com/t/wcharh-from-SDK-100261000-cannot-be/10860122 ↩

[Fulgen301]

Fix cc to not emit /arch:IA32, but instead either omit it or explicitly set /arch:SSE2 for clang-cl on x86:

This has been fixed in the meantime, but to have it noted: Before this was fixed, C code with clang-cl was compiled without SSE2 support, whereas Rust code using that C code would be compiled with SSE2 enabled. The default function call ABI is likely fine, since extern "C" passes it all over the stack anyway, but I'm not an opsem person, maybe this could have broken something that nobody ever noticed.

[Epixu]

Apologies for randomly intruding here.
I just encountered a CI failure for the same reasons and found only this relevant thread. In my case it was the cherry on top, because I'm at the brink of abandoning 32bit MSVC builds in their entirety at this point, because they have been bringing me nothing but grief lately.
I was just curious, have you often encountered as many code generation errors, slow build speeds and constexpr-related ICE's associated with MSVC v143 x86 builds in particular?
If so, considering you're still supporting them, am I doing something wrong and is it worth it to invest the time to research your coping mechanisms? Or would it be wiser for me, as a single developer, to just outright abandon these configurations...

[jieyouxu]

I just encountered a CI failure for the same reasons and found only this relevant thread. In my case it was the cherry on top, because I'm at the brink of abandoning 32bit MSVC builds in their entirety at this point, because they have been bringing me nothing but grief lately. I was just curious, have you often encountered as many code generation errors, slow build speeds and constexpr-related ICE's associated with MSVC v143 x86 builds in particular? If so, considering you're still supporting them, am I doing something wrong and is it worth it to invest the time to research your coping mechanisms? Or would it be wiser for me, as a single developer, to just outright abandon these configurations...

Our most recent 32-bit msvc CI issue (this issue) was very specific. Known cause, one of our deps was incorrectly instructing msvc to use /arch:IA32 when we should've been using default or /arch:SSE2. I don't recall running into

as many code generation errors, slow build speeds and constexpr-related ICE's associated with MSVC v143 x86 builds in particular

in our CI recently. Fix for our CI purposes is to just fix the dep to emit the right /arch, which I suspect is not remotely close to what you are experiencing (sorry to hear that).

[jieyouxu]

You will probably have to take that up with the MSVC team, since I assume your CI is exercising C++ code much more heavily?

[Epixu]

Yes, I've already documented about three compiler bugs, one even got fixed recently. But I'm running out of energy to cope with those. I tend to test all possible combinations and it's always the particular configurations that interfere. Like, I literally yesterday excluded SSE/AVX/AVX2 x86 builds due to ICE's and code generation errors. And just now, when I decided it's all figured out, I get this absolutely horrid runner update introducing problems even for clang-cl, which pissed me even more.
Anyways, thanks for the feedback!