Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libgcrypt: bump to v1.11. #9680

Merged
merged 1 commit into from
Oct 25, 2024
Merged

libgcrypt: bump to v1.11. #9680

merged 1 commit into from
Oct 25, 2024

Conversation

maleadt
Copy link
Contributor

@maleadt maleadt commented Oct 24, 2024

No description provided.

@maleadt
Copy link
Contributor Author

maleadt commented Oct 25, 2024

Well, this is weird. Locally, julia +1.7 --project build_tarballs.jl --verbose --debug i686-w64-mingw32 passes without any build error. And for FreeBSD, why doesn't it pick up the v1.50 dependency?

@maleadt
Copy link
Contributor Author

maleadt commented Oct 25, 2024

And for FreeBSD, why doesn't it pick up the v1.50 dependency?

Fixed by forcing v1.50, although I'm still confused why it wouldn't use this with a compat of v1.49...

Locally, julia +1.7 --project build_tarballs.jl --verbose --debug i686-w64-mingw32 passes without any build error.

I wonder if part of the host environment is leaking into the build, making it end up building different sources, or using different options? The code is hidden behind a feature flag: https://github.com/gpg/libgcrypt/blob/4876a1a45c25d7e3ffb45a0af4e7c8995ce3a552/random/jitterentropy-base-user.h#L81-L98
The config output is identical.

EDIT: so locally, the build does reach that branch, which means CLOCK_REALTIME somehow compiles just fine locally, but not on Yggy? I'm not sure how that can happen; my Manifest is up to date...

@maleadt maleadt force-pushed the tb/libgcrypt branch 3 times, most recently from c3d2374 to e871619 Compare October 25, 2024 12:30
@maleadt
Copy link
Contributor Author

maleadt commented Oct 25, 2024

MWE:

#include <stdint.h>
typedef uint64_t u64;

#include <time.h>

static void
jent_get_nstime(u64 *out)
{
  struct timespec tv;
  u64 tmp;

  /* On Linux we could use CLOCK_MONOTONIC(_RAW), but with
   * CLOCK_REALTIME we get some nice extra entropy once in a while
   * from the NTP actions that we want to use as well... though, we do
   * not rely on that extra little entropy.  */
  if (!clock_gettime (CLOCK_REALTIME, &tv))
    {
      tmp = tv.tv_sec;
      tmp = tmp << 32;
      tmp = tmp | tv.tv_nsec;
    }
  else
    tmp = 0;
  *out = tmp;
}

With Yggdrasil cloned on amdci7 and instantiating the CI Manifest, I get:

# cc -c test.c -o test.o
test.c: In function ‘jent_get_nstime’:
test.c:16:23: error: ‘CLOCK_REALTIME’ undeclared (first use in this function)
   if (!clock_gettime (CLOCK_REALTIME, &tv))
                       ^
test.c:16:23: note: each undeclared identifier is reported only once for each function it appears in

On my local system, with otherwise the exact same set-up, everything works fine...

Some part of the set-up isn't identical though, but I'm not sure how or why. Locally:

sandbox:${WORKSPACE}/srcdir/libgcrypt-1.11.0/random # grep CLOCK_REALTIME /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include/time.h
#define CLOCK_REALTIME           0
#define CLOCK_REALTIME_COARSE    5
#define CLOCK_REALTIME_ALARM     8

On Yggdrasil:

sandbox:${WORKSPACE}/srcdir/libgcrypt-1.11.0/random # grep CLOCK_REALTIME /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include/time.h
grep: /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include/time.h: No such file or directory

@maleadt
Copy link
Contributor Author

maleadt commented Oct 25, 2024

It even reproduces with a fresh depot... Here's the verbose logs of both invocations, one on my local system, one on amdci7: https://gist.github.com/maleadt/e5764478bf01b161842ee1e0bc0edea9

@giordano I'm out of ideas here, do you have any? The logs look mostly identical.


Despite starting with a fresh depot, one time.h header mismatches again...

sandbox:${WORKSPACE}/srcdir/libgcrypt-1.11.0/random # find / -name time.h -exec md5sum {} \; | sort
find: ‘/proc/tty/driver’: Permission denied
29e1e232fdf102bf5c56856c8a066ed8  /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/include/time.h

sandbox:${WORKSPACE}/srcdir/libgcrypt-1.11.0/random # gcc -c test.c
test.c: In function ‘jent_get_nstime’:
test.c:16:23: error: ‘CLOCK_REALTIME’ undeclared (first use in this function)
   if (!clock_gettime (CLOCK_REALTIME, &tv))
                       ^
test.c:16:23: note: each undeclared identifier is reported only once for each function it appears in
sandbox:${WORKSPACE}/srcdir/libgcrypt-1.11.0/random #
sandbox:${WORKSPACE}/srcdir/libgcrypt-1.11.0/random # find / -name time.h -exec md5sum {} \; | sort
find: ‘/proc/tty/driver’: Permission denied
25b6994f6b4e75f2ce9582f5ac81af78  /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/include/time.h

sandbox:${WORKSPACE}/srcdir/libgcrypt-1.11.0/random # gcc -c test.c
sandbox:${WORKSPACE}/srcdir/libgcrypt-1.11.0/random #

That specific header doesn't provide CLOCK_REALTIME, but the pthread_time.h one included from there does:

# grep REALTIME /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/include/pthread_time.h
57:#ifndef CLOCK_REALTIME
58:#define CLOCK_REALTIME              0

vs. on the 'bad' system:

# grep REALTIME /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/include/pthread_time.h
# <crickets...>

Another breadcrumb: pthread_time.h is provided by two artifacts, one providing CLOCK_REALTIME, the other not:

% find /depot/artifacts/ -name pthread_time.h
/depot/artifacts/8bf751a4dcefa5ad453409fad19cd15dddaeae3f/x86_64-w64-mingw32/sys-root/include/pthread_time.h
/depot/artifacts/6ec2723fc87baf2e18db4dc5321a635580a64087/x86_64-w64-mingw32/sys-root/include/pthread_time.h

% grep CLOCK_REALTIME /depot/artifacts/8bf751a4dcefa5ad453409fad19cd15dddaeae3f/x86_64-w64-mingw32/sys-root/include/pthread_time.h
#ifndef CLOCK_REALTIME
#define CLOCK_REALTIME              0
#ifndef CLOCK_REALTIME_COARSE
#define CLOCK_REALTIME_COARSE       4

% grep CLOCK_REALTIME /depot/artifacts/6ec2723fc87baf2e18db4dc5321a635580a64087/x86_64-w64-mingw32/sys-root/include/pthread_time.h
%

These two artifacts map onto the PlatformSupport-2024.8.10 and GCCBootstrap-4.8.5 artifacts for x86_64-w64-mingw32. Could it be that there's a conflicting header here?


Aha, something corroborating that: In the log I posted above, you can see how both systems map the artifacts in a different order:

--> Mounting overlay of /depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-w64-mingw32/PlatformSupport-2024.8.10:/depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-w64-mingw32/GCCBootstrap-4.8.5 at /depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-w64-mingw32 (modifications in /proc/upper/x86_64-w64-mingw32, workspace in /proc/work/x86_64-w64-mingw32)
--> Mounting overlay of /depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-linux-musl/LLVMBootstrap-18.1.7:/depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-linux-musl/PlatformSupport-2024.8.10:/depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-linux-musl/GCCBootstrap-4.8.5 at /depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-linux-musl (modifications in /proc/upper/x86_64-linux-musl, workspace in /proc/work/x86_64-linux-musl)

vs

--> Mounting overlay of /depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-linux-musl/GCCBootstrap-4.8.5:/depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-linux-musl/PlatformSupport-2024.8.10:/depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-linux-musl/LLVMBootstrap-18.1.7 at /depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-linux-musl (modifications in /proc/upper/x86_64-linux-musl, workspace in /proc/work/x86_64-linux-musl)
--> Mounting overlay of /depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-w64-mingw32/GCCBootstrap-4.8.5:/depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-w64-mingw32/PlatformSupport-2024.8.10 at /depot/artifacts/73b10e7f9f6b94c9754056030a3adbcf64bf5b90/opt/x86_64-w64-mingw32 (modifications in /proc/upper/x86_64-w64-mingw32, workspace in /proc/work/x86_64-w64-mingw32)

Notice how GCCBootstrap is in different places of the overlay. I guess this is a nondeterminism in how the sandbox processes options, but presumably there shouldn't be an overlapping header in those?

@giordano
Copy link
Member

giordano commented Oct 25, 2024

% find build/i686-w64-mingw32/*/.mounts -name time.h -exec md5sum {} \; 2> /dev/null | sort                 
14a0dab31a3dea92cbbe2fd9a40c9072  build/i686-w64-mingw32/yNJdRiPh/.mounts/GCCBootstrap-x86_64-linux-musl.v4.8.5.x86_64-linux-musl.squashfs/x86_64-linux-musl/sys-root/usr/include/time.h
25b6994f6b4e75f2ce9582f5ac81af78  build/i686-w64-mingw32/yNJdRiPh/.mounts/GCCBootstrap-i686-w64-mingw32.v4.8.5.x86_64-linux-musl.squashfs/i686-w64-mingw32/sys-root/include/time.h
29e1e232fdf102bf5c56856c8a066ed8  build/i686-w64-mingw32/yNJdRiPh/.mounts/PlatformSupport-i686-w64-mingw32.v2024.8.10.x86_64-linux-musl.squashfs/i686-w64-mingw32/sys-root/include/time.h
7e6318050651a97ad6076a81e6384cb1  build/i686-w64-mingw32/yNJdRiPh/.mounts/LLVMBootstrap.v18.1.7.x86_64-linux-musl.squashfs/lib/clang/18/include/llvm_libc_wrappers/time.h
849dabb9216c8f53bbf721c4720a9356  build/i686-w64-mingw32/yNJdRiPh/.mounts/GCCBootstrap-i686-w64-mingw32.v4.8.5.x86_64-linux-musl.squashfs/i686-w64-mingw32/sys-root/include/sys/time.h
849dabb9216c8f53bbf721c4720a9356  build/i686-w64-mingw32/yNJdRiPh/.mounts/PlatformSupport-i686-w64-mingw32.v2024.8.10.x86_64-linux-musl.squashfs/i686-w64-mingw32/sys-root/include/sys/time.h
a8f8234ef9e90aae9be3252e80c1d2b6  build/i686-w64-mingw32/yNJdRiPh/.mounts/GCCBootstrap-x86_64-linux-musl.v4.8.5.x86_64-linux-musl.squashfs/x86_64-linux-musl/sys-root/usr/include/linux/time.h
a8f8234ef9e90aae9be3252e80c1d2b6  build/i686-w64-mingw32/yNJdRiPh/.mounts/PlatformSupport-x86_64-linux-musl.v2024.8.10.x86_64-linux-musl.squashfs/x86_64-linux-musl/sys-root/usr/include/linux/time.h
adeb9c7928c2fb4c364d63f7938472d9  build/i686-w64-mingw32/yNJdRiPh/.mounts/GCCBootstrap-x86_64-linux-musl.v4.8.5.x86_64-linux-musl.squashfs/x86_64-linux-musl/sys-root/usr/include/sys/time.h

that seems to be the problem, yes. Interesting we never ran into this before, also not great the two artifacts step onto each other's toes.

@giordano
Copy link
Member

Perhaps using two wildly different versions of mingw at

ArchiveSource("https://sourceforge.net/projects/mingw-w64/files/mingw-w64/mingw-w64-release/mingw-w64-v11.0.1.tar.bz2",
"3f66bce069ee8bed7439a1a13da7cb91a5e67ea6170f21317ac7f5794625ee10"),
and
ArchiveSource("https://sourceforge.net/projects/mingw-w64/files/mingw-w64/mingw-w64-release/mingw-w64-v7.0.0.tar.bz2",
"aa20dfff3596f08a7f427aab74315a6cb80c2b086b4a107ed35af02f9496b628"),
doesn't help. Also not really sure why we install mingw header files twice.

@maleadt
Copy link
Contributor Author

maleadt commented Oct 25, 2024

Interesting we never ran into this before, also not great the two artifacts step onto each other's toes.

# Windows build fail because `CLOCK_REALTIME` is not declared.
# See `N/Notcurses/bundled/headers/pthread_time.h` for a possible fix.
filter!(!Sys.iswindows, platforms)

Also, FWIW, the --map arguments to sandbox are in the same order both times, so the nondeterminism is in sandbox. I'm not sure how though, as the structures used in the userns sandbox are simple linked lists...

@giordano
Copy link
Member

Ah, the reference to notcurses brought me to #4059 (comment) and #4059 (comment). I'm somewhat convinced that in platformsupports we should at very least match what we do in gccbootstrap, or just remove that, since I'm missing why we duplicate the work (and do it wrong).

I wonder if @staticfloat would have some opinions about this 😇

@maleadt maleadt merged commit 65c21c0 into master Oct 25, 2024
20 checks passed
@maleadt maleadt deleted the tb/libgcrypt branch October 25, 2024 14:32
@staticfloat
Copy link
Member

I wonder if @staticfloat would have some opinions about this 😇

My opinion is that PlatformTools is very ill-defined, and instead we should be directly using the Mingw JLLs where needed, which makes it much more explicit when and where these header files are included. BB2 does this, but progress on that front continues, despite its slowness.

avik-pal pushed a commit to avik-pal/Yggdrasil that referenced this pull request Oct 25, 2024
jmert added a commit to jmert/Yggdrasil that referenced this pull request Nov 23, 2024
This takes inspiration from JuliaPackaging#9680 to raise the compatibility bound of
the Libgpg_error_jll dependency to include a version that has the
necessary FreeBSD support
giordano pushed a commit that referenced this pull request Nov 26, 2024
* [XSLT] Update dependency for freebsd-aarch64 support

This takes inspiration from #9680 to raise the compatibility bound of
the Libgpg_error_jll dependency to include a version that has the
necessary FreeBSD support

* [XSLT] Update to v1.1.42

* Work around win32 build error by requesting older build version of XML2

* Revert "Work around win32 build error by requesting older build version of XML2"

This reverts commit 36f9939.

* Change to requiring GCC 11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants