Tracking issue: OS X (aarch64) build/test status

[cdleary]

Just an issue to track and take notes on the current (merged to google/xls) status of OS X support. Taking notes in an issue in the google/xls repo as it's (currently) canonical mainline. This is all work-aroundable, but my intent is to get mainline into a state where it "usually" works on OS X (via fixes and PRs if it happens to break).

• The hermetic LLVM toolchain does not work on OS X
• But if you comment out the registration of LLVM toolchain from the WORKSPACE file, almost everything works with the clang-15 installation that's present on latest OS X
• iverilog build doesn't work, which I think is understandable because we did a lot of hacktastic stuff to make it work on Linux
• current grpc dep (at HEAD) has issues
• AoT compilations fail here:

  xls/xls/jit/aot_compiler.cc

  Line 99 in ec4df61

  "Support AOT feature flag setting for aarch64 compilation.");

$ clang --version
Apple clang version 15.0.0 (clang-1500.3.9.4)
Target: arm64-apple-darwin23.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Note that if you don't comment out the hermetic toolchain registration it gives you an error indicating "you invoked llvm-ar with invalid command line arguments". Aside: I often use //xls/ir:bits_test as a simple canary target for a binary to build/test.

$ bazel build -c opt //xls/ir:bits_test
INFO: Analyzed target //xls/ir:bits_test (1 packages loaded, 2104 targets configured).
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/external/com_google_absl/absl/base/BUILD.bazel:60:11: Linking external/com_google_absl/absl/base/liblog_severity.a [for tool] failed: (Exit 1): llvm-ar failed: error executing command (from target @com_google_absl//absl/base:log_severity) external/llvm_toolchain/bin/llvm-ar @bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_absl/absl/base/liblog_severity.a-2.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/llvm_toolchain/bin/llvm-ar: error: an archive name must be specified
OVERVIEW: LLVM Archiver

USAGE: llvm-ar [options] [-]<operation>[modifiers] [relpos] [count] <archive> [files]
       llvm-ar -M [<mri-script]

OPTIONS:
  --format              - archive format to create
    =default            -   default
    =gnu                -   gnu
    =darwin             -   darwin
    =bsd                -   bsd
    =bigarchive         -   big archive (AIX OS)
  --plugin=<string>     - ignored for compatibility
  -h --help             - display this help and exit
  --output              - the directory to extract archive members to
  --rsp-quoting         - quoting style for response files
    =posix              -   posix
    =windows            -   windows
  --thin                - create a thin archive
  --version             - print the version and exit
  -X{32|64|32_64|any}   - object mode (only for AIX OS)
  @<file>               - read options from <file>

OPERATIONS:
  d - delete [files] from the archive
  m - move [files] in the archive
  p - print contents of [files] found in the archive
  q - quick append [files] to the archive
  r - replace or insert [files] into the archive
  s - act as ranlib
  t - display list of files in archive
  x - extract [files] from the archive

MODIFIERS:
  [a] - put [files] after [relpos]
  [b] - put [files] before [relpos] (same as [i])
  [c] - do not warn if archive had to be created
  [D] - use zero for timestamps and uids/gids (default)
  [h] - display this help and exit
  [i] - put [files] before [relpos] (same as [b])
  [l] - ignored for compatibility
  [L] - add archive's contents
  [N] - use instance [count] of name
  [o] - preserve original dates
  [O] - display member offsets
  [P] - use full names when matching (implied for thin archives)
  [s] - create an archive index (cf. ranlib)
  [S] - do not build a symbol table
  [T] - deprecated, use --thin instead
  [u] - update only [files] newer than archive contents
  [U] - use actual timestamps and uids/gids
  [v] - be verbose about actions taken
  [V] - display the version and exit
Target //xls/ir:bits_test failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.735s, Critical Path: 0.38s
INFO: 9 processes: 9 internal.
FAILED: Build did NOT complete successfully

---

grpc issue:

ERROR: /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/external/com_github_grpc_grpc/src/core/BUILD:2247:16: Compiling src/core/lib/event_engine/cf_engine/cfstream_endpoint.cc [for tool] failed: (Exit 1): wrapped_clang_pp failed: error executing command (from target @com_github_grpc_grpc//src/core:cf_event_engine) external/local_config_cc/wrapped_clang_pp '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG ... (remaining 87 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/com_github_grpc_grpc/src/core/lib/event_engine/cf_engine/cfstream_endpoint.cc:59:15: error: no member named 'StrCat' in namespace 'absl'
        absl::StrCat("getsockname:", grpc_core::StrError(errno)));
        ~~~~~~^

[cdleary]

Failures:

$ bazel test -c opt --keep_going -- //xls/... -//xls/contrib/...  |& tee /tmp/output.txt
[snip]
FAILED: Build did NOT complete successfully
//xls/build_rules/tests:__add_one_combinational_rtl             FAILED TO BUILD
//xls/build_rules/tests:__add_one_dslx_verilog                  FAILED TO BUILD
//xls/build_rules/tests:__add_one_pipeline_rtl                  FAILED TO BUILD
//xls/build_rules/tests:__add_one_pipeline_rtl_proto            FAILED TO BUILD
//xls/build_rules/tests:__add_one_pipeline_rtl_proto_fail       FAILED TO BUILD
(Skipping other failed to build tests)
//xls/interpreter:block_interpreter_test                                 FAILED in 0.3s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/interpreter/block_interpreter_test/test.log
//xls/interpreter:channel_queue_test                                     FAILED in 0.3s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/interpreter/channel_queue_test/test.log
//xls/interpreter:ir_interpreter_test                                    FAILED in 0.3s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/interpreter/ir_interpreter_test/test.log
//xls/interpreter:proc_interpreter_test                                  FAILED in 0.3s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/interpreter/proc_interpreter_test/test.log
//xls/interpreter:serial_proc_runtime_test                               FAILED in 0.9s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/interpreter/serial_proc_runtime_test/test.log
//xls/jit:block_jit_test                                                 FAILED in 1.0s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/jit/block_jit_test/test.log
//xls/jit:jit_channel_queue_test                                         FAILED in 1.2s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/jit/jit_channel_queue_test/test.log
//xls/jit:proc_jit_test                                                  FAILED in 0.9s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/jit/proc_jit_test/test.log
//xls/tools:gather_design_stats_test                                     FAILED in 1.4s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/tools/gather_design_stats_test/test.log
//xls/tools:package_bazel_build_test                                     FAILED in 1.0s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/tools/package_bazel_build_test/test.log
//xls/jit:function_jit_test                                              FAILED in 1 out of 50 in 4.9s
  Stats over 50 runs: max = 4.9s, min = 0.1s, avg = 0.3s, dev = 0.7s
  /private/var/tmp/_bazel_cdleary/60c3e5ce2889de0dfd8db639786fbce7/execroot/com_google_xls/bazel-out/darwin_arm64-opt/testlogs/xls/jit/function_jit_test/shard_31_of_50/test.log

Executed 1248 out of 1451 tests: 1237 tests pass, 203 fail to build and 11 fail locally.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

[proppy]

grpc issue:

looks like abseil/abseil-cpp#1614, should be fixed w/ grpc/grpc@bc04417

should we try bumping grpc to https://github.com/grpc/grpc/releases/tag/v1.64.0 ?

xls/dependency_support/load_external.bzl

Lines 269 to 283 in ec4df61

	# Released on 2023-11-28, current as of 2024-02-01.
	# https://github.com/grpc/grpc/releases/tag/v1.60.0
	# Can be updated to latest once https://github.com/grpc/grpc/issues/36132
	# is fixed and made it into a grpc release.
	# TODO(google/xls#1408) Once updated, fix related TODOs.
	http_archive(
	name = "com_github_grpc_grpc",
	urls = ["https://github.com/grpc/grpc/archive/v1.60.0.tar.gz"],
	integrity = "sha256-Q3BouLd307M52pTTSY8dwgZCrJv6dttDq91SIYaxVCs=",
	strip_prefix = "grpc-1.60.0",
	repo_mapping = {
	"@local_config_python": "@python39",
	"@system_python": "@python39",
	},
	)

[cdleary]

Thanks for the pointer @proppy ! I'll try to do the dependency bump dance in a bit here and see how it goes.

[hzeller]

The current 1.64.0 still has the c++20 regression (but a fix is in at head and they plan to backport it to 1.64).

For us, to update 1.64.0, we can use that release and then cherry-pick the fixing patch and applying it in the load_external.bzl.

Note, when updating grpc, this will also require to update or-tools last time I checked, so it might be a bit involved as that dependency also has not updated in a while.

[cdleary]

Just an update here, I have a release branch with builds for both x86_64 and arm64 on OS X here: https://github.com/xlsynth/xlsynth/releases/tag/v0.0.87 So far it's been easy enough to do the "packaging dance" where I patch up things that happen to go wrong with system toolchains vs hermetic clang/llvm, I always download/install clang-17 on all the platforms and use it as a system level toolchain dependency.