Compiling Godot web export template with LTO fails with Emscripten 3.1.42+

[radlotus]

Godot version

3.5.2

System information

AMD EPYC 7B13; 32 GB RAM
Ubuntu 22.04.2 LTS; Python 3.11.1; Emscripten 3.1.44; clang 17; Scons 4.0.1

Issue description

Compiling with LTO gives this error:

[Initial build] Compiling ==> core/bind/core_bind.cpp
[Initial build] Creating 'bin/godot.javascript.opt.engine.js'
[Initial build] Install file: "misc/dist/html/offline-export.html" as "bin/.javascript_zip/godot.offline.html"
[Initial build] Install file: "misc/dist/html/service-worker.js" as "bin/.javascript_zip/godot.service.worker.js"
[Initial build] progress_finish(["progress_finish"], [])
[Initial build] Linking Static Library ==> core/libcore.javascript.opt.a
Ranlib Library         ==> core/libcore.javascript.opt.a
[Initial build] Linking Program        ==> bin/godot.javascript.opt.js
cache:INFO: generating system asset: symbol_lists/df45337bb9df6876919ba6b0bb4825c9a26cbd2f.json... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/symbol_lists/df45337bb9df6876919ba6b0bb4825c9a26cbd2f.json" for subsequent builds)
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libGL-webgl2-ofb.a... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libGL-webgl2-ofb.a" for subsequent builds)
system_libs:INFO: compiled 4 inputs
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libal.a... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libal.a" for subsequent builds)
system_libs:INFO: compiled 1 inputs
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libhtml5.a... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libhtml5.a" for subsequent builds)
system_libs:INFO: compiled 3 inputs
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libstubs.a... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libstubs.a" for subsequent builds)
system_libs:INFO: compiled 2 inputs
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libc.a... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libc.a" for subsequent builds)
system_libs:INFO: compiled 1023 inputs
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libdlmalloc.a... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libdlmalloc.a" for subsequent builds)
system_libs:INFO: compiled 1 inputs
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libcompiler_rt.a... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libcompiler_rt.a" for subsequent builds)
system_libs:INFO: compiled 175 inputs
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libc++-noexcept.a... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libc++-noexcept.a" for subsequent builds)
system_libs:INFO: compiled 49 inputs
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libc++abi-noexcept.a... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libc++abi-noexcept.a" for subsequent builds)
system_libs:INFO: compiled 16 inputs
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libsockets.a... (this will be cached in "/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libsockets.a" for subsequent builds)
system_libs:INFO: compiled 21 inputs
cache:INFO:  - ok
wasm-ld: error: /workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libc.a(scalbnf.o): attempt to add bitcode file after LTO.
em++: error: '/workspace/web/emsdk/upstream/bin/wasm-ld -o bin/godot.javascript.opt.wasm platform/javascript/audio_driver_javascript.javascript.opt.o platform/javascript/godot_webgl2.javascript.opt.o platform/javascript/http_client_javascript.javascript.opt.o platform/javascript/javascript_singleton.javascript.opt.o platform/javascript/javascript_main.javascript.opt.o platform/javascript/os_javascript.javascript.opt.o platform/javascript/api/javascript_tools_editor_plugin.javascript.opt.o platform/javascript/javascript_runtime.javascript.opt.o main/libmain.javascript.opt.a modules/libmodules.javascript.opt.a platform/libplatform.javascript.opt.a drivers/libdrivers.javascript.opt.a scene/libscene.javascript.opt.a servers/libservers.javascript.opt.a core/libcore.javascript.opt.a modules/freetype/libfreetype_builtin.javascript.opt.a -L/workspace/web/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto -lGL-webgl2-ofb -lal -lhtml5 -lstubs -lc -ldlmalloc -lcompiler_rt -lc++-noexcept -lc++abi-noexcept -lsockets -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr /tmp/tmpk0a0yca9libemscripten_js_symbols.so --strip-debug --export-if-defined=main --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=__main_argc_argv --export-if-defined=fflush --export=stackSave --export=stackRestore --export=stackAlloc --export=__errno_location --export=__get_temp_ret --export=__set_temp_ret --export=__funcs_on_exit --export=__wasm_call_ctors --export-table -z stack-size=65536 --initial-memory=33554432 --no-entry --max-memory=2147483648 --global-base=1024' failed (returned 1)
scons: *** [bin/godot.javascript.opt.js] Error 1
scons: building terminated because of errors

Steps to reproduce

1. Install the requirements listed on https://docs.godotengine.org/en/3.6/development/compiling/compiling_for_web.html.
2. Run scons platform=javascript tools=no target=release use_lto=yes

Does anyone else have this issue?

Is it something related to a newer version of Emscripten? Python? Scons?

I did not have this issue before. I am using a fresh build setup.

[radlotus]

whoops, closed accidentaly.

[akien-mga]

Official builds are made with Emscripten 3.1.18. Newer versions often cause breaking regressions, so this might be the case too. Could you test with 3.1.18 and see if that solves it?

[radlotus]

@akien-mga, yes, using 3.1.18 solves it. I've checked and the latest version that still works is 3.1.36.

[radlotus]

The issue is also present in Godot 4. Last working Emscripten version is also 3.1.36

[Faless]

This seems to be an upstream bug, see:
emscripten-core/emscripten#19774 and emscripten-core/emscripten#19781 .

[akien-mga]

For the record, I can confirm the issue is still reproducible with latest Emscripten 3.1.45.

Tested build with scons p=web target=template_release production=yes verbose=yes.
Fails with:

wasm-ld: error: /home/akien/Projects/godot/emscripten/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libc-mt-debug.a(scalbnf.o): attempt to add bitcode file after LTO.
em++: error: '/home/akien/Projects/godot/emscripten/emsdk/upstream/bin/wasm-ld -o bin/godot.web.template_release.wasm32.wasm platform/web/audio_driver_web.web.template_release.wasm32.o platform/web/display_server_web.web.template_release.wasm32.o platform/web/http_client_web.web.template_release.wasm32.o platform/web/javascript_bridge_singleton.web.template_release.wasm32.o platform/web/web_main.web.template_release.wasm32.o platform/web/os_web.web.template_release.wasm32.o platform/web/api/web_tools_editor_plugin.web.template_release.wasm32.o platform/web/web_runtime.web.template_release.wasm32.o main/libmain.web.template_release.wasm32.a modules/libmodules.web.template_release.wasm32.a modules/libmodule_ktx.web.template_release.wasm32.a modules/libmodule_msdfgen.web.template_release.wasm32.a modules/libmodule_theora.web.template_release.wasm32.a modules/libmodule_vorbis.web.template_release.wasm32.a modules/libmodule_basis_universal.web.template_release.wasm32.a modules/libmodule_bmp.web.template_release.wasm32.a modules/libmodule_csg.web.template_release.wasm32.a modules/libmodule_dds.web.template_release.wasm32.a modules/libmodule_enet.web.template_release.wasm32.a modules/libmodule_freetype.web.template_release.wasm32.a modules/libmodule_gdscript.web.template_release.wasm32.a modules/libmodule_glslang.web.template_release.wasm32.a modules/libmodule_gltf.web.template_release.wasm32.a modules/libmodule_gridmap.web.template_release.wasm32.a modules/libmodule_hdr.web.template_release.wasm32.a modules/libmodule_jpg.web.template_release.wasm32.a modules/libmodule_jsonrpc.web.template_release.wasm32.a modules/libmodule_lightmapper_rd.web.template_release.wasm32.a modules/libmodule_mbedtls.web.template_release.wasm32.a modules/libmodule_meshoptimizer.web.template_release.wasm32.a modules/libmodule_minimp3.web.template_release.wasm32.a modules/libmodule_mobile_vr.web.template_release.wasm32.a modules/libmodule_multiplayer.web.template_release.wasm32.a modules/libmodule_navigation.web.template_release.wasm32.a modules/libmodule_noise.web.template_release.wasm32.a modules/libmodule_ogg.web.template_release.wasm32.a modules/libmodule_raycast.web.template_release.wasm32.a modules/libmodule_regex.web.template_release.wasm32.a modules/libmodule_squish.web.template_release.wasm32.a modules/libmodule_svg.web.template_release.wasm32.a modules/libmodule_text_server_adv.web.template_release.wasm32.a modules/libmodule_tga.web.template_release.wasm32.a modules/libmodule_upnp.web.template_release.wasm32.a modules/libmodule_vhacd.web.template_release.wasm32.a modules/libmodule_webp.web.template_release.wasm32.a modules/libmodule_webrtc.web.template_release.wasm32.a modules/libmodule_websocket.web.template_release.wasm32.a modules/libmodule_webxr.web.template_release.wasm32.a modules/libmodule_zip.web.template_release.wasm32.a platform/libplatform.web.template_release.wasm32.a drivers/libdrivers.web.template_release.wasm32.a scene/libscene.web.template_release.wasm32.a servers/libservers.web.template_release.wasm32.a core/libcore.web.template_release.wasm32.a modules/text_server_adv/libharfbuzz_builtin.web.template_release.wasm32.a modules/text_server_adv/libgraphite_builtin.web.template_release.wasm32.a modules/text_server_adv/libicu_builtin.web.template_release.wasm32.a modules/freetype/libfreetype_builtin.web.template_release.wasm32.a modules/msdfgen/libmsdfgen_builtin.web.template_release.wasm32.a -L/home/akien/Projects/godot/emscripten/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto /home/akien/Projects/godot/emscripten/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/crtbegin.o -lGL-mt-webgl2-ofb -lal -lhtml5 -lbulkmemory -lstubs-debug -lc-mt-debug -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-debug-mt-noexcept -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr /tmp/tmpowj6vflhlibemscripten_js_symbols.so --import-memory --shared-memory --strip-debug --export-if-defined=main --export-if-defined=_emscripten_thread_init --export-if-defined=_emscripten_thread_exit --export-if-defined=_emscripten_thread_crashed --export-if-defined=_emscripten_tls_init --export-if-defined=pthread_self --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=__main_argc_argv --export-if-defined=fflush --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_get_current --export=emscripten_stack_init --export=stackSave --export=stackRestore --export=stackAlloc --export=__errno_location --export=_emscripten_thread_free_data --export=emscripten_main_runtime_thread_id --export=emscripten_main_thread_process_queued_calls --export=_emscripten_run_on_main_thread_js --export=emscripten_stack_set_limits --export=__get_temp_ret --export=__set_temp_ret --export=__funcs_on_exit --export=__wasm_call_ctors --export=_emscripten_thread_init --export=_emscripten_thread_exit --export-table -z stack-size=65536 --initial-memory=33554432 --max-memory=2147483648 --no-entry --stack-first' failed (returned 1)
scons: *** [bin/godot.web.template_release.wasm32.js] Error 1

Full build log:
godot_build_web_lto_emscripten_3.1.45.log.zip

[akien-mga]

The issue is also present in Godot 4. Last working Emscripten version is also 3.1.36

I tried to confirm this and got different results, for me all versions up to 3.1.41 are working fine.
The LTO linking step breaks from 3.1.42 onward, up to 3.1.45 (current latest).
That seems consistent with upstream bug reports which report a regression in 3.1.42.

Did you test specifically that 3.1.37 and later weren't working? Are you sure it was the same error?

[akien-mga]

We can probably try to define -Wl,-u,scalbnf when we detect Emscripten >= 3.1.42, as described in emscripten-core/emscripten#19781 (comment).

I'll have a look at this after finalizing #80719 which refactors the compiler version detection, so this can be done cleanly.