Draft: Support dynamic loading of PHP extensions via dlopen

[adamziel]

What is this PR doing?

Explores support for dynamic loading of PHP extensions.

Related to #314, #655, #89

Minimal dlopen example without PHP involved.

Testing instructions

Clone this branch to a directory with path /Users/cloudnik/www/Automattic/core/plugins/playground-2/wordpress-playground. It's not great, I know. I want to remove the explicit path dependency soon.

Then, create a /usr/local/etc/php.ini file with the following contents:

allow_url_fopen=1
extension=/Users/cloudnik/www/Automattic/core/plugins/playground-2/wordpress-playground/packages/php-wasm/node/public/mbstring.so
zend_extension=/Users/cloudnik/www/Automattic/core/plugins/playground-2/wordpress-playground/.libs/xdebug.so
xdebug.mode=debug
sys_temp_dir=/tmp

Finally, run:

$ nx reset; PHP=8.2 nx start php-wasm-cli -z ./.libs/xdebug.so ./test.php

Here's what you should see:

Remaining work

• Make this work with any disk path, not just the one listed above
• Clean up the code
• Make the code modular so that PHP extensions may be loaded via a configuration option
• Patch the final php.js files as mentioned in the comments below
• Ship the following extensions and their required libraries as .so files:
  • mbstring
  • gd
  • iconv
  • xdebug (optionally)
• Remove unused exports from the final php.js file (they inflate the code from 200KB to 5MB)
• Provide a way of requesting specific PHP extension on the web and in Node.js

Follow-up work

• Remove unused symbols from shared libraries (like libpng)
• Explore a Make-based build system to avoid repeated rebuilding of the same libraries

[adamziel]

The Failed loading error message comes from here:

https://github.com/php/php-src/blob/d24f07cbfe08585b16ca6e7f200798f5063d4626/Zend/zend_extensions.c#L33-L36

Which means that DL_LOAD itself (is PHP using a macro?) returned NULL.

This could be an Asyncify issue. I'll try allowlisting zend_load_extension and other related functions.

Edit: This is absolutely an Asyncify issue. Here's how __dlopen_js is implemented:

var __dlopen_js = function(handle) {
 return Asyncify.handleSleep((wakeUp => {
  dlopenInternal(handle, {
   loadAsync: true0
  }).then(wakeUp).catch((() => wakeUp(0)));
 }));
};

Edit: Turns out there was another error waiting down the call stack:

Error: need the dylink section to be first
    at failIf (/playground/dist/packages/php-wasm/node/index.cjs:25010:15)
    at getDylinkMetadata (/playground/dist/packages/php-wasm/node/index.cjs:25026:7)
    at loadWebAssemblyModule (/playground/dist/packages/php-wasm/node/index.cjs:25351:20)
    at /playground/dist/packages/php-wasm/node/index.cjs:25625:30

Something must be wrong with the xdebug binary.

Edit: I built xdebug with -sSIDE_MODULE (by hacking the Makefile) and replaced the __dlopen_js definition with:

var __dlopen_js = function(handle) {
  var jsflags = { loadAsync: false }
  return dlopenInternal(handle, jsflags);
};
__dlopen_js.isAsync = false;

Now I'm seeing some more specific errors:

bad export type for 'compiler_globals': undefined
bad export type for 'zend_write': undefined
bad export type for 'executor_globals': undefined
bad export type for 'zend_post_startup_cb': undefined
bad export type for 'OnUpdateString': undefined
bad export type for 'OnUpdateBool': undefined
bad export type for 'OnUpdateLong': undefined
bad export type for 'zend_ini_boolean_displayer_cb': undefined
bad export type for 'zend_pass_function': undefined
bad export type for 'zend_error_cb': undefined
bad export type for 'zend_execute_ex': undefined
bad export type for 'zend_ce_error': undefined
bad export type for 'module_registry': undefined
bad export type for 'zend_execute_internal': undefined
bad export type for 'zend_compile_file': undefined
bad export type for 'core_globals': undefined
bad export type for 'zend_throw_exception_hook': undefined
bad export type for 'zend_ce_closure': undefined
bad export type for 'zend_empty_string': undefined
bad export type for 'sapi_module': undefined
bad export type for 'php_ini_opened_path': undefined
bad export type for 'php_ini_scanned_path': undefined
bad export type for 'php_ini_scanned_files': undefined
bad export type for 'zend_string_init_interned': undefined
bad export type for 'zend_sort': undefined
bad export type for 'zend_extensions': undefined
bad export type for 'sapi_globals': undefined
bad export type for '__THREW__': undefined
bad export type for 'zend_eval_string': undefined
bad export type for '__threwValue': undefined
bad export type for 'zend_get_exception_base': undefined
bad export type for 'zend_known_strings': undefined
bad export type for 'zend_read_property_ex': undefined
bad export type for 'zval_get_string_func': undefined
bad export type for 'zend_clear_exception': undefined
bad export type for 'zend_ce_throwable': undefined
bad export type for 'gc_collect_cycles': undefined
bad export type for 'emscripten_console_log': undefined
bad export type for 'emscripten_console_error': undefined
bad export type for 'emscripten_console_warn': undefined
bad export type for 'emscripten_out': undefined
bad export type for 'emscripten_err': undefined
bad export type for 'emscripten_dbg': undefined

Most likely these symbols need to be listed in EXPORTED_FUNCTIONS when linking the PHP binary.

Edit: I added all those exports:

"_compiler_globals", \n\
"_zend_write", \n\
"_executor_globals", \n\
"_zend_post_startup_cb", \n\
"_OnUpdateString", \n\
"_OnUpdateBool", \n\
"_OnUpdateLong", \n\
"_zend_ini_boolean_displayer_cb", \n\
"_zend_pass_function", \n\
"_zend_error_cb", \n\
"_zend_execute_ex", \n\
"_zend_ce_error", \n\
"_module_registry", \n\
"_zend_execute_internal", \n\
"_zend_compile_file", \n\
"_core_globals", \n\
"_zend_throw_exception_hook", \n\
"_zend_ce_closure", \n\
"_zend_empty_string", \n\
"_sapi_module", \n\
"_php_ini_opened_path", \n\
"_php_ini_scanned_path", \n\
"_php_ini_scanned_files", \n\
"_zend_string_init_interned", \n\
"_zend_sort", \n\
"_zend_extensions", \n\
"_sapi_globals", \n\
"___THREW__", \n\
"_zend_eval_string", \n\
"___threwValue", \n\
"_zend_get_exception_base", \n\
"_zend_known_strings", \n\
"_zend_read_property_ex", \n\
"_zval_get_string_func", \n\
"_zend_clear_exception", \n\
"_zend_ce_throwable", \n\
"_gc_collect_cycles", \n\
"_emscripten_console_log", \n\
"_emscripten_console_error", \n\
"_emscripten_console_warn", \n\
"_emscripten_out", \n\
"_emscripten_err", \n\
"_emscripten_dbg", \n\

And also included some additional definitions in php_wasm.c (likely can be removed in some way):

EMSCRIPTEN_KEEPALIVE void emscripten_console_log(char *a){}
EMSCRIPTEN_KEEPALIVE void emscripten_console_warn(char *a){}
EMSCRIPTEN_KEEPALIVE void emscripten_console_error(char *a){}
EMSCRIPTEN_KEEPALIVE void emscripten_out(char *a){}
EMSCRIPTEN_KEEPALIVE void emscripten_err(char *a){}
EMSCRIPTEN_KEEPALIVE void emscripten_dbg(char *a){}

Now it no longer complains about missing exports, but still fails when trying to call emscripten_builtin_malloc as its missing from wasmImports. I made the following changes:

  function resolveSymbol(sym) {
   var resolved = resolveGlobalSymbol(sym).sym;
   if (!resolved && localScope) {
    resolved = localScope[sym];
   }
   if (!resolved) {
    resolved = moduleExports[sym];
   }
+    if (!resolved && sym === 'emscripten_builtin_malloc') {
+      resolved = _malloc;
+    }
+    if (!resolved && sym === '_emscripten_get_progname') {
+      resolved = function () { return 0; };
+    }
   return resolved;
  }

And what whaaat, I'm seeing this message now:

Xdebug requires Zend Engine API version 420220829.
The Zend Engine API version 320190902 which is installed, is outdated.

Edit: I built PHP 8.2 to fix the API version mismatch. Then I couldn't get it to load at all. Then I commented out this line in php-wasm/cli/src/main.ts:

if (!hasMinusCOption) {
+	// args.unshift('-c', defaultPhpIniPath);
}

And got to another symbolResolution error, this time for zend_startup_module. I added that to the EXPORTED_FUNCTIONS list and now it complains about calloc. Finding these one by one would take forever so instead here's the list of all wasm imports from xdebug.so obtained via wasm2wat .libs/xdebug.so | grep import | sort | grep -v '\00.

gist

Edit: I used -sMAIN_MODULE instead of -sMAIN_MODULE=2 to export all symbols. Guess what? Loading xdebug worked!

The downside is that exporting all symbols increased the size of php_8_2.js from 194kb to 5MB! The .wasm module is still 10MB, though. For the web version, we'll need to find a way of removing most of these unused exports.

[adamziel]

I was able to build shared .so libraries for core PHP extensions with --with-libxml=shared.

However, that seemed to also have enabled them by default when starting PHP and now it complains about undefined xml symbols. Perhaps building PHP twice is needed here? One time to generate these shared libraries, and another time to build a version that doesn't try to load them unless explicitly configured via php.ini?

Edit: It worked for mbstring!

php.ini:

extension=/Users/cloudnik/www/Automattic/core/plugins/playground/packages/php-wasm/node/public/mbstring.so

File.php:

var_dump(mb_convert_case('zażółć gęślą jaźń', MB_CASE_UPPER, 'UTF-8'));

However, it didn't work for gd.so:

Warning: PHP Startup: Unable to load dynamic library 'gd.so' (tried: gd.so (Could not load dynamic lib: gd.so
Error: bad export type for 'png_sig_cmp': undefined

Sounds like we'd need to load libpng.so first. That's progress! And also we're getting into a territory of managing a repository of libraries, figuring out dependency graphs etc. Surely there must be some prior art for this?

Edit: I patched loadWebAssemblyModule in php.js as follows and got the gd extension to work! 🎉

+      if (!global.hookedIntoWasmLoading) {
+        global.hookedIntoWasmLoading = true;
+          metadata.neededDynlibs.push(
+            'libpng16.so'
+          );
+          console.log(metadata.neededDynlibs);
+      }
  
      metadata.neededDynlibs.forEach((needed) => loadDynamicLibrary(needed, flags, localScope));
      return loadModule();
    }

Edit: The libxml extension doesn't seem to support building to a shared library. Interestingly, there's no such restriction on windows builds. We may need to figure out how ubuntu php-xml packages are built and use a custom build config to reflect that. Until then, perhaps we could ship libxml by default.

cc @seanmorris

[adamziel]

I just force-pushed. The previous HEAD was c24c153

[adamziel]

Something with git history got very wrong here and I'm not investing more time into untangling it. Let's close this PR and continue in #831