XTensa / FreeRTOS support

[ryankurte]

hi there,

do you have any plans to support the ESP32 (xtensa/freertos)? i've been having a bash at porting this but it's, well, a lot over my head.

cheers,

ryan

[wenyongh]

Hi @ryankurte, many thanks for porting WAMR to freertos! Currently we have no plans to port it yet, it would be great if we enable WAMR on freertos. Are there any issues which need our help?

BTW: we saw that your patch creates a new file invokeNative_xtensa.s, but the content is same as invokeNative_arm.s. Could we just use the arm's version? And modify BUILD_XTENSA_SUPPORT in products/freertos/CMakeLists.txt to BUILD_ARM_SUPPORT:

set(BUILD_ARM_SUPPORT YES)

And in vmcore.cmake, modify to:

if (${BUILD_ARM_SUPPORT} STREQUAL "YES")
  set (source_all ${c_source_all} ${VMCORE_LIB_DIR}/invokeNative_arm.s)
elseif (${BUILD_MIPS_SUPPORT} STREQUAL "YES")
  set (source_all ${c_source_all} ${VMCORE_LIB_DIR}/invokeNative_mips.s)
elseif (${BUILD_AS_64BIT_SUPPORT} STREQUAL "YES")
  set (source_all ${c_source_all} ${VMCORE_LIB_DIR}/invokeNative_em64.s)
else ()
  set (source_all ${c_source_all} ${VMCORE_LIB_DIR}/invokeNative_ia32.s)
endif ()

[ryankurte]

hey thanks for the quick response! i don't think anything immediate, i'm stalled on the native assembly at the moment, but i will update this if i have any other questions.

invokeNative_xtensa.s needs to be xtensa assembly rather than arm assembly, i just copied it with the intention of replacing it bit by bit, and i haven't been able to find documentation on function call conventions just yet, though there is some isa documentation to help with the assembly bit.

[wenyongh]

The assembly source code of invokeNative function is small, but it may be complex to implement it in a short time if we are not familiar with the xtensa instruction set. I checked your CMakeLists.txt, it seems that xtensa is 32-bit, so I suggest that you might try using invokeNative_general.c firstly. It works in some 32-bit platforms, like i386, arm. But it can only call native functions with maximum 20 arguments, unlike assembly code which can push a lot of arguments to stack.

The other issue is whether xtensa 64-bit arguments need to be put in address of 8 bytes aligned like arm and mips? If yes, then in wasm_runtime.c, function wasm_runtime_invoke_native, when setting arguments for native function, remember to handle them like arm and mips.

Maybe you can try using invokeNative_general.c, porting other files firstly, and after they work, start to port assembly code.

[wenyongh]

Now we have added BUILD_TARGET setting in latest patch to replace BUILD_AS_64BIT_SUPPORT in the CMakeLists.txt. The BUILD_TARGET value can be "X86_64", "X86_32", "ARM_32", and also "XTENSA_32", which causes vmcore.cmake to build the invokeNative_xtensa.s. So you can just copy CMakeLists.txt to a new version, set BUILD_TARGET to "XTENSA_32" in your CMakeLists.txt, and then implement vmcore_wasm/invokeNative_xtensa.s.

[xwang98]

@ryankurte, in the last commit, the XTENSA arch was already supported for Zephyr. Just wonder if you can complete your patch for freeRTOS on XTENSA and submit it? thanks!

[vshymanskyy]

@ryankurte we had some success running WebAssembly (and even WASI) on ESP32/ESP8266 using wasm3 engine:

You can check out examples here: https://github.com/wasm3/wasm3/tree/master/platforms

[xwang98]

@vshymanskyy, we noticed following statement in the the wasm3 github project
"∼ 15.8x faster than other wasm interpreters (wac, life, wasm-micro-runtime)"

We have done some benchmark test and found the ratio is dramatically different with what we tested. Could you please leave your email address, so we can do some calibration offline? Or contact me by xin.wang@intel.com?

[vshymanskyy]

@xwang98 yes sure, we're also interested in figuring it out. As for me, the number was a little bit unexpected, but that is what CoreMark test spills out.
A more simple test (recursive fib(40) calculation) showed 2x speedup of WAMR over WAC. Wasm3 was far faster. All tests were perforned on an x86-64 machine.
Let's continue discussion over email.

[xwang98]

@vshymanskyy If you leave your email address, we can send you the data by our test.

[xwang98]

@vshymanskyy Could you please some 3rd parties to run your test and verify the data? Because our test result is so different with the wasm3 claimed number, we want to ensure it is correct.

[nickzoic]

I've been taking a wild swing at it too: see https://github.com/nickzoic/wasm-micro-runtime/tree/wamr-for-esp-idf and https://nick.zoic.org/art/web-assembly-on-esp32-with-wasm-wamr/

[xwang98]

@nickzoic it is awesome! It would be great if you can finish the work and contribute it to this project.

[nickzoic]

Cheers ... slowly making progress, I think ...

[igrr]

@nickzoic Please let me know if you need some help with the Xtensa port, can assist.

Regarding porting to FreeRTOS, my 2c: it might not be required to support ESP32, since ESP-IDF provides a pthread compatibility layer. So you may try using platform/linux/bh_thread.c as a starting point.

[nickzoic]

G'day Ivan! Thanks for joining in!

Yeah, I'd started on linking in the freertos API but then noticed the pthreads code from @ryankurte's branch -- I was kicking myself for not remembering pthreads was already in esp-idf! I'll switch over to using that ASAP, it'll be far easier.

I got as far as "hello world" in interpreter mode but have lots more to do ... I'll come back to it soon and if I get stuck I'll post some specific questions here.

I also made a start on getting the JIT to run but since that depends on LLVM I've had to pull in the espressif Xtensa LLVM branch, which is okay, but I'm a bit confused by how to build LLVM libraries targeting Xtensa and also compiled for in Xtensa. I think this is just a CMake/make issue, I've got to convince it to compile LLVM using the xtensa gcc etc. As it is, it builds the llvm libraries but won't link them because they're the wrong architecture. I'm not sure though, I'm pretty new to CMake so there's a lot of fumbling around in the dark. I'll add some specifics to this issue soon.

Cheers, thanks again!

[wenyongh]

Great, @nickzoic, @igrr! It should be easier to use the pthread API, actually if you don't want to enable the app framework to dynamically install wasm app from host to board to run, we can just simply implement the API's in bh_thread.c, e.g. return 0 or keep empty, since product-mini only runs in a single thread currently. If app framework is enabled, e.g. simple/littlevgl/gui in samples folder, API's in bh_thread.c should be fully implemented.

For the AOT and JIT, we haven't enabled the Xtensa target support yet. What is the "espressif Xtensa LLVM branch"? Does LLVM support Xtensa? We may have a test and try to support it for AOT and JIT.

For JIT, we need to compile LLVM library and link it to iwasm binary, which is very large (50M+ in Linux). I am not sure could you run the large binary in ESP32?

[nickzoic]

For JIT, we need to compile LLVM library and link it to iwasm binary, which is very large (50M+ in Linux). I am not sure could you run the large binary in ESP32?

Yeah i was hoping there'd be a subset of LLVM we could build, just the stuff necessary for JIT. Otherwise, yes, JIT via LLVM isn't going to be a good fit for ESP32 ...

[xwang98]

@nickzoic The WAMR AoT is particularly designed for embedded environment. You can compile the wasm bytecode ahead of time and then load the AoTed module to the target device. The WAMR AoT loader doesn't rely on any system loader by the operating system (given no such support in the embedded systems)

[nickzoic]

@wenyongh see https://github.com/espressif/llvm-project for LLVM fork with Xtensa support.

[wenyongh]

@nickzoic, got it, thanks a lot!

[xwang98]

@vshymanskyy the last patch has brought some fine tune for pereformance and improved wamr 2X on coremark benchmark. you probably need to update your data.

[wenyongh]

@vshymanskyy WAMR has implemented the AOT feature, could you please also add WAMR AOT's result to your data?

[vshymanskyy]

Thanks. Sure, I'll try to build it and update the table. It would be nice to see AOT/JIT compilation directly on the embedded device, otherwise direct comparison doesn't make any sense.

…

On Wed, 12 Feb 2020, 03:22 wenyongh, ***@***.***> wrote: @vshymanskyy <https://github.com/vshymanskyy> WAMR has implemented the AOT feature, could you please also add WAMR AOT's result to your data? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#134?email_source=notifications&email_token=AALP3FBTLVKZR7IGJT2A3EDRCNFMHA5CNFSM4JGFDC6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELO6TNI#issuecomment-584968629>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALP3FFAWWL75HIIWDTQPILRCNFMHANCNFSM4JGFDC6A> .