[Docker][CI][RISC-V] Build riscv-isa-sim (spike) in ci_riscv Docker image to enable RISC-V unit testing #12534

PhilippvK · 2022-08-22T10:10:23Z

Context

See Discussion in https://discuss.tvm.apache.org/t/coordination-of-risc-v-integration-in-tvm/13133

Summary

Update Dockerfile.ci_riscv to download RISC-V toolchain and compile Spike simulator (including proxy kernel)
- Using SiFives GCC toolchain for Embedded targets (See below)
- Two variants of proxy kernel (pk, pk64) are build to support rv32gc as well as rv64gc targets
Introduce AotTestRunner to use spike in unit tests (currently rv32gc only)
Enable spike based AoTTestRunner for AoT/CRT related test cases

Future work

I have a followup ready based on this PR, which enable usage of spike for MicroTVM deployment (ProjectAPI Server).

Questions

Why not use existing CSI-NN GCC toolchain?

That toolchain (Xuantie-900-gcc-linux) is mainly targeting 64-bit devices capable of running Linux. While it could be used for 32-bit bare metal targets as well (using the -static compile flag), it is missing relevant multilib arches, such as rv32gc+ilp32d to be usable. Thus, I use Sifives GCC (which does currenlty not support the RVV1.0 extension)

Should we fix the version of spike being build or just use the latest development branch?

cc @Mousius @areusch @driazati @gigiblender

areusch

i retriggered the PR due to i think a network issue, and just have one question here.

cc @alter-xp can you take a look at the compiler used here and see if it would also work for you (or whether there's one in Zephyr we should consider)?

areusch · 2022-08-22T16:35:38Z

python/tvm/micro/testing/aot_test_utils.py

@@ -65,6 +65,17 @@
    },
 )

+AOT_SPIKE_RUNNER = AOTTestRunner(


just curious--i think this shouldn't be necessary if we are using Project API. did you need to use the AOTTestRunner infra?

AFAIK AOTTestRunner and the MicroTVM Project API are just two different approaches. The latter is more powerful (allows autotuning, profiling) but comes with some overheads (due to RPC communication) while the test runner is very minimalistic but good enough for effective unit testing.

I went the AOTTestRunner approach to run the existing AOT/CRT related test cases in a straightforward fashion. I am not sure if this was one of you goals? Feel free to propose which test should be run using spike instead.

ah gotcha; it's probably fine to take that approach. I think we do want to migrate the unit tests to Project API at some point (the advantage is that it forces the set of information passed from compiler to runtime to be presented in Model Library Format, thereby ensuring that we don't need side-channeled information beyond that format). it might be easier to leave the tests as-is for now, but i'm wondering if you foresee any challenge with using Project API with SPIKE? cc @mehrdadh @mkatanbaf @gromero

I very much encourage to use project API from the first place, unless if there's a blocker. Initially, AOTTestRunner was introduced because at that point we didn't have host driven AOT and multi model support in MLF artifact. We're gonna remove AOTTestRunner in near future and consolidate all testing to a single flow.

areusch · 2022-08-22T17:34:07Z

turns out the network issue was actually a forgotten step in adding the image, which i've now hopefully rectified.

PhilippvK · 2022-08-23T02:17:52Z

turns out the network issue was actually a forgotten step in adding the image, which i've now hopefully rectified.

@areusch it is probably a fault on my side. Seems like the spike test runner is enabled for every docker image instead of only in ci_riscv. I will try to come up with a followup.

PhilippvK · 2022-08-23T02:29:02Z

@areusch

cc @alter-xp can you take a look at the compiler used here and see if it would also work for you (or whether there's one in Zephyr we should consider)?

I got in touch with @alter-xp a while ago regarding the multilib limitations in their toolchain but if i remember correctly I did not get a response.

What we have to consider is that in the end we will have to support both baremetal (most likely 32-bit MCUs) and linux-driven (i.e. Allwinner D1 based on C906 chip) RISC-V targets.

The baremetal targets will most likely just use the MicroTVM platforms (Zephyr, Arduino, Spike) while the larger chips should just follow the usual TVM (RPC-Server) flow similar to the Aarch64 targets such as Raspberry Pis.

Having a single toolchain for both strategies is desirable to keep maintenance low, but not a requirement for now IMHO.

alter-xp · 2022-08-23T03:05:14Z

@PhilippvK @areusch

cc @alter-xp can you take a look at the compiler used here and see if it would also work for you (or whether there's one in Zephyr we should consider)?

for baremetal we have the other newlib toolchain. Its instruction support is the same as that of Linux tools.It may solve the compilation problem here.

I got in touch with @alter-xp a while ago regarding the multilib limitations in their toolchain but if i remember correctly I did not get a response.

I'm sorry I didn't fully understand your question at that time. you can try this tool Xuantie-900-gcc-elf-newlib-x86_64-V2.6.0-20220715.tar.gz

Having a single toolchain for both strategies is desirable to keep maintenance low, but not a requirement for now IMHO.

It should be a common practice in the industry to provide two sets of compilation tools for baremetal and Linux driven.

areusch · 2022-08-23T20:16:20Z

@alter-xp @PhilippvK gotcha--it's fine to have two toolchains in ci_riscv if they have a defined purpose. could we document that purpose in the install scripts?

alter-xp · 2022-08-24T03:35:17Z

@areusch Currently, there is only one Xuantie compiler in ci_riscv. I ignored the situation of baremetal before. I can add it in image. But It seems better to add it directly to this pr because this PR needs to use it. What do you @PhilippvK think?

areusch · 2022-08-24T19:44:44Z

one piece of advice: we don't quite have the debug flow well-polished for the type of PR that both adds a dependency to a docker container and then tries to use that dependency. the issue is that you can't pull the docker image built by the CI. for that reason, I suggest to split adds/uses of a new dependency into two PRs (happy to accept the first without the second as long as we're reasonably confident in landing the second).

PhilippvK · 2022-08-31T14:24:46Z

Sorry for the delay, I was not at home for the past few days. I will update the PR later this week

@alter-xp Thank your for providing the Xuantie-900-gcc-elf-newlib-x86_64-V2.6.0-20220715.tar.gz. It seems to support the relevant extensions for baseline TVM integration and it does make sense to have both the the Linux and Newlib toolchain from a similar codebase. However it have two questions (unrelated to this pr) after shortly trying it out by myself:

While the P extension seems to be somehow supported (intrinsics are available) I could not link a 32bit program (-march=rv32gcp -mabi=ilp32d) using these intrinsics, while linking 64bit programs (-march=rv64gcp -mabi=lp64d) works… Am I doing something wrong or is the 32bit version of the RVP extensions currently not supported.
I was also wondering why the p extension is not listed in the output of riscv64-unknown-elf-gcc -print-multi-lib.

It would be great if you could help me out with that!

alter-xp · 2022-09-01T06:13:24Z

hi @PhilippvK

While the P extension seems to be somehow supported (intrinsics are available) I could not link a 32bit program (-march=rv32gcp -mabi=ilp32d) using these intrinsics, while linking 64bit programs (-march=rv64gcp -mabi=lp64d) works… Am I doing something wrong or is the 32bit version of the RVP extensions currently not supported.

To link a 32bit program with P extensions, you can try this command -march=rv32gcp_zpn_zpsfoperand -mabi=ilp32d.
And for this case, the compiler may only support this option at present.

I was also wondering why the p extension is not listed in the output of riscv64-unknown-elf-gcc -print-multi-lib.

Since the file size needs to be taken into account when releasing the package, there is no library for all instruction combinations. The latest version of GCC source code will be opened in about two weeks, which may be helpful to you.

PhilippvK · 2022-09-07T20:01:45Z

Here is a short update:

I dropped the commits related to the SpikeAOTTestRunner and will look into testing using the MicroTVM Project API for a future PR. Thanks @areusch and @mehrdadh for letting me know that the currently used AOT test infrastructure is to be deprecated soon.
The baremetal toolchain used in the image is now the Xuantie900 version proposed by @alter-xp
For consistency, I refactored the CSI-NN2 insalll script to move the installation of the Linux toolchain to a separate script. As this has a few consequences, I would like to ask @alter-xp to review these changes:
- CSI-NN is not anymore included in Dockerfile.ci_cortexm
- Both toolchains are now installed into /opt/riscv instead of /opt/csi-nn2/
- The script/download_toolchain.sh script is not used anymore. This also means that the installed version of the toolchain has to be kept in sync with the used version of CSI-NN2 by updating the TVM CI script.
- The used download URLs are hopefully to be replaced with more stable/similar ones once the toolchain will be open sourced
- After moving the gcc download out of the CSI-NN2 script, if might make sense to do the same with the QEMU installation. What do you think @alter-xp?

areusch

thanks @PhilippvK !

docker/Dockerfile.ci_riscv

areusch

@PhilippvK looks like the PR is missing a file

areusch

removing approval til the missing file is in

PhilippvK · 2022-09-07T20:38:08Z

removing approval til the missing file is in

File is added now. Thanks for the hint.

areusch

thanks @PhilippvK !

docker/install/ubuntu_install_spike_sim.sh

areusch · 2022-09-07T20:41:33Z

docker/install/ubuntu_install_spike_sim.sh

+mkdir build
+cd build
+../configure --prefix=$RISCV --with-isa=RV32IMAC
+make -j`nproc`


slight pref for nproc - 1, in case you're locally building.

Oops, I forgot to fix this... sorry for that, we can change it in a later commit.

file added

alter-xp · 2022-09-08T05:31:34Z

docker/install/ubuntu_download_xuantie_gcc_linux.sh

+RISCV_GCC_RELEASE="1.12.1"
+RISCV_GCC_VERSION="2.4.0"
+RISCV_GCC_KERNEL_VERSION="5.10.4"
+RISCV_GCC_DATE="20220428"
+RISCV_GCC_ARCH="x86_64"
+RISCV_GCC_BASE="Xuantie-900-gcc-linux-${RISCV_GCC_KERNEL_VERSION}-glibc-${RISCV_GCC_ARCH}-V${RISCV_GCC_VERSION}-${RISCV_GCC_DATE}"
+RISCV_GCC_EXT="tar.gz"
+RISCV_GCC_URL="https://github.com/T-head-Semi/csi-nn2/releases/download/v${RISCV_GCC_RELEASE}/${RISCV_GCC_BASE}.${RISCV_GCC_EXT}"
+DOWNLOAD_PATH="/tmp/${RISCV_GCC_BASE}.tar.gz"


RISCV_GCC_VERSION="2.6.0" RISCV_GCC_ID="1659325511536" RISCV_GCC_KERNEL_VERSION="5.10.4" RISCV_GCC_DATE="20220715" RISCV_GCC_ARCH="x86_64" RISCV_GCC_BASE="Xuantie-900-gcc-linux-${RISCV_GCC_KERNEL_VERSION}-glibc-${RISCV_GCC_ARCH}-V${RISCV_GCC_VERSION}-${RISCV_GCC_DATE}" RISCV_GCC_EXT="tar.gz" RISCV_GCC_URL="https://occ-oss-prod.oss-cn-hangzhou.aliyuncs.com/resource//${RISCV_GCC_ID}/${RISCV_GCC_BASE}.${RISCV_GCC_EXT}" DOWNLOAD_PATH="/tmp/${RISCV_GCC_BASE}.tar.gz"

You can use this code to update the toolchain version here to make it consistent with the newlib version

Great, this is what I was looking for!

alter-xp · 2022-09-08T06:03:20Z

hi @PhilippvK

For consistency, I refactored the CSI-NN2 insalll script to move the installation of the Linux toolchain to a separate script. As this has a few consequences, I would like to ask @alter-xp to review these changes:

CSI-NN is not anymore included in Dockerfile.ci_cortexm

Both toolchains are now installed into /opt/riscv instead of /opt/csi-nn2/

The script/download_toolchain.sh script is not used anymore. This also means that the installed version of the toolchain has to be kept in sync with the used version of CSI-NN2 by updating the TVM CI script.

The used download URLs are hopefully to be replaced with more stable/similar ones once the toolchain will be open sourced

These changes don't have much effect on me. Regarding the installation script here, I am also planning to separate it recently. I'm very glad you did it.

After moving the gcc download out of the CSI-NN2 script, if might make sense to do the same with the QEMU installation. What do you think @alter-xp?

I also intend to do this. Should we also do it in this PR?

PhilippvK · 2022-09-09T00:19:25Z

I also intend to do this. Should we also do it in this PR?

I added it to this PR. Please check if the changes are okay or if I am missing something @alter-xp.

docker/install/ubuntu_download_xuantie_qemu.sh

…iles [Docker] [RISC-V] move gcc toolchain installation out of csi-nn2 script [Docker] [RISC-V] move qemu installation out of csi-nn2 script

…u_install_core.sh script

PhilippvK · 2022-09-09T14:25:07Z

I also intend to do this. Should we also do it in this PR?

@alter-xp done :)

areusch · 2022-09-15T21:02:45Z

thanks @PhilippvK @alter-xp ! we'll work on moving to newly-built docker images so you can start using this.

PhilippvK · 2022-10-14T07:05:17Z

The latest version of GCC source code will be opened in about two weeks, which may be helpful to you.

@alter-xp Is there any update on this?

alter-xp · 2022-10-14T08:01:16Z

The latest version of GCC source code will be opened in about two weeks, which may be helpful to you.

@alter-xp Is there any update on this?

hi, @PhilippvK. because of some other factors, the open time of gcc source code is delayed. The latest timetable is before December. Once the code is ready, I will inform you here.

…mage to enable RISC-V unit testing (apache#12534) * Remove CSI-NN from ci_cortexm docker image * [Docker] [RISC-V] Split up CSI-NN2 installation script into several files [Docker] [RISC-V] move gcc toolchain installation out of csi-nn2 script [Docker] [RISC-V] move qemu installation out of csi-nn2 script * use updated version of qemu * [Docker] [RISC-V] Install newlib (baremetal) gcc toolchain * [Docker] [RISC-V] Install spike simulator * [Docker] move initialization of timezone and DEBIAN_FRONTEND to ubuntu_install_core.sh script

alter-xp · 2022-12-05T01:39:43Z

hi, @PhilippvK the new gcc source code has been opened, you can find here. I hope this will help you.

areusch reviewed Aug 22, 2022

View reviewed changes

PhilippvK force-pushed the feature_docker_spike branch from 6e56a17 to 49aef09 Compare September 5, 2022 09:46

github-actions bot requested review from Mousius and driazati September 5, 2022 09:47

PhilippvK force-pushed the feature_docker_spike branch from 49aef09 to 869f95c Compare September 5, 2022 12:38

Remove CSI-NN from ci_cortexm docker image

f99385c

PhilippvK force-pushed the feature_docker_spike branch from 869f95c to 0025677 Compare September 7, 2022 13:44

areusch approved these changes Sep 7, 2022

View reviewed changes

docker/Dockerfile.ci_riscv Outdated Show resolved Hide resolved

areusch reviewed Sep 7, 2022

View reviewed changes

areusch previously requested changes Sep 7, 2022

View reviewed changes

PhilippvK force-pushed the feature_docker_spike branch from 0025677 to 7c9ab89 Compare September 7, 2022 20:37

PhilippvK force-pushed the feature_docker_spike branch from 7c9ab89 to 5fe5fe7 Compare September 7, 2022 20:39

areusch reviewed Sep 7, 2022

View reviewed changes

alter-xp reviewed Sep 8, 2022

View reviewed changes

PhilippvK force-pushed the feature_docker_spike branch from 5fe5fe7 to de32558 Compare September 8, 2022 05:55

PhilippvK force-pushed the feature_docker_spike branch from de32558 to c64a400 Compare September 9, 2022 00:18

alter-xp reviewed Sep 9, 2022

View reviewed changes

docker/install/ubuntu_download_xuantie_qemu.sh Outdated Show resolved Hide resolved

PhilippvK added 5 commits September 9, 2022 14:16

[Docker] [RISC-V] Split up CSI-NN2 installation script into several f…

f13a37f

…iles [Docker] [RISC-V] move gcc toolchain installation out of csi-nn2 script [Docker] [RISC-V] move qemu installation out of csi-nn2 script

use updated version of qemu

828aac1

[Docker] [RISC-V] Install newlib (baremetal) gcc toolchain

589d809

[Docker] [RISC-V] Install spike simulator

0b2b2b4

[Docker] move initialization of timezone and DEBIAN_FRONTEND to ubunt…

f17e2b4

…u_install_core.sh script

PhilippvK force-pushed the feature_docker_spike branch from c64a400 to f17e2b4 Compare September 9, 2022 12:16

areusch approved these changes Sep 15, 2022

View reviewed changes

areusch merged commit f5517d4 into apache:main Sep 15, 2022

AndrewZhaoLuo mentioned this pull request Oct 4, 2022

TVM v0.10.0.rc0 Release Candidate Notes #12979

Closed

PhilippvK mentioned this pull request Jul 25, 2023

[microTVM][RISCV] Tensorization for conv_2d_nchw_int8 with RVV extension #14836

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Docker][CI][RISC-V] Build riscv-isa-sim (spike) in ci_riscv Docker image to enable RISC-V unit testing #12534

[Docker][CI][RISC-V] Build riscv-isa-sim (spike) in ci_riscv Docker image to enable RISC-V unit testing #12534

PhilippvK commented Aug 22, 2022 •

edited by github-actions bot

Loading

areusch left a comment

areusch Aug 22, 2022

PhilippvK Aug 23, 2022

areusch Aug 23, 2022

mehrdadh Sep 1, 2022 •

edited

Loading

areusch commented Aug 22, 2022

PhilippvK commented Aug 23, 2022

PhilippvK commented Aug 23, 2022

alter-xp commented Aug 23, 2022 •

edited

Loading

areusch commented Aug 23, 2022

alter-xp commented Aug 24, 2022

areusch commented Aug 24, 2022

PhilippvK commented Aug 31, 2022

alter-xp commented Sep 1, 2022 •

edited

Loading

PhilippvK commented Sep 7, 2022 •

edited

Loading

areusch left a comment

areusch left a comment

areusch left a comment

PhilippvK commented Sep 7, 2022

areusch left a comment

areusch Sep 7, 2022

PhilippvK Sep 15, 2022

alter-xp Sep 8, 2022 •

edited

Loading

PhilippvK Sep 8, 2022

alter-xp commented Sep 8, 2022 •

edited

Loading

PhilippvK commented Sep 9, 2022

PhilippvK commented Sep 9, 2022

areusch commented Sep 15, 2022

PhilippvK commented Oct 14, 2022

alter-xp commented Oct 14, 2022

alter-xp commented Dec 5, 2022

[Docker][CI][RISC-V] Build riscv-isa-sim (spike) in ci_riscv Docker image to enable RISC-V unit testing #12534

[Docker][CI][RISC-V] Build riscv-isa-sim (spike) in ci_riscv Docker image to enable RISC-V unit testing #12534

Conversation

PhilippvK commented Aug 22, 2022 • edited by github-actions bot Loading

Context

Summary

Future work

Questions

areusch left a comment

Choose a reason for hiding this comment

areusch Aug 22, 2022

Choose a reason for hiding this comment

PhilippvK Aug 23, 2022

Choose a reason for hiding this comment

areusch Aug 23, 2022

Choose a reason for hiding this comment

mehrdadh Sep 1, 2022 • edited Loading

Choose a reason for hiding this comment

areusch commented Aug 22, 2022

PhilippvK commented Aug 23, 2022

PhilippvK commented Aug 23, 2022

alter-xp commented Aug 23, 2022 • edited Loading

areusch commented Aug 23, 2022

alter-xp commented Aug 24, 2022

areusch commented Aug 24, 2022

PhilippvK commented Aug 31, 2022

alter-xp commented Sep 1, 2022 • edited Loading

PhilippvK commented Sep 7, 2022 • edited Loading

areusch left a comment

Choose a reason for hiding this comment

areusch left a comment

Choose a reason for hiding this comment

areusch left a comment

Choose a reason for hiding this comment

PhilippvK commented Sep 7, 2022

areusch left a comment

Choose a reason for hiding this comment

areusch Sep 7, 2022

Choose a reason for hiding this comment

PhilippvK Sep 15, 2022

Choose a reason for hiding this comment

alter-xp Sep 8, 2022 • edited Loading

Choose a reason for hiding this comment

PhilippvK Sep 8, 2022

Choose a reason for hiding this comment

alter-xp commented Sep 8, 2022 • edited Loading

PhilippvK commented Sep 9, 2022

PhilippvK commented Sep 9, 2022

areusch commented Sep 15, 2022

PhilippvK commented Oct 14, 2022

alter-xp commented Oct 14, 2022

alter-xp commented Dec 5, 2022

PhilippvK commented Aug 22, 2022 •

edited by github-actions bot

Loading

mehrdadh Sep 1, 2022 •

edited

Loading

alter-xp commented Aug 23, 2022 •

edited

Loading

alter-xp commented Sep 1, 2022 •

edited

Loading

PhilippvK commented Sep 7, 2022 •

edited

Loading

alter-xp Sep 8, 2022 •

edited

Loading

alter-xp commented Sep 8, 2022 •

edited

Loading