Is execution of docker images for foreign architectures supported?

[eine]

Since kata-runtime uses QEMU under-the-hood, I wonder whether it is possible to execute docker images for foreign architectures. E.g., execute an arm32v7/ubuntu container on a amd64 host. So, instead of creating a VM for the same architecture as the host, kata would create a VM for the guest architecture.

[egernst]

@1138-4eb It isn't a use case that we really had in mind. How would you see end users utilizing this, and in what use-case?

[eine]

Hi @egernst!, there are two main approaches to build/develop/debug docker images for foreign architectures with QEMU and runc, without requiring any modification to the docker images:

full-system emulation

The user must setup the QEMU, install docker (runtime) and then build/run the container inside. On the one hand, I am not sure about specific privileges that the VM might need for docker engine to work inside. On the other hand, although I believe that scripts to automate this do exist, the user is required to be aware of both docker parameters and QEMU parameters.

Hence, a docker user who is new to QEMU must learn how to handle images, launch options and communication with the VM (e.g. https://www.ibm.com/developerworks/library/l-qemu-development/index.html). Also, resources from the container need to be bind to the VM and from there to the host. So docker and QEMU parameters must match.

Moreover, I don't know if it is possible to run qemu-system on a host which is itself a VM. I am thinking on CI providers. http://blog.kragniz.eu/raspbian-on-travis-ci/

user-mode emulation

The setup is much easier if qemu-user-static is used: just execute qemu-binfmt-conf.sh --qemu-suffix -static -p yes on the host. The QEMU binaries are loaded to memory and any docker build or docker run will just work. Therefore, a docker user can use the container on the amd64 host as if it was being executed on the target architecture. I.e., docker is effectively used as an alternative for emulating a rootfs (see https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/). Of course, this approach has two main caveats. First, user-mode emulation seems to be less polished than full-system emulation, so depending on the application to be developed, it might crash. Second, the underlying machine is the host, which is likely to lack hardware resources available on the target device/system.

---

The question is if kata-containers can fill the gap and provide a seamless transition for docker users changing from user-mode to full-system emulation in order to build/execute images for the foreign architectures listed at https://github.com/docker-library/official-images#architectures-other-than-amd64. I see end users utilizing it mostly to develop/debug docker containers for SBCs and/or SoCs with FPGAs based on armv7 or armv8, such as RPi, PYNQ, 96boards, Pine64, etc.

This use case might certainly be out of the scope of kata-containers as a project, because it seems to be focused on workload isolation and security advantages of VMs. Nonetheless, disregarding those very interesting features, kata-runtime is a middleware that allows to use docker as a frontend for QEMU. It would potentially allow to execute docker run <image> on the target device and docker run --runtime=kata-runtime --qemu-sys=rpi3b <image> on the host to get a as good as qemu-system can provide emulation.

Nevertheless, my perspective about executing it on a user laptop or workstation might be quite narrow, so I wouldn't like to limit the idea/concept to this use case. I wonder if such a feature would also fit in the stack of CI providers based on x86-64 (e.g. Travis-CI).

EDIT

See also https://youtu.be/4MaGnMGPIq0?t=365

[caoruidong]

Performance is a big problem.

[eine]

Performance is a big problem.

Sure. That's why:

• It is an alternative when user-mode emulation does not work/fit. Otherwise, user-mode is strongly recommended. Certainly, if the app/project can be tested on x86 directly, that's the first and best option (using docker with or without kata).
• I wouldn't expect performance to be any better than using qemu-system directly. But I would neither expect it to be significantly worse.

Let's word it different. Can kata-runtime be considered a tool to use docker as a frontend for QEMU system-mode emulation? I.e. kata provides some kernels/images that complement docker filesystems and the CLI tools to combine them together. If so, it should be relatively straighforward to replace the kernel/image and the qemu-system binary. However, I am afraid that I might be overlooking several layers of complexity.

EDIT

Note about why KVM cannot be used for cross-architecture virtualization: https://unix.stackexchange.com/questions/340912/qemu-with-kvm-with-differing-host-guess-architectures. However: https://dl.acm.org/citation.cfm?id=3012405.2996798

https://wiki.qemu.org/Features/Containers

[egernst]

Thanks @1138-4eb -- imo this doesn't clearly align with the goals of the project as is today, but I understand how it'd be useful in your case.

Other AC members - wdyt? @jon @gnawux @sameo @WeiZhang555

[WeiZhang555]

This will bring lots of complexicity. Running an arm64 kata container on amd64 Host will require (correct me) kata-runtime: amd64 version, qemu: amd64 version, guest kernel: arm64 version, docker image: arm64 version, guest agent: arm64 version. Now we need to handle GRPC protocol between amd64 kata-runtime and arm64 kata-agent, this could be disaster and hard to maintain in my mind. Though the scenario is interesting.

I don't think it's a good idea to do this currently, it's my 2 cents.

[gnawux]

Thanks for raising this question. Sure, it's pretty interesting. But I don't think we should add the complexity just because it is interesting. However, I am still open to this requirement as long as there are enough use cases. IMHO, emulation will introduce too much penalty on performance and is not practical for large scale deployment. This will increase the complexity of configuration as well.

…

On Wed, Mar 20, 2019, 17:30 zhangwei_cs ***@***.***> wrote: This will bring lots of complexicity. Running an arm64 kata container on amd64 Host will require (correct me) kata-runtime: amd64 version, qemu: amd64 version, guest kernel: arm64 version, docker image: arm64 version, guest agent: arm64 version. Now we need to handle GRPC protocol between amd64 kata-runtime and arm64 kata-agent, this could be disaster and hard to maintain in my mind. Though the scenario is interesting. I don't think it's a good idea to do this currently, it's my 2 cents. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1280 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADTbGK-kkJFUbVF_yvskmE6WghI8z5gks5vYf-qgaJpZM4bQkio> .

[eine]

Running an arm64 kata container on amd64 Host will require (correct me)
kata-runtime: amd64 version,
qemu: amd64 version,
guest kernel: arm64 version,
docker image: arm64 version,
guest agent: arm64 version.
Now we need to handle GRPC protocol between amd64 kata-runtime and arm64 kata-agent, this could be disaster and hard to maintain in my mind.

Thnks for this description. This is the kind of explanation I was expecting to receive.

Overall, I see that none of you think this to be a good idea currently, mostly because you are focused on large scale deployment. Still, I'd be glad if we could leave this issue open, just for the record, as it might be revisited in the future. This will allow users with other use cases to add a reference.

[WeiZhang555]

@1138-4eb Yep, we can keep this issue open for collecting more scenarios 😄

[eine]

@jodh-intel should this be kept in 'To Do' or marked as 'Wontdo', instead of 'Done'?