I looked into this and there are some problems. The first problem is the inability to fine tune any parameters we might want to for target cpu/microarchitecture (for arm, that would be building packages with specifics for ThunderX, McBin, Raspberry PI etc.). I'm not sure how Qemu does the emulation, but I doubt there's extensive support for a myriad of cpus. And on top of that, it's slow.
The other thing is with using the x86 version of aarch64-linux-gnu-gcc. In that case it's just regular cross-compilation (x86 version of aarch64-linux-gnu-gcc is the cross-compiler) and what we gain from these target platform containers is having the proper libraries that VPP build depends on. For Ubuntu I've found that it's actually not worth it - it's easier to have an x86 container and install arm dependency packages that to install x86 cross compiler inside an aarch64 container. Another thing I've observed (in htop) is that even when using the x86 version of aarch64-linux-gnu-gcc inside the aarch64 container, it was still going through the emulator, but maybe I did something wrong. Damjan, you mentioned that my current patch doesn't solve anything. It certainly isn't a comprehensive solution, but it does one thing and that is it allows users to specify platform specific config args (well, at least some of the supported ones in build-data/platforms/<platform>.mk) which then get propagated to all parts of the build and it's possible to do cross compilation given that the environemnt has been already set up. It modifies the current ebuild system, but that might not be the appropriate place to do that. However, I don't how else would we do this. I'm not sure what all of this means, but the docker solution is certainly incomplete, if not outright unsuitable. Maybe we could use containers for just environment setup (e.g. for Ubuntu, installing both host and target packages) and then could run cross-compilation in them with a solution that would do something like my patch (i.e. cross-compile DPDK and VPP with config args defined in one file). Thoughts? Juraj From: Damjan Marion via Lists.Fd.Io <dmarion=me....@lists.fd.io> Sent: Thursday, October 31, 2019 1:49 PM To: Benoit Ganne (bganne) <bga...@cisco.com> Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Change in vpp[master]: ebuild: Cross compilation aarch64 Ubuntu support > On 31 Oct 2019, at 13:18, Benoit Ganne (bganne) > <bga...@cisco.com<mailto:bga...@cisco.com>> wrote: > >> I was going to remain silent, but since there's now multiple people saying >> this sounds good -- I think this sounds horrible. :) >> To wit, it seems too complex and too much setup/overhead. I'll try and >> look closer at this soon to see if I can feed back our local changes that >> seem to be working. > > It is not that bad in my opinion [1] : > 1) add support for multiarch (must be done once after reboot) > ~# docker run --rm --privileged multiarch/qemu-user-static --reset > --persistent yes --credential yes > 2) create your chroot (must be done once - I am sharing my homedir with my > chroot and same UID/GID) > ~# docker run --name aarch64_u1804 --privileged --net host -v $HOME:$HOME -v > /dev:/dev -v/lib/modules:/lib/modules/host:ro -td arm64v8/ubuntu:18.04 > /bin/bash > ~# docker container exec aarch64_u1804 sh -c "apt -qy update && apt > dist-upgrade -qy && apt install -qy vim sudo make git && groupadd -g $(id > -rg) $USER && useradd -u $(id -ru) -g $(id -rg) -M -d $HOME -s /bin/bash > $USER && echo '$USER ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && echo > aarch64_u1804 > /etc/debian_chroot" > 3) compile vpp (I already checked out VPP in $HOME/src/vpp but you can > checkout it there too if you prefer) > ~# docker container exec aarch64_u1804 su "$USER" -l -c "UNATTENTED=y make -C > src/vpp install-dep install-ext-deps pkg-deb" > [...] > dpkg-deb: building package 'libvppinfra-dev' in > '../libvppinfra-dev_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp-dbg' in > '../vpp-dbg_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'libvppinfra' in > '../libvppinfra_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp-api-python' in > '../vpp-api-python_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp' in > '../vpp_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp-plugin-dpdk' in > '../vpp-plugin-dpdk_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'python3-vpp-api' in > '../python3-vpp-api_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp-dev' in > '../vpp-dev_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp-plugin-core' in > '../vpp-plugin-core_20.01-rc0~538-gbb41ee925_arm64.deb'. > make[2]: Leaving directory > '/home/bganne/src/vpp/build-root/build-vpp-native/vpp' > dpkg-genbuildinfo --build=binary > dpkg-genchanges --build=binary >../vpp_20.01-rc0~538-gbb41ee925_arm64.changes > dpkg-genchanges: info: binary-only upload (no source code included) > dpkg-source --after-build vpp > dpkg-buildpackage: info: binary-only upload (no source included) > make[1]: Leaving directory '/home/bganne/src/vpp/build-root' > make: Leaving directory '/home/bganne/src/vpp' > > Enjoy :) This is so cool :) I wanted to invest time to investigate how to do exactly this, so thanks for saving my time Only nice thing will be to find way to run x86 version of aarch64-linux-gnu-gcc instead of aarch64 version of gcc so it runs faster... -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14412): https://lists.fd.io/g/vpp-dev/message/14412 Mute This Topic: https://lists.fd.io/mt/39781289/899915 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [juraj.lin...@pantheon.tech] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14792): https://lists.fd.io/g/vpp-dev/message/14792 Mute This Topic: https://lists.fd.io/mt/39781289/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
