Test on arm devices #4

HidenoriMatsubayashi · 2021-03-09T00:52:00Z

This issue is a memo for testing on different arm devices. I would like to test on a lot of arm devices.

Tested devices

Jetson Nano (OS: Ubuntu 18.04, JetPack)
- Wayland backend
- DRM backend (Support EGLStream #1)
Raspberry Pi4 (OS: Ubuntu 20) Doesn't work on Raspberry Pi4 when using DRM backend #9
- Wayland backend
- DRM backend
i.MX 8M Mini EVK (OS: Zeus, imx-5.4.70-2.3.1.xml)
- Wayland backend
  ~~DRM backend~~ NXP doesn't support EGL for GBM.
RB5 (OS: Based Ubuntu 18.04)
- DRM-GBM backend
  ~~Wayland backend~~ Weston version is 3.0 and it doesn't support xdg-shell

HidenoriMatsubayashi · 2021-03-09T07:12:12Z

@Mai-Matsuura Could you try testing on Raspberry Pi4?

ghost · 2021-03-09T07:16:40Z

Yes, I will try it.

psstoyanov · 2021-03-10T21:15:55Z

Observations from my devices:

PinePhone (Allwinner A64) and PinebookPro (rk3399) both running ManjaroARM with kernel 5.11 and their respective open source driver:
- drm backend doesn't work on either system (different errors are being produced)
- running the wayland client works provided the app is running within a Weston environment. Using plasma wayland/ plasma mobile/ wlroots based environment results in the same error: [ERROR][linuxes_window_wayland.cc(627)] Invalid compositor and shell.

Currently I can't verify on my desktop system using RX6800 (main desktop environment - Plasma Wayland). For whatever reason the Dart VM snapshot version doesn't match:

./flutter-drm-backend sample_2/build/linux/x64/release/bundle           
amdgpu_device_initialize: amdgpu_query_info(ACCEL_WORKING) failed (-13)
amdgpu: amdgpu_device_initialize failed.
[ERROR][context_egl.h(95)] Failed to create EGL off-screen surface.(eglGetError: EGL_BAD_ALLOC)
[ERROR][linuxes_surface_gl_drm.h(50)] Off-Screen surface is invalid.
[ERROR][native_window_drm.cc(89)] TODO: implement here!!
[ERROR][flutter_linuxes_view.cc(60)] Failed to change surface size.
[ERROR][native_window_drm.cc(311)] Unsupported cursor: 
[ERROR][native_window_drm.cc(176)] Could not move the mouse cursor: -13
[ERROR][native_window_drm.cc(130)] Failed to set cursor buffer. (-13)
[FATAL:flutter/runtime/dart_vm_initializer.cc(22)] Error while initializing the Dart VM: Wrong full snapshot version, expected '39a9141bbcc3cae43e6f9f6b7fbaafe3' found 'aac992673ad0e4a5e41716d2a9911c88'
[1]    171859 abort (core dumped)  ./flutter-drm-backend sample_2/build/linux/x64/release/bundle

HidenoriMatsubayashi · 2021-03-11T05:12:08Z

Thank you for your comment.

[FATAL:flutter/runtime/dart_vm_initializer.cc(22)] Error while initializing the Dart VM: Wrong full snapshot version, expected '39a9141bbcc3cae43e6f9f6b7fbaafe3' found 'aac992673ad0e4a5e41716d2a9911c88'
[1]    171859 abort (core dumped)  ./flutter-drm-backend sample_2/build/linux/x64/release/bundle

The first problem is that the Flutter SDK and Engine (libflutter_engine.so) versions don't match. Could you re-build Flutter Engine (libflutter_engine.so) with the same version using in the Flutter SDK you are using? Flutter SDK specifies the version of Flutter Engine in ${INSTALL_PATH}/bin/internal/engine.version.

But I need to update README.

psstoyanov · 2021-03-11T10:11:53Z

Thank you for the information! This makes it a lot easier to verify if the correct versions are in use.

The issue on my x64 machine could lie somewhere else as the versions are indeed matching 🤔 I will clean and update my setup later today to ensure there are no lingering caches remaining. I've already thoroughly cleaned the Flutter SDK && sample projects but I haven't done the same for the depot_tools yet.

Versions matching

$ cat flutter/bin/internal/engine.version && cat depot_tools/.gclient
9b5f59ec3d54815931662a78a6ef34aa38bd5270
solutions = [
  {
    "managed": False,
    "name": "src/flutter",
    "url": "https://github.com/flutter/engine.git@9b5f59ec3d54815931662a78a6ef34aa38bd5270",
    "custom_deps": {},
    "deps_file": "DEPS",
    "safesync_url": "",
  },
]

psstoyanov · 2021-03-11T13:02:10Z

Update with RK3399 system - PinebookPro running ManjaroARM kernel 5.11.3 && mesa 20.3.4 using the DRM mode:

Switching to tty3 and running the drm mode produces the expected app. The text is visible, the touchpad is functional 👍
The output produces some errors and warnings but I don't see problem with the app itself

Detailed log with lldb

lldb ./flutter-drm-backend ./sample/build/linux/arm64/release/bundle
(lldb) target create "./flutter-drm-backend"
Current executable set to '/home/pak0st/flutter-embedded-linux/build/flutter-drm-backend' (aarch64).
(lldb) settings set -- target.run-args  "./sample/build/linux/arm64/release/bundle"
(lldb) run
Process 2714 launched: '/home/pak0st/flutter-embedded-linux/build/flutter-drm-backend' (aarch64)
[WARNING][linuxes_window_drm.cc(102)] FLUTTER_DRM_DEVICE is not set, use /dev/dri/card0
[WARNING][context_egl.h(95)] Failed to create EGL off-screen surface.(eglGetError: EGL_BAD_ALLOC)
[WARNING][linuxes_surface_gl_drm.h(50)] Off-Screen surface is invalid.
[ERROR][native_window_drm.cc(89)] TODO: implement here!!
[ERROR][flutter_linuxes_view.cc(60)] Failed to change surface size.
[ERROR:flutter/shell/platform/embedder/embedder_surface_gl.cc(102)] Could not create a resource context for async texture uploads. Expect degraded performance. Set a valid make_resource_current callback on FlutterOpenGLRendererConfig.
[ERROR:flutter/shell/platform/embedder/embedder_surface_gl.cc(102)] Could not create a resource context for async texture uploads. Expect degraded performance. Set a valid make_resource_current callback on FlutterOpenGLRendererConfig.
[ERROR][native_window_drm.cc(311)] Unsupported cursor:

psstoyanov · 2021-03-11T20:30:16Z

Small update towards my experiments on x64 with my Ryzen/RDNA2 machine:

Still experiencing the error with wrong Dart VM version when compiling the libraries (fresh flutter sdk/ depot_tools). What I find even more perplexing is that the exact same setup used on my aarch64 devices works.
When using the debug library from Google's infrastructure, I get the following error when running the wayland client in Weston:

[ERROR:flutter/shell/common/shellcc(103)] Dart Error: Can't load Kernel binary: Invalid kernel binary format version.

The closest Flutter ticket to what I experience appears to be this one: flutter/flutter#64849

LefixBlue · 2021-03-12T04:31:00Z

Hi, I follow your work regarding flutter and embedded for a while already and I am very happy about it, because I share the same view of Sony regarding flutter and embedded linux! And it's amazing to see it running now on embedded devices..

Now lets come to my test:

I have running a Yocto (master branch for all layers) core-image-base for raspberry-64. I have added all needed dependencies, as well as clang from meta-clang and can successfully build everything and use your flutter-client within weston aswell as the drm-backend directly from within the CLI. Great stuff!
First I was thinking the flutter-client can be used without weston, directly from within CLI, but that was probably more related to my little understanding about it. For now I would say the DRM-Backend is the way to go if you only need a single application for your embedded device and then the wayland and weston stuff can be dropped completely. But of course that's project decision.
However I got some compilation errors due to wrongly set includes when building directly on the Pi. I use clang12 built from meta-clang into the raspberrry image. Error was about unordered_map includes missing in at least one header file (key_event_plugin.h). But building the source cross from yocto didn't throw that error but for my understanding it has to be fixed.
Furthermore I am still a bit unhappy about the fact that we still need GTK+3 to be able to build die Linux-Desktop-App before being able to use it with the embedder. Isn't it the case that we just need to app.so plus the assets from the desktop bundle? But for long term I will integrate everything into my yocto build, I am just waiting to cross compile feature will be enabled.
Another issue I would like to address, which probably more flutter related -> Very HIGH CPU load for Linux Desktop either on x64 with a normal desktop build and as well as on arm64, and of course also with your embedder, 1 rasperry core is always running on 100% when flutter app in running with drm-backend. But that seems a general flutter thing, because I have an old release build of an x64 desktop app here which is not that cpu hungry, as when I rebuild it with the current master branch. I would be interested if i am the only one facing this cpu related problems or at least someone else could confirm that.

So far from my side. I am looking forward to the cross compile feature! If someone can give me a hint how to generate the app.so manually on an x64 (without the needs to wait the related PR to be merged) would also be cool.

HidenoriMatsubayashi · 2021-03-12T06:07:55Z

@LefixBlue

Thank you for your comments. Those are helpful.

Error was about unordered_map includes missing in at least one header file (key_event_plugin.h).

I fixed that (#22)

HidenoriMatsubayashi · 2021-03-12T06:11:38Z

@LefixBlue

In terms of cross-building, I'm going to send a PR to add cross-building support into the Flutter SDK. I'll do it in March.

If someone can give me a hint how to generate the app.so manually on an x64

Yes, see: https://github.com/flutter/flutter/wiki/Custom-Flutter-Engine-Embedding-in-AOT-Mode#building-the-aot-snapshot

LefixBlue · 2021-03-12T06:50:54Z

@HidenoriMatsubayashi
Thank you for the link regarding snapshots.
I will have a look into it.

LefixBlue · 2021-03-12T07:37:54Z

@HidenoriMatsubayashi

After reading the provided link I was still unsure where to get the correct gen_snapshot from. But now I just realized that building the engine for arm64 on x64 will produce a gen_snapshot in clang_x64/exe.unstripped folder with simarm64 target. So this one can be used on x64 to "cross-compile" the app.dill to app.so for arm64.

Thanks for your help!

psstoyanov · 2021-03-12T10:05:57Z

@LefixBlue check #21 for:

First I was thinking the flutter-client can be used without weston, directly from within CLI, but that was probably more related to my little understanding about it.

I agree that side-stepping the GTK app compile will be great once the embedder matures enough :)

psstoyanov · 2021-03-14T00:19:17Z

Had some fun doing something unconventional tonight by compiling the embedder on MSM8916 device (Wileyfox Swift) with postmarketOS. It resulted in the following errors:

pmOS linker errors

➜  build git:(master) make                                                                                              
[  3%] Linking CXX executable flutter-drm-backend
/usr/bin/ld: warning: libdl.so.2, needed by /usr/lib/libflutter_engine.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libpthread.so.0, needed by /usr/lib/libflutter_engine.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libm.so.6, needed by /usr/lib/libflutter_engine.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libc.so.6, needed by /usr/lib/libflutter_engine.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: ld-linux-aarch64.so.1, needed by /usr/lib/libflutter_engine.so, not found (try using -rpath or -rpath-link)

The errors have less to do with the embedder itself and more with Flutter/ Alpine linux.

There are several issues with the method I've use to conduct the attempt on this device tonight:

postmarketOS is not meant to use dev libraries on the end user side
The attempt was not conducted through pmbootstrap's chroot environment to compile the embedder
PostmarketOS is great and has very close ties to Alpine linux - a great small distribution, perfect for embedded or specialized uses.

I will need to prepare a proper setup and try again but thought it would be interesting to share nonetheless.

LefixBlue · 2021-03-15T06:15:32Z

Some new input from my side as well:

Building the "flutter-weston-dekstop-shell" not succeed in my environment. It already failed on my first attempt directly on the raspberry pi some days ago, but now I retried building it cross with yocto and getting the same result.

in main.cc:
config.parser.h not found (i guess because my .pc file for libweston points to /usr/inlclude as include dir but the relevant headers are in /usr/include/libweston-9/libweston)
also in main.cc:
3rd argument of 'weston_config_section_get_bool' shoud be a 'bool-' instead of an 'int-'pointer

HidenoriMatsubayashi · 2021-03-15T06:18:43Z

Thank you for your report! Actually, Weston is not backward compatible and we are looking for ways to support multiple versions. See also: #3

LefixBlue · 2021-03-15T06:22:45Z

@HidenoriMatsubayashi

Havn't noticed #3 yet.
Thanks for the information.

HidenoriMatsubayashi · 2021-03-15T06:25:24Z

No. Thank YOU!

HidenoriMatsubayashi · 2021-03-16T06:32:33Z

I've tried on i.MX 8M Mini EVK (OS: Zeus, imx-5.4.70-2.3.1.xml), but the embedder with DRM backend didn't work. Wayland backend is no problem.

root@imx8mmevk:~# FLUTTER_DRM_DEVICE=/dev/dri/card1 ./flutter-drm-backend ./bundle/
[     1] ioctl(DRM_IOCTL_GEM_CLOSE) failed
[ERROR][native_window_drm.cc(42)] Failed to create the compositor surface.
[ERROR][linuxes_window_drm.cc(110)] Failed to create the native window

psstoyanov · 2021-03-17T10:12:41Z

Wohoo! After some extensive cleanup on my host machine, I've found the offending piece causing me problems with the depot_tools. As it turns out, I've had a number of remnants of an AUR package - flutter-git that collided with my main Flutter install and was using different version. Whatever unintended referencing was happening there is no more.

So, I can finally confirm - the embedder does work with AMD RX6800 with the open source drivers using both the DRM and wayland clients. Mesa version installed on the x64 system - 20.3.4.
Note: Unlike with the arm devices I have, for the DRM session there was one card to be targeted - /dev/dri/card0

LefixBlue · 2021-04-13T10:14:41Z

@HidenoriMatsubayashi

I also had a try on the actual iMX8M-MiniEVK BSP Release (OS: Gatesgarth 5.10.9) with the drm-backend.
And unfortunately I got the same result as you mentioned here a while ago (ioctl failed).
Is there already an explanation why this error happens?
Or is the drm-backend not meant to run on the iMX8M-Platform at all?

HidenoriMatsubayashi · 2021-04-13T10:30:22Z

We want to support the i.MX8M platform, so we need to investigate this issue. I think i.MX8M supports DRM and actually, Weston which uses the DRM backend works on i.MX8M mini. I'll investigate this week. Please tell me more if you have any information.

LefixBlue · 2021-04-13T12:04:01Z

I am unfortunately not that deep into all these graphical backend stuff..

The only thing I observed today is that I ran into the same build-error like someone reported in #30 while building the drm-backend cross with yocto. You mentioned in #30 that the WL_EGL_PLATFORM define would be missing, but that didn't worked for me because in the drm-backend case we dealing with gbm. To get around the build-errors I finally (after try many different other things) tried to cast the relevant parameter in the calls eglCreateWindowSurface and eglGetDisplay to EGLNativeWindowType. It did build through then, but the result at runtime is that ioctl error.

No idea if that helps.

HidenoriMatsubayashi · 2021-04-14T00:37:19Z

To get around the build-errors I finally (after try many different other things) tried to cast the relevant parameter in the calls eglCreateWindowSurface and eglGetDisplay to EGLNativeWindowType

If it's a bug, I'd like to fix it. Now, we are preparing yocto recipe examples.

~~In terms of i.mx8m, it looks like it doesn't support GBM_BO_FORMAT_ARGB8888. Probably I need to fix the code.~~

topolittle · 2022-01-07T15:58:41Z

I tested on i.MX8M-Mini with the toolchain generated by Yocto. Unfortunately, this doesn’t works.

The Yocto’s toolchain doesn’t put the cross tools (cross-compiler, linker, etc) in the /usr/bin/ folder, instead they are in /usr/bin//

The toolchain sets the environment variables “CC”, “CXX”, “CLANGCC”, “CLANGCXX” and “LD” to the location of the corresponding cross-tools in /usr/bin// but flutter-elinux doesn’t seems to use these environment variables, like a traditional ‘Make’ would. Instead flutter-elinux looks for the cross-tools directly in /usr/bin/ but with the Yocto’s toolchain, it’s the native x86 tools at this location.

Is there a way to tell flutter-elinux to look for the cross-tools in a specific folder ?
Can flutter-elinux make use of the above environment variables ?

topolittle · 2022-01-10T20:17:01Z

A quick update:
I managed to make it work by adding the following to my toolchain file 'OEToolchainConfig.cmake':

set ( CMAKE_CXX_COMPILER_WORKS 1 )
set( CMAKE_CXX_COMPILER_TARGET "aarch64-poky-linux" )

Now I am able to cross-compile using flutter-elinux and the SDK generated by Yocto.
It still doesn't works because of the bug #70

LefixBlue · 2022-01-11T03:53:12Z

Hi @topolittle ,

See my comment here: sony/meta-flutter#27

Another way to cross build flutter app with Yocto SDK is to patch flutter-elinux to use the correct compiler triple. I wasn't aware of editing the cmake toolchain file like you mentioned also does the trick. I would prefer to add a cmdline argument to elinux where the compiler-target/triple can be specified for cross building.

Regarding sony/flutter-elinux#70:
As far as I understood the current workaround is to use and copy the correct gen_snapshot-binary from the flutter-engine build results to elinux. See the wiki here or also my comment mentioned above.

atornqvist · 2022-04-05T17:39:52Z

Hi,

Cross compiling with a pure Yocto Poky dunfell SDK fails. The following can be made to
reproduce the error. It would be nice with cmdline argument where the compiler prefix could be specified.
Alternatively the environment variable CROSS_COMPILE as with other packages if possible.

Build a dunfell Poky SDK:
mkdir ~/flutter_tests
cd ~/flutter_tests

git clone git://git.yoctoproject.org/poky.git -b dunfell
git clone https://github.com/kraj/meta-clang -b dunfell
git clone https://github.com/sony/meta-flutter.git
source poky/oe-init-build-env build

cat >> conf/local.conf << EOF
MACHINE ?= "qemuarm64"
CLANGSDK = "1"
EOF

bitbake-layers add-layer ../meta-clang
bitbake-layers add-layer ../meta-flutter

bitbake core-image-weston -c populate_sdk

tmp/deploy/sdk/poky-glibc-x86_64-core-image-weston-aarch64-qemuarm64-toolchain-3.1.15.sh

Source the SDK
. /scratch/opt/poky/3.1.15/environment-setup-aarch64-poky-linux

git clone https://github.com/sony/flutter-elinux.git
export PATH="$PATH:pwd/flutter-elinux/bin"
flutter-elinux precache
flutter-elinux create exampleapp
cd exampleapp
flutter-elinux build elinux --target-arch=arm64 --target-sysroot=/scratch/opt/poky/3.1.15/sysroots/aarch64-poky-linux/

=====> Linker error since it is the Yocto x86_64 ld that is used and not the cross-build ld.

Modifying the SDK file OEToolchainConfig.cmake to include these two lines does not help.
set ( CMAKE_CXX_COMPILER_WORKS 1 )
set( CMAKE_CXX_COMPILER_TARGET "aarch64-poky-linux" )

Editing the flutter-elinux files from aarch64-linux-gnu to aarch64-poky-linux does not help either.

HidenoriMatsubayashi · 2022-04-06T05:10:35Z

@atornqvist

Could you try the following?

$ export CC=${CLANGCC}
$ export CXX=${CLANGCXX}
$ flutter-elinux create exampleapp
$ cd exampleapp
$ flutter-elinux build elinux --target-arch=arm64

See also: https://github.com/sony/meta-flutter#cross-building-using-yocto-sdk

atornqvist · 2022-04-06T06:09:49Z

@HidenoriMatsubayashi

Thanks for the response.

Unfortunately it makes no difference. The linking still goes wrong.

HidenoriMatsubayashi · 2022-04-06T07:34:23Z

@atornqvist

=====> Linker error since it is the Yocto x86_64 ld that is used and not the cross-build ld.

Could you inform me of the error messages?

atornqvist · 2022-04-06T08:12:41Z

@HidenoriMatsubayashi . Yes, sure.
Below is the last part.
It seems like the flutter build is not using the cross clang++ and then that leads to the error with ld.

CMake Error at
/scratch/opt/poky/3.1.15/sysroots/x86_64-pokysdk-linux/usr/share/cmake-3.16/Modules/CMakeTestCXXComp
iler.cmake:53 (message):
  The C++ compiler

    "/scratch/opt/poky/3.1.15/sysroots/x86_64-pokysdk-linux/usr/bin/clang++"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/anders/flutter_tests/exampleapp/build/elinux/arm64/release/CMakeFiles/CMakeTmp

    Run Build Command(s):/usr/bin/make cmTC_5e7f1/fast && /usr/bin/make -f
    CMakeFiles/cmTC_5e7f1.dir/build.make CMakeFiles/cmTC_5e7f1.dir/build
    make[1]: Entering directory
    '/home/anders/flutter_tests/exampleapp/build/elinux/arm64/release/CMakeFiles/CMakeTmp'
    Building CXX object CMakeFiles/cmTC_5e7f1.dir/testCXXCompiler.cxx.o
    /scratch/opt/poky/3.1.15/sysroots/x86_64-pokysdk-linux/usr/bin/clang++
    --target=aarch64-poky-linux --sysroot=/scratch/opt/poky/3.1.15/sysroots/aarch64-poky-linux
    -O2 -pipe -g -feliminate-unused-debug-types     -o
    CMakeFiles/cmTC_5e7f1.dir/testCXXCompiler.cxx.o -c
    /home/anders/flutter_tests/exampleapp/build/elinux/arm64/release/CMakeFiles/CMakeTmp/testCXXComp
    iler.cxx
    Linking CXX executable cmTC_5e7f1
    /scratch/opt/poky/3.1.15/sysroots/x86_64-pokysdk-linux/usr/bin/cmake -E cmake_link_script
    CMakeFiles/cmTC_5e7f1.dir/link.txt --verbose=1
    /scratch/opt/poky/3.1.15/sysroots/x86_64-pokysdk-linux/usr/bin/clang++
    --target=aarch64-poky-linux --sysroot=/scratch/opt/poky/3.1.15/sysroots/aarch64-poky-linux   -O2
    -pipe -g -feliminate-unused-debug-types    -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed
    -fstack-protector-strong -Wl,-z,relro,-z,now  CMakeFiles/cmTC_5e7f1.dir/testCXXCompiler.cxx.o
    -o cmTC_5e7f1
    /scratch/opt/poky/3.1.15/sysroots/x86_64-pokysdk-linux/usr/bin/ld: unrecognised emulation mode:
    aarch64linux
    Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu elf_l1om elf_k1om i386pep
    i386pe
    clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
    make[1]: *** [CMakeFiles/cmTC_5e7f1.dir/build.make:87: cmTC_5e7f1] Error 1
    make[1]: Leaving directory
    '/home/anders/flutter_tests/exampleapp/build/elinux/arm64/release/CMakeFiles/CMakeTmp'
    make: *** [Makefile:121: cmTC_5e7f1/fast] Error 2





  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:2 (project)

topolittle · 2022-04-06T11:26:59Z

Here is what I done to patch the Yocto's SDK to make it works with flutter-elinux.
After installing the SDK, I have done the following:

export TARGET_ARCH="aarch64-poky-linux" && \
. /opt/fsl-imx-wayland/5.4-zeus/environment-setup-$TARGET_ARCH && \
echo 'set ( CMAKE_C_COMPILER_WORKS 1 )' >> $OECORE_NATIVE_SYSROOT/usr/share/cmake/OEToolchainConfig.cmake && \
echo 'set ( CMAKE_C_COMPILER_TARGET "'$TARGET_ARCH'" )' >> $OECORE_NATIVE_SYSROOT/usr/share/cmake/OEToolchainConfig.cmake && \
echo 'set ( CMAKE_CXX_COMPILER_WORKS 1 )' >> $OECORE_NATIVE_SYSROOT/usr/share/cmake/OEToolchainConfig.cmake && \
echo 'set ( CMAKE_CXX_COMPILER_TARGET "'$TARGET_ARCH'" )' >> $OECORE_NATIVE_SYSROOT/usr/share/cmake/OEToolchainConfig.cmake && \
mv $OECORE_NATIVE_SYSROOT/usr/bin/ld $OECORE_NATIVE_SYSROOT/usr/bin/ld.x86_64 && \
cp -a $OECORE_NATIVE_SYSROOT/usr/bin/$TARGET_ARCH/${TARGET_PREFIX}ld $OECORE_NATIVE_SYSROOT/usr/bin/ld

In a nutshell, it disables the compiler test for C and C++, it sets the target architecture and it swaps the x86_64 and the ARM64 linker since the SDK is using the wrong linker.

I am running with this fix for months without any issues.

HidenoriMatsubayashi · 2022-04-07T00:08:23Z

@topolittle

Thank you very much. I'd like to add the doc how to build with Yocto SDK in flutter-elinux. Could you summarize how to build? I'll copy and paste your document into here.

LefixBlue · 2022-04-07T03:54:17Z

Hi @HidenoriMatsubayashi @topolittle @atornqvist ,

as far as i remember (I built everything for/with imx-yocto zeus release 2 months ago) I just had to patch flutter-elinux to match the compiler-triple used in Yocto:

The file:
flutter-elinux/lib/elinux_build_target.dart

The patched two lines:
'-DCMAKE_C_COMPILER_TARGET=aarch64-poky-linux',
'-DCMAKE_CXX_COMPILER_TARGET=aarch64-poky-linux',

And ensure a rebuild of flutter-elinux afterwards!

Then set the Yocto-SDK environment and build the application with flutter-elinux

I havn't tried it with yocto-dunfell yet.
But I wonder what could cause this issue as it was really that easy when using yocto-zeus release.

So the time I built it, I thought it would be enough to let flutter-elinux allow to give the compiler-triple (in our case: 'aarch64-poky-linux') as a cmd-line argument.

HidenoriMatsubayashi · 2022-04-07T06:48:36Z

@LefixBlue Thank you for your comment.

So the time I built it, I thought it would be enough to let flutter-elinux allow to give the compiler-triple (in our case: 'aarch64-poky-linux') as a cmd-line argument.

I'll create a patch! Please watch sony/flutter-elinux#48

HidenoriMatsubayashi · 2022-04-07T07:36:38Z

Now, I added target-compiler-triple option for cross-building. sony/flutter-elinux#88

LefixBlue · 2022-04-07T07:44:05Z

@HidenoriMatsubayashi

That looks good to me!

I will try the new option in my environment as soon as I'll have some free time.

atornqvist · 2022-04-07T09:25:40Z

@HidenoriMatsubayashi @LefixBlue

Looks good.

I made a quick test. It does not make any difference in my build.

What makes it build is the changing of the ld as @topolittle suggest with:

mv $OECORE_NATIVE_SYSROOT/usr/bin/ld $OECORE_NATIVE_SYSROOT/usr/bin/ld.x86_64 && \
cp -a $OECORE_NATIVE_SYSROOT/usr/bin/$TARGET_ARCH/${TARGET_PREFIX}ld $OECORE_NATIVE_SYSROOT/usr/bin/ld

What I can see in the build is that it is "clang++" that is used and not "aarch64-poky-linux-clang++". Then it will be the clang++ in the SDK directory that will used and that uses the x86_64 "ld".

topolittle · 2022-04-07T13:18:03Z

@atornqvist @HidenoriMatsubayashi
In the Yocto SDK, the cross-compiler are in the <native_sysroot_folder>/usr/bin/<cpu_arch>/ and the native compilers are in <native_sysroot_folder>/usr/bin/

For the triplets, is not possible to move the clang compiler binaries like I do for the linker. The compiler in both folder are the same binaries and when invoked, they behave according to their filename, somewhat like busybox do. Setting CMAKE_C_COMPILER_TARGET and CMAKE_CXX_COMPILER_TARGET in the toolchain cmake configuration file make it use the right compiler path and filename.

atornqvist · 2022-05-10T06:06:23Z

Hi, the compiler triplet does not seem to be used for CXX since flutter-elinux/lib/elinux_build_target.dart line"environment: <String, String>{'CC': 'clang', 'CXX': 'clang++'}" sets it to "clang++".

Deleteing that line makes the cross build work fine with Poky dunfell.

No changes in the toolchain cmake file is needed as long as the environment is set with this:
export CC=${CLANGCC}
export CXX=${CLANGCXX}

As reference the situation can be reproduced with:

# Build a dunfell Poky SDK:
git clone git://git.yoctoproject.org/poky.git -b dunfell
pushd poky && git checkout 64f632c93f487004e721ec7b4ae61b817157037b && popd
git clone https://github.com/kraj/meta-clang -b dunfell
pushd meta-clang && git checkout 3bb001d3f364bbf6588fed04b9ee2c7e74a0beba && popd
git clone https://github.com/sony/meta-flutter.git
pushd meta-flutter && git checkout 5818f865c5a7df2955b6de60bc1cdfd8b2ef7a2e && popd

source poky/oe-init-build-env build

cat >> conf/local.conf << EOF
MACHINE ?= "qemuarm64"
CLANGSDK = "1"
EOF

bitbake-layers add-layer ../meta-clang
bitbake-layers add-layer ../meta-flutter

bitbake core-image-weston -c populate_sdk

tmp/deploy/sdk/poky-glibc-x86_64-core-image-weston-aarch64-qemuarm64-toolchain-3.1.16.sh

export SDK_INSTALL_PATH=/opt/poky/3.1.16

git clone https://github.com/sony/flutter-elinux.git
pushd flutter-elinux && git checkout e3b9d8e9ca676f49ad7febdca49f54dbd2f462a8 && popd

export TARGET_ARCH="aarch64-poky-linux" && \
. ${SDK_INSTALL_PATH}/environment-setup-$TARGET_ARCH

# Remove flutter-elinux line that sets CXX to a native clang++
# Doing this makes the cross build work.
# Skip this line to reproduce the build error
sed -i "/environment: <String, String>{'CC': 'clang', 'CXX': 'clang++'},/d" flutter-elinux/lib/elinux_build_target.dart

export CC=${CLANGCC}
export CXX=${CLANGCXX}

export PATH="$PATH:`pwd`/flutter-elinux/bin"
flutter-elinux precache
flutter-elinux create exampleapp
cd exampleapp
flutter-elinux build elinux --target-arch=arm64 --target-compiler-triple=$TARGET_ARCH --target-sysroot=$SDK_INSTALL_PATH/sysroots/aarch64-poky-linux/ -v

HidenoriMatsubayashi · 2022-05-10T13:20:04Z

@atornqvist Perhaps, Case 2: Use Yocto SDK helps you.

atornqvist · 2022-05-10T13:47:29Z

The moving of ld is not required if the flutter-elinux/lib/elinux_build_target.dart is modified as above. Then the CXX environment variable can be used to specify the appropriate clang++ that contains the toolchain triple for cross-building which will result in a working usage of ld.

Masterxiechengxun · 2022-07-20T16:16:22Z

Thank you!!! I have the same problem as you.Executing the followings you mentioned solves my problem.
$ sed -i "/environment: <String, String>{'CC': 'clang', 'CXX': 'clang++'},/d" /opt/flutter-elinux/lib/elinux_build_target.dart

gencay3535 · 2022-08-18T11:27:32Z

Has anyone ever tried flutter with stm32mp15 on kirkstone branch? I've been trying for days but could not able to build an sdk for my flutter application.

justdoGIT · 2022-09-12T12:30:00Z

Hi @ALL
while running flutter app in imx8m mini I am getting this error:

root@imx8mm-var-dart:~/my_app# flutter-client -b /home/root/my_app/build/elinux/arm64/release/bundle/
[FATAL:flutter/runtime/dart_vm_initializer.cc(22)] Error while initializing the Dart VM: Wrong full snapshot version, expected '1441d6b13b8623fa7fbf61433abebd31' found 'b0e899ec5a90e4661501f0b69e9dd70f'
Aborted

Flutter version

root@imx8mm-var-dart:~/my_app# flutter-elinux run -d elinux-wayland --release
Woah! You appear to be trying to run flutter as root.
We strongly recommend running the flutter tool without superuser privileges.
/
📎
Flutter 3.3.1 • channel unknown • unknown source
Framework • revision 4f9d92fbbd (5 days ago) • 2022-09-06 17:54:53 -0700
Engine • revision 3efdf03e73
Tools • Dart 2.18.0 • DevTools 2.15.0
Launching lib/main.dart on eLinux in release mode...
Building an eLinux application with wayland backend in release mode for arm64 target... 245.3s

Flutter run key commands.
h List all available interactive commands.
c Clear the screen
q Quit (terminate the application on the device).

Application finished.

HidenoriMatsubayashi · 2022-09-12T12:51:40Z

Please try:

$ rm -r <path_to_flutter-elinux>/flutter/bin/cache/
$ flutter doctor

justdoGIT · 2022-09-13T06:06:21Z

Hi, after removing the flutter/bin/cache directory and then running $ flutter doctor command I am facing the same issue. The output is as shown below:

root@imx8mm-var-dart:/opt/flutter-elinux/flutter/bin/cache# rm -rf *
root@imx8mm-var-dart:/opt/flutter-elinux/flutter/bin/cache# ls
root@imx8mm-var-dart:/opt/flutter-elinux/flutter/bin/cache# cd ..
root@imx8mm-var-dart:/opt/flutter-elinux/flutter/bin# ls
cache dart dart.bat flutter flutter.bat internal
root@imx8mm-var-dart:/opt/flutter-elinux/flutter/bin# ./flutter doctor
Woah! You appear to be trying to run flutter as root.
We strongly recommend running the flutter tool without superuser privileges.
/
📎
Downloading Linux arm64 Dart SDK from Flutter engine 3efdf03e73dd352873fc5a7e1c0de04e52863b3a...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 118M 100 118M 0 0 5799k 0 0:00:20 0:00:20 --:--:-- 9116k
Building flutter tool...
Downloading Material fonts... 2,108ms
Downloading Gradle Wrapper... 50ms
Downloading package sky_engine... 547ms
Downloading flutter_patched_sdk tools... 1,617ms
Downloading flutter_patched_sdk_product tools... 1,025ms
Downloading linux-arm64 tools... 5.9s
Downloading linux-arm64/font-subset tools... 799ms
Doctor summary (to see all details, run flutter doctor -v):
[!] Flutter (Channel unknown, 3.3.1, on FSLC Wayland with XWayland 3.1 (dunfell) 5.4.142+gb84afa992407, locale en_US)
! Flutter version 3.3.1 on channel unknown at /opt/flutter-elinux/flutter
! Upstream repository unknown
[✗] Chrome - develop for the web (Cannot find Chrome executable at google-chrome)
! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.
[✗] Linux toolchain - develop for Linux desktop
✗ ninja is required for Linux development.
It is likely available from your distribution (e.g.: apt install ninja-build), or can be downloaded from
https://github.com/ninja-build/ninja/releases
✗ GTK 3.0 development libraries are required for Linux development.
They are likely available from your distribution (e.g.: apt install libgtk-3-dev)
[!] Flutter IDE Support (No supported IDEs installed)
[✓] Connected device (1 available)
[✓] HTTP Host Availability

! Doctor found issues in 4 categories.

root@imx8mm-var-dart:~/my_app# flutter-client -b /home/root/my_app/build/elinux/arm64/release/bundle/
[FATAL:flutter/runtime/dart_vm_initializer.cc(22)] Error while initializing the Dart VM: Wrong full snapshot version, expected '1441d6b13b8623fa7fbf61433abebd31' found 'b0e899ec5a90e4661501f0b69e9dd70f'
Aborted

Steps to reproduce:

Create fsl-image-gui yocto image with clang and cmake.
Clone flutter-elinux in arm64 board.
run $ flutter-elinux devices
run $ flutter-elinux create my app
run $ flutter-elinux run -d elinux-wayland --release
run $ flutter-client -b /home/root/my_app/build/elinux/arm64/release/bundle

Please let me know if anything else is required from my side.

justdoGIT · 2022-09-13T06:50:47Z

Hi, after removing the flutter/bin/cache directory and then running $ flutter doctor command I am facing the same issue. The output is as shown below:

root@imx8mm-var-dart:/opt/flutter-elinux/flutter/bin/cache# rm -rf * root@imx8mm-var-dart:/opt/flutter-elinux/flutter/bin/cache# ls root@imx8mm-var-dart:/opt/flutter-elinux/flutter/bin/cache# cd .. root@imx8mm-var-dart:/opt/flutter-elinux/flutter/bin# ls cache dart dart.bat flutter flutter.bat internal root@imx8mm-var-dart:/opt/flutter-elinux/flutter/bin# ./flutter doctor Woah! You appear to be trying to run flutter as root. We strongly recommend running the flutter tool without superuser privileges. / paperclip Downloading Linux arm64 Dart SDK from Flutter engine 3efdf03e73dd352873fc5a7e1c0de04e52863b3a... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 118M 100 118M 0 0 5799k 0 0:00:20 0:00:20 --:--:-- 9116k Building flutter tool... Downloading Material fonts... 2,108ms Downloading Gradle Wrapper... 50ms Downloading package sky_engine... 547ms Downloading flutter_patched_sdk tools... 1,617ms Downloading flutter_patched_sdk_product tools... 1,025ms Downloading linux-arm64 tools... 5.9s Downloading linux-arm64/font-subset tools... 799ms Doctor summary (to see all details, run flutter doctor -v): [!] Flutter (Channel unknown, 3.3.1, on FSLC Wayland with XWayland 3.1 (dunfell) 5.4.142+gb84afa992407, locale en_US) ! Flutter version 3.3.1 on channel unknown at /opt/flutter-elinux/flutter ! Upstream repository unknown [✗] Chrome - develop for the web (Cannot find Chrome executable at google-chrome) ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable. [✗] Linux toolchain - develop for Linux desktop ✗ ninja is required for Linux development. It is likely available from your distribution (e.g.: apt install ninja-build), or can be downloaded from https://github.com/ninja-build/ninja/releases ✗ GTK 3.0 development libraries are required for Linux development. They are likely available from your distribution (e.g.: apt install libgtk-3-dev) [!] Flutter IDE Support (No supported IDEs installed) [✓] Connected device (1 available) [✓] HTTP Host Availability

! Doctor found issues in 4 categories.

root@imx8mm-var-dart:~/my_app# flutter-client -b /home/root/my_app/build/elinux/arm64/release/bundle/ [FATAL:flutter/runtime/dart_vm_initializer.cc(22)] Error while initializing the Dart VM: Wrong full snapshot version, expected '1441d6b13b8623fa7fbf61433abebd31' found 'b0e899ec5a90e4661501f0b69e9dd70f' Aborted

Steps to reproduce:

Create fsl-image-gui yocto image with clang and cmake.

Clone flutter-elinux in arm64 board.

run $ flutter-elinux devices

run $ flutter-elinux create my app

run $ flutter-elinux run -d elinux-wayland --release

run $ flutter-client -b /home/root/my_app/build/elinux/arm64/release/bundle

Please let me know if anything else is required from my side.

Also while running $ flutter-elinux build elinux I am getting these warnings/ errors as shown below:

root@imx8mm-var-dart:~/my_app# flutter-elinux build elinux
   Woah! You appear to be trying to run flutter as root.
   We strongly recommend running the flutter tool without superuser privileges.
  /
📎
Flutter 3.3.1 • channel unknown • unknown source
Framework • revision 4f9d92fbbd (6 days ago) • 2022-09-06 17:54:53 -0700
Engine • revision 3efdf03e73
Tools • Dart 2.18.0 • DevTools 2.15.0
Downloading linux-arm64/linux-arm64-flutter-gtk tools...            6.1s
Downloading linux-arm64-profile/linux-arm64-flutter-gtk tools...         4.2s
Downloading linux-arm64-release/linux-arm64-flutter-gtk tools...         3.4s
Downloading an artifact that may not be reachable in some environments (e.g. firewalled environments):
https://github.com/sony/flutter-embedded-linux/releases/download/3efdf03e73/elinux-common.zip
This should not have happened. This is likely a Flutter SDK bug. Please file an issue at
https://github.com/flutter/flutter/issues/new?template=1_activation.md
Downloading elinux-common tools...                                     ⣯Downloading elinux-common tools...                                    2,995ms
Downloading an artifact that may not be reachable in some environments (e.g. firewalled environments):
https://github.com/sony/flutter-embedded-linux/releases/download/3efdf03e73/elinux-arm64-debug.zip
This should not have happened. This is likely a Flutter SDK bug. Please file an issue at
https://github.com/flutter/flutter/issues/new?template=1_activation.md
Downloading elinux-arm64-debug tools...                                ⣯Downloading elinux-arm64-debug tools...                                  5.6s
Downloading an artifact that may not be reachable in some environments (e.g. firewalled environments):
https://github.com/sony/flutter-embedded-linux/releases/download/3efdf03e73/elinux-arm64-profile.zip
This should not have happened. This is likely a Flutter SDK bug. Please file an issue at
https://github.com/flutter/flutter/issues/new?template=1_activation.md
Downloading elinux-arm64-profile tools...                              ⣯Downloading elinux-arm64-profile tools...                                3.3s
Downloading an artifact that may not be reachable in some environments (e.g. firewalled environments):
https://github.com/sony/flutter-embedded-linux/releases/download/3efdf03e73/elinux-arm64-release.zip
This should not have happened. This is likely a Flutter SDK bug. Please file an issue at
https://github.com/flutter/flutter/issues/new?template=1_activation.md
Downloading elinux-arm64-release tools...                              ⣯Downloading elinux-arm64-release tools...                             2,304ms
Downloading an artifact that may not be reachable in some environments (e.g. firewalled environments):
https://github.com/sony/flutter-embedded-linux/releases/download/3efdf03e73/elinux-x64-debug.zip
This should not have happened. This is likely a Flutter SDK bug. Please file an issue at
https://github.com/flutter/flutter/issues/new?template=1_activation.md
Downloading elinux-x64-debug tools...                                  ⣯Downloading elinux-x64-debug tools...                                    5.9s
Downloading an artifact that may not be reachable in some environments (e.g. firewalled environments):
https://github.com/sony/flutter-embedded-linux/releases/download/3efdf03e73/elinux-x64-profile.zip
This should not have happened. This is likely a Flutter SDK bug. Please file an issue at
https://github.com/flutter/flutter/issues/new?template=1_activation.md
Downloading elinux-x64-profile tools...                                ⣯Downloading elinux-x64-profile tools...                                 12.6s
Downloading an artifact that may not be reachable in some environments (e.g. firewalled environments):
https://github.com/sony/flutter-embedded-linux/releases/download/3efdf03e73/elinux-x64-release.zip
This should not have happened. This is likely a Flutter SDK bug. Please file an issue at
https://github.com/flutter/flutter/issues/new?template=1_activation.md
Downloading elinux-x64-release tools...                                ⣯Downloading elinux-x64-release tools...                                  7.0s

💪 Building with sound null safety 💪

Building an eLinux application with wayland backend in release mode for arm64 target...        255.9s
⡿root@imx8mm-var-dart:~/my_app#

Let me know if these warnings are okay or not

justdoGIT · 2022-09-13T12:11:33Z

Hi,
I was able to run the flutter application finally. The cause was flutter-engine was using dunfell branch while the flutter sdk was using master branch. So checkout 3.0.5 tag in flutter sdk resolved the issue for me

HidenoriMatsubayashi mentioned this issue Mar 11, 2021

Change output log level when the off-screen surface creation failed #20

Merged

HidenoriMatsubayashi mentioned this issue Mar 12, 2021

unordered_map header file is missing #22

Merged

HidenoriMatsubayashi added the help wanted Extra attention is needed label Mar 12, 2021

HidenoriMatsubayashi mentioned this issue Mar 12, 2021

Switch from wl_shell to xdg_shell (doesn't work on Wayland compositors other than weston) #25

Closed

LefixBlue mentioned this issue Mar 23, 2021

Add sleep to wait for the next event in the main loop properly #43

Closed

HidenoriMatsubayashi mentioned this issue Apr 17, 2021

Fix GBM color format #78

Merged

HidenoriMatsubayashi mentioned this issue Apr 7, 2022

Building flutter app with Yocto SDK. sony/flutter-elinux#85

Closed

toygg96 mentioned this issue Apr 26, 2022

Cross building for arm64 on x64 host issues sony/flutter-elinux#94

Closed

HidenoriMatsubayashi added good first issue Good for newcomers question Further information is requested labels Aug 9, 2023

Test on arm devices #4

Test on arm devices #4

Comments

HidenoriMatsubayashi commented Mar 9, 2021 • edited

Tested devices

HidenoriMatsubayashi commented Mar 9, 2021

ghost commented Mar 9, 2021

psstoyanov commented Mar 10, 2021

HidenoriMatsubayashi commented Mar 11, 2021 • edited

psstoyanov commented Mar 11, 2021

psstoyanov commented Mar 11, 2021 • edited

psstoyanov commented Mar 11, 2021

LefixBlue commented Mar 12, 2021 • edited

HidenoriMatsubayashi commented Mar 12, 2021

HidenoriMatsubayashi commented Mar 12, 2021 • edited

LefixBlue commented Mar 12, 2021

LefixBlue commented Mar 12, 2021

psstoyanov commented Mar 12, 2021

psstoyanov commented Mar 14, 2021

LefixBlue commented Mar 15, 2021

HidenoriMatsubayashi commented Mar 15, 2021

LefixBlue commented Mar 15, 2021

HidenoriMatsubayashi commented Mar 15, 2021

HidenoriMatsubayashi commented Mar 16, 2021 • edited

psstoyanov commented Mar 17, 2021

LefixBlue commented Apr 13, 2021

HidenoriMatsubayashi commented Apr 13, 2021

LefixBlue commented Apr 13, 2021

HidenoriMatsubayashi commented Apr 14, 2021 • edited

topolittle commented Jan 7, 2022

topolittle commented Jan 10, 2022

LefixBlue commented Jan 11, 2022

atornqvist commented Apr 5, 2022 • edited

HidenoriMatsubayashi commented Apr 6, 2022

atornqvist commented Apr 6, 2022

HidenoriMatsubayashi commented Apr 6, 2022

atornqvist commented Apr 6, 2022 • edited

topolittle commented Apr 6, 2022

HidenoriMatsubayashi commented Apr 7, 2022

LefixBlue commented Apr 7, 2022

HidenoriMatsubayashi commented Apr 7, 2022

HidenoriMatsubayashi commented Apr 7, 2022

LefixBlue commented Apr 7, 2022

atornqvist commented Apr 7, 2022

topolittle commented Apr 7, 2022

atornqvist commented May 10, 2022 • edited

HidenoriMatsubayashi commented May 10, 2022

atornqvist commented May 10, 2022

Masterxiechengxun commented Jul 20, 2022 • edited

gencay3535 commented Aug 18, 2022

justdoGIT commented Sep 12, 2022

HidenoriMatsubayashi commented Sep 12, 2022

justdoGIT commented Sep 13, 2022

justdoGIT commented Sep 13, 2022

justdoGIT commented Sep 13, 2022

HidenoriMatsubayashi commented Mar 9, 2021 •

edited

HidenoriMatsubayashi commented Mar 11, 2021 •

edited

psstoyanov commented Mar 11, 2021 •

edited

LefixBlue commented Mar 12, 2021 •

edited

HidenoriMatsubayashi commented Mar 12, 2021 •

edited

HidenoriMatsubayashi commented Mar 16, 2021 •

edited

HidenoriMatsubayashi commented Apr 14, 2021 •

edited

atornqvist commented Apr 5, 2022 •

edited

atornqvist commented Apr 6, 2022 •

edited

atornqvist commented May 10, 2022 •

edited

Masterxiechengxun commented Jul 20, 2022 •

edited