Deploy the yolort on mobile by react-native-pytorch-core

[JohnZcp]

🐛 Describe the bug

This is the post follow the recent discussion about the problem of deploying yolort on mobile. Here is the link to the recent discussion: facebookresearch/playtorch#10 . I received the feedback from pytorch live developer:
'''
The TorchScripted yolort model fails inference with the following error:

terminating with uncaught exception of type c10::Error: isIntList()INTERNAL ASSERT FAILED at "../../../../src/main/cpp/libtorch_include/x86/ATen/core/ivalue_inl.h":1808, please report a bug to PyTorch. Expected IntList but got GenericList
Exception raised from toIntList at ../../../../src/main/cpp/libtorch_include/x86/ATen/core/ivalue_inl.h:1808 (most recent call first):
(no backtrace available)" failed'
'''

It seems like these is a problem the datatype conflict between IntList and GenericList, and it is able to be solved from either sides. Is there anyone familiar with this error? I would like to deliver the message between yolort team and Pytorch Mobile team.

Versions

Collecting environment information...
PyTorch version: 1.10.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0
Libc version: glibc-2.26

Python version: 3.7.13 (default, Mar 16 2022, 17:37:17) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: False
CUDA runtime version: 11.1.105
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] torch==1.10.0+cu111
[pip3] torchaudio==0.10.0+cu111
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.11.0
[pip3] torchvision==0.11.1+cu111
[conda] Could not collect

[zhiqwang]

Hi @JohnZcp , Thanks for reporting this problem to us. And is there any minimal codes to reproduce this bug here? As such we could resolve this issue more quickly.

[JohnZcp]

@zhiqwang I just got the response, and the developer will share repo in this post soon.

[raedle]

Hi @zhiqwang, @JohnZcp reached out and asked if I could provide a repo for the crash when loading the yolo5s_scriptmodule.ptl with PyTorch Mobile 1.10.0.

I created a repro example based on the PyTorch Android HelloWorldApp demo. You can access the GitHub repo raedle/android-demo-app and there the yolo5s branch.

https://github.com/raedle/c/tree/yolov5s/HelloWorldApp

The detailed steps are below or in the HelloWorldApp README.md.

Let me know if you have any questions!

Quickstart

HelloWorld is a simple image classification application that demonstrates how to use PyTorch Android API.
This application runs TorchScript serialized TorchVision pretrained MobileNet v3 model on static image which is packaged inside the app as android asset.

1. Model

The TorchScript yolort model is part of the repo in app/src/main/assets/yolo5s_scriptmodule.ptl.

2. Cloning from GitHub

git clone https://github.com/raedle/android-demo-app.git
cd HelloWorldApp

3. Build and install debug build

If Android SDK and Android NDK are already installed you can install this application to the connected android device or emulator with:

./gradlew installDebug

We recommend you to open this project in Android Studio 3.5.1+ (At the moment PyTorch Android and demo applications use android gradle plugin of version 3.5.0, which is supported only by Android Studio version 3.5.1 and higher),
in that case you will be able to install Android NDK and Android SDK using Android Studio UI.

4. Gradle dependencies

PyTorch Android is added to the HelloWorld as gradle dependencies in build.gradle:

repositories {
    jcenter()
}

dependencies {
    implementation 'org.pytorch:pytorch_android_lite:1.10.0'
    implementation 'org.pytorch:pytorch_android_torchvision_lite:1.10.0'
}

Where org.pytorch:pytorch_android is the main dependency with PyTorch Android API, including libtorch native library for all 4 android abis (armeabi-v7a, arm64-v8a, x86, x86_64).
Further in this doc you can find how to rebuild it only for specific list of android abis.

org.pytorch:pytorch_android_torchvision - additional library with utility functions for converting android.media.Image and android.graphics.Bitmap to tensors.

The JNI bits of the PyTorch Android dependency will not be used, but instead it extracts the PyTorch C++ frontend API (i.e., libpytorch_jni_lite.so) and uses the C++ API (see src/main/cpp/OnLoad.cpp).

5. Get Crash Report

The app will instacrash on start with the following error:

signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x7ac331b7e0

Start logcat to receive crash report

adb logcat --buffer=crash

Then run the ./gradlew installDebug command again.

6. When it does not crash anymore

If the model works with PyTorch Mobile, the app should not crash. Additionally, the module_.forward(inputs) in src/main/cpp/OnLoad.cpp should should return a sensible result.

auto output = module_.forward(inputs);

7. Alternative Model

Check that alternative model works. Go to app/src/main/java/org/pytorch/helloworld/MainActivity.java and replace

final String filePath = assetFilePath(this, "yolov5s_scriptmodule.ptl");

with

final String filePath = assetFilePath(this, "mobilenet_v3_small.ptl");

Rebuild debug build and install it again with

./gradlew installDebug

The app will load successfully without crashing

[JohnZcp]

@zhiqwang How is the progress? Did you find the reason for the bug?