check failure of TensorRT 8.5 when running trtexec on Orin

[natureXD]

Description

I use the command to transfer ONNX model to trt on Orin:
/lib/bin/trtexec --onnx=bevformer_tiny_epoch_24_cp.onnx --int8 --saveEngine=bevformer_tiny_epoch_24_cp_int8.trt --plugins=libtensorrt_ops.so --workspace=1024 --verbose
but I met the following error:
[05/24/2023-06:53:31] [E] Error[2]: [shuffleBuilder.cpp::writeGlob::154] Error Code 2: Internal Error (Assertion outputScalesLen >= quantizations.outputs[0].scale.count() failed. ) [05/24/2023-06:53:31] [E] Error[2]: [builder.cpp::buildSerializedNetwork::743] Error Code 2: Internal Error (Assertion engine != nullptr failed. ) [05/24/2023-06:53:31] [E] Engine could not be created from network [05/24/2023-06:53:31] [E] Building engine failed [05/24/2023-06:53:31] [E] Error[3]: [builder.cpp::~Builder::299] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/builder.cpp::~Builder::299, condition: mObjectCounter.use_count() == 1. Destroying a builder object before destroying objects it created leads to undefined behavior.
I transfer the same model on 3060, it works well.
here are my onnx file and the transfer log on Orin and 3060.
https://github.com/natureXD/MyRepoFiles/blob/master/bevformer_tiny_epoch_24_cp.onnx
onnx2trt_3060_int8.log
onnx2trt_Orin_int8.log

Can you tell me how could I solve the problem on the Orin and why the problem occurs?

Environment

Orin平台信息：

• Selected Device: Orin
• CUDA Version: 11.4
• TensorRT Version: 8510

3060平台信息：

• Selected Device: 3060
• CUDA Version: 11.6
• TensorRT Version: 8503

Operating System: Linux arm64(Orin), Linux x86_64(3060)

Relevant Files

https://github.com/natureXD/MyRepoFiles/blob/master/bevformer_tiny_epoch_24_cp.onnx

Steps To Reproduce

/lib/bin/trtexec --onnx=bevformer_tiny_epoch_24_cp.onnx --int8 --saveEngine=bevformer_tiny_epoch_24_cp_int8.trt --plugins=libtensorrt_ops.so --workspace=1024 --verbose

Tasks

Preview Give feedback

No tasks being tracked yet.

[zerollzeng]

You will have to provide the code for libtensorrt_ops.so

Could you please try the latest 8.6.1 first on x86? I've seen a similar bug but it's been fixed.

[natureXD]

@zerollzeng
Hi, the code for libtensorrt_ops.so is here: https://github.com/DerryHub/BEVFormer_tensorrt/tree/main/TensorRT
It works well on x86. The assert error occurs on the Orin. Additionally, it seems that the version of TensorRT on Orin is newer than x86, but it still has problems.

[zerollzeng]

If you can upgrade and test on 6.0.7.0(TRT 8.6.10), I think it should be fixed.

[natureXD]

If you can upgrade and test on 6.0.7.0(TRT 8.6.10), I think it should be fixed.

@zerollzeng
Thank you very much for your patient answer.
I have checked the version, but the newest version of DRIVE OS is 6.0.6(the version I am using), not 6.0.7 is available, the link is https://developer.nvidia.com/drive/downloads.

[zerollzeng]

Please kindly wait for the new release :-)

[ttyio]

closing since no activity for more than 3 weeks, pls reopen if you still have question, thanks!

[lix19937]

also occur in drive os 6060

[lix19937]

Internal Error (Assertion outputScalesLen >= quantizations.outputs[0].scale.count() failed. )

It seems a bug.

[lix19937]

ref #3904