[FR] enable onnxruntime-directml support on windows

[ccddos]

RT, 提供原生gpu支持，甚至可以调用核显, 并且依赖大小比较小，其他平台可以fallback到cpu

[SWHL]

我是第一次知道onnxruntime-directml，不是太明白。
如果可以的话，欢迎展开说说。或者提交PR都是很欢迎的。

[ccddos]

我是第一次知道onnxruntime-directml，不是太明白。 如果可以的话，欢迎展开说说。或者提交PR都是很欢迎的。

这是gemini的参考回答 https://sdk.vercel.ai/s/1XYJ7spaktu3LaB35oTjG
简单来说就是可以在windows上调用gpu，包括核显 , amd， 不只是cuda。 而且依赖项比较小，支持windows10及以上。

win7不太清楚，但我觉得可以fallback到其他provider之类的

按我之前其他python项目的集成经验，只需要把依赖项从onnxruntime 改到 onnxruntime-directml

这是一段参考代码

    @classmethod
    @time_it
    def setup_model(cls, model_path: str):
        provider = ['DmlExecutionProvider', 'CPUExecutionProvider']
        so = ort.SessionOptions()
        so.enable_mem_pattern = False
        try:
            if cls.ort_sess:
                del cls.ort_sess
            cls.ort_sess = ort.InferenceSession(
                model_path, providers=provider, sess_options=so)
            model_warm_up(cls.ort_sess, cls.hps)

            cls.model_is_ok = True
            # model_warm_up(cls.ort_sess, cls.hps)
        except Exception as e:
            logger.error(e)
            raise e

gpt总结

ONNX Runtime 是一个开源的机器学习推理引擎，支持在各种平台和设备上部署训练好的模型。ONNX Runtime-DirectML 是其一个组件，利用 DirectML 库为 Windows 系统上的 ONNX 模型推理提供 GPU 加速，带来显著的性能提升和更广泛的硬件支持。

使用 DirectML 的主要优势包括：

• 加速推理： 比 CPU 执行速度更快
• 提高效率： 优化资源利用
• 广泛兼容性： 支持各种 GPU
• 易于使用： 简化集成过程

总而言之，ONNX Runtime-DirectML 为 Windows 用户提供了高效便捷的 ONNX 模型推理解决方案，尤其适合需要 GPU 加速的机器学习任务。

[ccddos]

测试时间 2024.5.14

ONNX Runtime-DirectML patch 已经合并到主分支。

以下是复现步骤:
0. 配置信息

win+r: 运行dxdiag

测试图片

2. 使用conda或mamba创建python3.11虚拟环境
3. 参考以下脚本运行

git clone https://github.com/RapidAI/RapidOCR.git

cd RapidOCR/python

pip install -r requirements_ort.txt

cp /path/to/testimg.png ./testimg.png

python -m rapidocr_onnxruntime.main --img ./testimg.png

hyperfine --warmup 3   "python -m rapidocr_onnxruntime.main --img  ./testimg.png"


pip uninstall  onnxruntime-directml

pip install onnxruntime

hyperfine --warmup 3   "python -m rapidocr_onnxruntime.main --img  ./testimg.png"

4. 测试结果

(p311) D:\work\RapidOCR\python>hyperfine --warmup 3   "python -m rapidocr_onnxruntime.main --img  ./testimg.png"
Benchmark 1: python -m rapidocr_onnxruntime.main --img  ./testimg.png
  Time (mean ± σ):      2.873 s ±  0.101 s    [User: 1.798 s, System: 0.470 s]
  Range (min … max):    2.725 s …  3.043 s    10 runs


(p311) D:\work\RapidOCR\python>pip uninstall  onnxruntime-directml
Found existing installation: onnxruntime-directml 1.17.3
Uninstalling onnxruntime-directml-1.17.3:
  Would remove:
    d:\scoop\scoop\persist\mambaforge\envs\p311\lib\site-packages\onnxruntime\*
    d:\scoop\scoop\persist\mambaforge\envs\p311\lib\site-packages\onnxruntime_directml-1.17.3.dist-info\*
    d:\scoop\scoop\persist\mambaforge\envs\p311\scripts\onnxruntime_test.exe
Proceed (Y/n)? y
  Successfully uninstalled onnxruntime-directml-1.17.3

(p311) D:\work\RapidOCR\python>pip install onnxruntime
Collecting onnxruntime
  Using cached onnxruntime-1.17.3-cp311-cp311-win_amd64.whl.metadata (4.6 kB)
Requirement already satisfied: coloredlogs in d:\scoop\scoop\apps\mambaforge\current\envs\p311\lib\site-packages (from onnxruntime) (15.0.1)
Requirement already satisfied: flatbuffers in d:\scoop\scoop\apps\mambaforge\current\envs\p311\lib\site-packages (from onnxruntime) (24.3.25)
Requirement already satisfied: numpy>=1.24.2 in d:\scoop\scoop\apps\mambaforge\current\envs\p311\lib\site-packages (from onnxruntime) (1.26.4)
Requirement already satisfied: packaging in d:\scoop\scoop\apps\mambaforge\current\envs\p311\lib\site-packages (from onnxruntime) (24.0)
Requirement already satisfied: protobuf in d:\scoop\scoop\apps\mambaforge\current\envs\p311\lib\site-packages (from onnxruntime) (5.26.1)
Requirement already satisfied: sympy in d:\scoop\scoop\apps\mambaforge\current\envs\p311\lib\site-packages (from onnxruntime) (1.12)
Requirement already satisfied: humanfriendly>=9.1 in d:\scoop\scoop\apps\mambaforge\current\envs\p311\lib\site-packages (from coloredlogs->onnxruntime) (10.0)
Requirement already satisfied: mpmath>=0.19 in d:\scoop\scoop\apps\mambaforge\current\envs\p311\lib\site-packages (from sympy->onnxruntime) (1.3.0)
Requirement already satisfied: pyreadline3 in d:\scoop\scoop\apps\mambaforge\current\envs\p311\lib\site-packages (from humanfriendly>=9.1->coloredlogs->onnxruntime) (3.4.1)
Using cached onnxruntime-1.17.3-cp311-cp311-win_amd64.whl (5.6 MB)
Installing collected packages: onnxruntime
Successfully installed onnxruntime-1.17.3

(p311) D:\work\RapidOCR\python>hyperfine --warmup 3   "python -m rapidocr_onnxruntime.main --img  ./testimg.png"
Benchmark 1: python -m rapidocr_onnxruntime.main --img  ./testimg.png
  Time (mean ± σ):      5.256 s ±  0.146 s    [User: 25.242 s, System: 8.866 s]
  Range (min … max):    5.081 s …  5.563 s    10 runs

[SWHL]

是我看错了吗？从测试结果来看，onnxruntime-directml推理时间 1.798s，测试结果下面onnxruntime部分，其推理时间是25.242s。

onnxruntime-directml看着是有提速的呀

[ccddos]

我是pr的，不是提问的 @SWHL
在这个disscussion中留一个参考

后续有性能疑问请参考这个步骤复现

[SWHL]

哈哈， 我的确看错了，以为还是刚才小伙伴提问的呢。我在之前PR哪里留了这个地址，他应该会看到。

[SkeathyTomas]

不是很懂hyperfine，user/system是什么意思？
配置：win11，i5-10210U，python3.12
onnxruntime-directml版本

C:\Users\skeat\Desktop>hyperfine --warmup 3 "rapidocr_onnxruntime --img 1.png"
Benchmark 1: rapidocr_onnxruntime --img 1.png
  Time (mean ± σ):      4.933 s ±  0.916 s    [User: 1.084 s, System: 0.190 s]
  Range (min … max):    4.579 s …  7.537 s    10 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (7.537 s). This could be caused by (filesystem) caches that were not filled until after the first run. You are already using the '--warmup' option which helps to fill these caches before the actual benchmark. You can either try to increase the warmup count further or re-run this benchmark on a quiet system in case it was a random outlier. Alternatively, consider using the '--prepare' option to clear the caches before each timing run.

onnxruntime版本

C:\Users\skeat\Desktop>hyperfine --warmup 3 "rapidocr_onnxruntime --img 1.png"
Benchmark 1: rapidocr_onnxruntime --img 1.png
  Time (mean ± σ):      5.944 s ±  0.019 s    [User: 10.608 s, System: 2.174 s]
  Range (min … max):    5.923 s …  5.991 s    10 runs

考虑到缓存加载第一次反而慢了？
自己的程序因为每次都会重新加载所以更慢了？

[ccddos]

gpt的回答 https://poe.com/s/8dlRciKj6xH770ZMRivB

我个人觉得是偶然误差？ 我这边测了几次，都没测出你的那个warning，你要不先把其他程序关了再测

[SkeathyTomas]

回家用自己机子又试了下，好像还和测试图有关，自己跑的项目确实慢了。
配置：win11, R7-4800u, python3.11

测试图1

1.3.17

(py311) C:\Users\skeathy\OneDrive\keqing>hyperfine "rapidocr_onnxruntime --img src/grab.png"
Benchmark 1: rapidocr_onnxruntime --img src/grab.png
  Time (mean ± σ):      2.053 s ±  0.041 s    [User: 5.499 s, System: 1.567 s]
  Range (min … max):    2.013 s …  2.148 s    10 runs

1.3.18

(py311) C:\Users\skeathy\OneDrive\keqing>hyperfine "rapidocr_onnxruntime --img src/grab.png"     
Benchmark 1: rapidocr_onnxruntime --img src/grab.png
  Time (mean ± σ):      3.478 s ±  0.189 s    [User: 1.773 s, System: 0.185 s]
  Range (min … max):    3.263 s …  3.966 s    10 runs

测试图2

1.3.17

(py311) C:\Users\skeathy\Desktop>hyperfine "rapidocr_onnxruntime --img 1.png"
Benchmark 1: rapidocr_onnxruntime --img 1.png
  Time (mean ± σ):      5.517 s ±  0.066 s    [User: 29.129 s, System: 4.945 s]
  Range (min … max):    5.411 s …  5.670 s    10 runs

1.3.18

(py311) C:\Users\skeathy\Desktop>hyperfine "rapidocr_onnxruntime --img 1.png"
Benchmark 1: rapidocr_onnxruntime --img 1.png
  Time (mean ± σ):      3.731 s ±  0.061 s    [User: 0.968 s, System: 0.146 s]
  Range (min … max):    3.653 s …  3.863 s    10 runs

[SWHL]

感觉需要warmup之后测试一下，类似下面:
hyperfine --warmup 3 "rapidocr_onnxruntime --img 1.png"

[SkeathyTomas]

区别不大，还是图1，warmup应该是针对大量IO排除掉缓存影响的
1.3.17

(py311) C:\Users\skeathy\OneDrive\keqing>hyperfine --warmup 3 "rapidocr_onnxruntime --img src/grab.png"
Benchmark 1: rapidocr_onnxruntime --img src/grab.png
  Time (mean ± σ):      2.030 s ±  0.042 s    [User: 5.816 s, System: 1.727 s]
  Range (min … max):    1.973 s …  2.105 s    10 runs

1.3.18

(py311) C:\Users\skeathy\OneDrive\keqing>hyperfine --warmup 3 "rapidocr_onnxruntime --img src/grab.png"
Benchmark 1: rapidocr_onnxruntime --img src/grab.png
  Time (mean ± σ):      3.255 s ±  0.016 s    [User: 1.748 s, System: 0.177 s]
  Range (min … max):    3.234 s …  3.278 s    10 runs

[ccddos]

@SkeathyTomas 最近有点事，可能晚点debug，看结果你的场景，还是先用1.3.17把。 dml 理论上应该是可以的，我这边也verfiy过了

[ccddos]

@SkeathyTomas 方便加个qq私聊吗，感觉你的情况有点trick, 应该不是调参能解决的，我想问一些配置信息 base64 qq号 UVEgMjM1Njk2OTM2

[HighDoping]

Per my testing (with AMD Ryzen 7 Pro 4750G, Vega 8, Windows 10 19045), the GPU is not recogized by onnxruntime-directml, and only shows CPU-DML provider. The inference time is about 40% longer than using CPU provider in onnxruntime-cpu. Uninstalling onnxruntime-directml may leave some unwanted files and mess up with onnxruntime.
Maybe more testing with different GPU is needed.

[SWHL]

@HAL9000COM H
I will add prompts for installing onnxruntime-gpu in the appropriate places in the next version.
Thanks for pointing it out. 👍

[SkeathyTomas]

@SWHL some weird issues in v1.3.19.

1. Uninstall onnxruntime and install onnxruntime-directml
   ModuleNotFoundError: No module named 'onnxruntime-directml'
2. Install onnxruntime and onnxruntime-directml will lose DMLProvider.

C:\Users\skeat\Desktop>python
Python 3.12.3 (tags/v3.12.3:f6650f9, Apr  9 2024, 14:05:25) [MSC v.1938 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import onnxruntime
>>> print(onnxruntime.get_available_providers())
['AzureExecutionProvider', 'CPUExecutionProvider']

3. Uninstall all and install rapidocr-onnxruntime 1.3.18 will find DMLProvider back.

C:\Users\skeat\Desktop>python
Python 3.12.3 (tags/v3.12.3:f6650f9, Apr  9 2024, 14:05:25) [MSC v.1938 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import onnxruntime
>>> print(onnxruntime.get_available_providers())
['DmlExecutionProvider', 'CPUExecutionProvider']

My conclusion: only install rapidocr and onnxruntime-directml clearly can use dml.

[SWHL]

I found that it is probably because after onnxruntime and onnxruntime-directml are installed, their module names are both onnxruntime, and they can be imported through import onnxruntime.

[SkeathyTomas]

@SWHL v1.3.20还有点问题，没有正常使用GPU，OrtInferSession跑了3次，EP_list后两次没有DmlExecutionProvider

(py312) C:\Users\skeathy\Desktop>rapidocr_onnxruntime -img 1.png --rec_use_dml --det_use_dml
2024-05-18 20:33:52,519 - OrtInferSession - INFO: Windows 10 or above detected, try to use DirectML as primary provider
[('DmlExecutionProvider', {'arena_extend_strategy': 'kSameAsRequested'}), ('CPUExecutionProvider', {'arena_extend_strategy': 'kSameAsRequested'})]
[('CPUExecutionProvider', {'arena_extend_strategy': 'kSameAsRequested'})]
[('CPUExecutionProvider', {'arena_extend_strategy': 'kSameAsRequested'})]

[SWHL]

@SkeathyTomas 感觉这个是onnxruntime-directml的问题了。
我这里手头没有Windows设备，你那里可以帮忙复现和尝试修复一下吗？感谢感谢。

[ok-oldking]

import onnxruntime as ort
device = ort.get_device()
print(f"Using device: {device}")

输出:
Using device: CPU-DML
我这个正常吗?是不是没用到GPU?
速度是比onnx cpu快点, 但是没openvino快

[ok-oldking]

onnxruntime-directml==1.8.0
因为1.8.1会报错找不到dll

[SWHL]

正常

[ok-oldking]

是已经用了GPU了 还是只能用CPU, 我是n卡4070用paddle的GPU是飞快, 这个dml还是只能跑CPU吗?

[SWHL]

从你这输出来看，应该是没有用GPU