Experimental WebGPU EP for node.js binding

[ExeVirus]

Hello, specifically @fs-eire,

Reading the current documentation on https://www.npmjs.com/package/onnxruntime-node,

1. I see that webGPU support is "experimental". So I try the @dev branch. Looks like webGPU is still not supported (this on a Nvidia 1660 Super with current drivers).

2. Then I try building from source, in release: build.bat --parallel --build_nodejs --config Release, take the resulting onnxruntime-common and onnxruntime-node dists/bin/lib and overwrite the contents of my test section (working with CPU EP no issue).

3. That still doesn't give webgpu as a "supportedDevice" deep in transformers.js. Then I see a commit here in the last two weeks by you related to how to build that webgpu binding by adding:

build.bat --parallel --build_nodejs --config Release --use_webgpu, currently running that build and seeing where it takes me...

4. Before I continue down this rabbit hole - would you expect Nvidia 1660 Super, Turing Architecture, to be enumerable and usable on a Windows 10** machine with latest Node and this binding (albeit with possible bugs related to "experimental")?

Thank you in advance,

 -ExeVirus

[ExeVirus]

The answer is:

I am using kokoro-js, which uses latest transformers-js, which by default is using onnxruntime-node version 1.21, which doesn't have this capability yet. So I update the package.json to override to 1.22, and now I have a likely working webgpu backend.

Except now my model is failing on both DML and webgpu backends in the same exact way, but still succeeding in the cpu backend....:

An error occurred during model execution: "Error: Non-zero status code returned while running DequantizeLinear node. Name:'DequantizeLinear' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\webgpu\program.cc:242 onnxruntime::webgpu::`anonymous-namespace'::GetRed
ucedShape shape.NumDimensions() > 0 && shape.GetDims()[shape.NumDimensions() - 1] % component == 0 was false. Cannot reduce shape {1,512,46} by component=4
".
Inputs given to model: {
  input_ids: {
    type: 'int64',
    dims: [ 1, 46 ],
    location: 'cpu',
    data: BigInt64Array(46) [
        0n, 50n, 83n,  54n, 156n,  57n, 135n,
        3n, 16n, 81n, 102n,  61n,  16n, 102n,
       68n, 16n, 70n,  16n,  62n, 156n,  86n,
       61n, 62n, 16n,  48n, 123n, 138n,  55n,
       81n, 83n, 16n,  53n,  83n,  55n, 156n,
       72n, 56n, 46n,  16n,  54n, 156n,  43n,
      102n, 56n,  4n,   0n
    ]
  },
  style: {
    type: 'float32',
    dims: [ 1, 256 ],
    location: 'cpu',
    data: Float32Array(256) [
       -0.22339411079883575,     0.163710355758667, -0.013009805232286453,
       -0.12385818362236023,  -0.27437737584114075,   0.02860426902770996,
      -0.059555958956480026,  -0.15487317740917206,  -0.26053789258003235,
       -0.16296090185642242,   0.02189430594444275,  -0.13216954469680786,
        0.31883466243743896,   0.19906209409236908,   0.28810837864875793,
        -0.3083314001560211,   -0.1061338409781456, -0.055566251277923584,
        0.08362426608800888,  -0.21951043605804443,   -0.2090497463941574,
       -0.19159287214279175,  0.017330599948763847,   0.05072310194373131,
      -0.009821131825447083,    0.1786910742521286,   -0.2452496737241745,
         0.1535724401473999,   0.04694422706961632,   0.04988643527030945,
         0.1460709124803543,  -0.09929021447896957,  0.040571488440036774,
      -0.044360220432281494,   -0.1218164935708046,   0.21403884887695312,
       -0.15888027846813202,  -0.22066539525985718,   0.14221134781837463,
         0.1803959161043167,  -0.05871300771832466,  0.023467302322387695,
      -0.034648433327674866,   0.17642895877361298,  -0.09464317560195923,
       -0.09218525141477585,   0.03665202856063843,   0.06863228231668472,
       -0.07507433742284775,   0.11187999695539474,   0.11986157298088074,
        0.06117059290409088,   0.20504866540431976,   0.24830861389636993,
      -0.018064403906464577,    0.2484755963087082,   0.22243762016296387,
       -0.07624555379152298, -0.023793527856469154,  -0.05118313059210777,
        0.24126644432544708,   -0.1573609858751297,  -0.21493594348430634,
       0.017533352598547935,   0.07216215133666992,    0.1749543994665146,
       0.012698079459369183,   0.04320225119590759,   0.09003549069166183,
       -0.02469470165669918,   0.11944770812988281,   0.07323233038187027,
        0.24487096071243286,  -0.17046435177326202,  -0.08503767102956772,
       -0.18016934394836426,  -0.20739471912384033,   0.22139210999011993,
        0.21467848122119904,  -0.06968650966882706,   0.09923344105482101,
       0.047699447721242905,   -0.1596110314130783,   0.18105240166187286,
        0.16261589527130127, 0.0004545344563666731,   -0.0456312857568264,
       -0.15743695199489594,  -0.25957584381103516,   -0.1838088482618332,
         0.1832876205444336,   -0.0683683529496193,  -0.04735468327999115,
        0.04575436934828758,   -0.2356453686952591,  -0.15800878405570984,
        0.03373640775680542,  -0.08166510611772537,  -0.10614915937185287,
        -0.0448690764605999,
      ... 156 more items
    ]
  },
  speed: {
    type: 'float32',
    dims: [ 1 ],
    location: 'cpu',
    data: Float32Array(1) [ 1 ]
  }
}
Error processing speech request: Error: Non-zero status code returned while running DequantizeLinear node. Name:'DequantizeLinear' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\webgpu\program.cc:242 onnxruntime::webgpu::`anonymous-namespace'::GetReducedShape
shape.NumDimensions() > 0 && shape.GetDims()[shape.NumDimensions() - 1] % component == 0 was false. Cannot reduce shape {1,512,46} by component=4

[fs-eire]

Hi @ExeVirus, thank you for the feedback. There are 2 different problems:

• webgpu is still not a "supportedDevice" in certain Transformer.js
  • my understanding is that it should already be there in latet transformer.js version (@xenova for confirm). you can also use package version override to use latest dev version of onnxruntime-node, and it should support webgpu.
• kokoro.js not working on webgpu.
  • didn't have a chance to take closer look. the error message shows that the size of input_ids should be of length that is times of 4, so you can try using 48 instead of 46 (padding 2 zeros)

[ExeVirus]

Thank you for the quick reply,

Did attempt lots of different input_ids sizes, doesn't have to be of multiples of 4 to cause the error (happens for all input ids).

@xenova , you did the work for converting https://huggingface.co/onnx-community/Kokoro-82M-v1.0-ONNX back in early February,

Can you either:

1. share your conversion steps - so I can debug further?
2. provide a good explanation as to why this kokoro onnx model only seems to support CPU execution from onnxruntime (both quantized and fp32) (I bet cuda on linux also works, but looking for webgpu and dml to work here).

Otherwise, looks like I'll be researching the pytorch dynamic exporter next.

[xenova]

my understanding is that it should already be there in latet transformer.js version (@xenova for confirm). you can also use package version override to use latest dev version of onnxruntime-node, and it should support webgpu.

It's currently in dev branch: huggingface/transformers.js#1306, but I'm hoping to put out a dev release on NPM soon.

the error message shows that the size of input_ids should be of length that is times of 4, so you can try using 48 instead of 46 (padding 2 zeros)

The model should work correctly with input_ids that aren't multiples of 4 (it works correctly on CPU), so this is probably a separate issue.

share your conversion steps - so I can debug further?

The conversion script is in the official repo: https://github.com/hexgrad/kokoro/blob/main/examples/export.py

Hope that helps!

[ExeVirus]

Thanks both,

The conversion script is in the official repo:

Thanks, found it about an hour after my last reply in this chain.

---

Working with another user on to fix the onnx export script on the official repo:

hexgrad/kokoro#79

The answer is the opset 17 that is being exported uses a convTranspose that is bugged in the DML backend (but working in CPU backend for opset 17):

https://github.com/hexgrad/kokoro/blob/2668b2e279d0f94977995230e523b0183763f30e/kokoro/custom_stft.py#L160

Through various methods, we were able to determine that converting the resulting exported onnx model from opset 17 to opset 22 (with onnx converter functions) works for some models (not all) and that does get a working dml backend execution with this model.

Taking a look at that issue, you'll see that the other user in that thread has made good progress at getting a more workable solution for getting the dml backend working for this model for all quantizations and versions. Should have a solid solution here soon.

Will continue dicussion there and not here - thanks again!

[ExeVirus]

@fs-eire

Further update, probably an actual bug in the webgpu implementation. I am converting the kokoro onnx model to different onnx opsets to try to avoid whatever error is baked into dml and webgpu backends, found a bug related to opset 11:

"Error: WebGPU validation failed. Error while parsing WGSL: :49:43 error: expected ';' for variable declaration
let outputIndices = o2i_output(global_idx)let batch = outputIndices[0];

Some sort of bug with autogeneration of that WGSL code.

[fs-eire]

I think I saw this error and it should have been fixed in latest dev build. Did you try overriding onnxruntime-node to 1.23.0-dev.20250509-3dc91e6c31?

[xenova]

Can confirm that 1.23.0-dev.20250509-3dc91e6c31 does fix this 👍