Feature request - Edge-ready models

[AliceCyber]

🚀 Feature

Your current models are well-suited for cloud/server inference, bun unsuited for edge (i.e. web/mobile) inference.

Problems

1. ONNX model isn't compatible with ONNX.js because of int64 weights/constants/computations.
2. TF model made by converting ONNX model (bad/messy graph) and isn't working after conversion to TF.js
3. Models are too large — 170+MB fp32 / 45MB int8 quantized, while they usually should be less than 10MB (for web) / 25MB (mobile app) for good user experience.

Possible solutions

• Convert any int64 tensors/constants in ONNX models to int32 (is it possible?) or wait for upstream fix Feature Request - Dealing with int64 in the exported ONNX model microsoft/onnxjs#233 ONNX Export - Dealing with INT64 Constants and Slices  pytorch/pytorch#45957.
• Remake model at least partially in TF or TF.js
• Release smaller models. Retrain/Distill/Prune?

Additional info

ONNX.js error: TypeError: int64 is not supported (failed to load model)
TF.js error: Error: Negative size values should be exactly -1 but got NaN for the slice() size at index 0. (multiple conversion runs, cpu backend, debug enabled)

[snakers4]

Hi,

Convert any int64 tensors/constants in ONNX models to int32 (is it possible?) or wait for upstream fix.
ONNX model isn't compatible with ONNX.js because of int64 weights/constants/computations.

Well, I have raised issues in pytorch/pytorch#45957 and microsoft/onnxjs#233. Thb, looks like an upstream problem to me, a similar issue was fixed in onnx-tensrorrt some time ago as well

Maybe someone proficient in JS can provide a quick fix / some monkey patching though

TF model made by converting ONNX model (bad/messy graph) and isn't working after conversion to TF.js
Remake model at least partially in TF or TF.js

Our main framework is PyTorch and we do not have enough time and resources to do this
But community contributions are always welcome (if anyone is willing to lend a hand, please ping me in Telegram)

Models are too large — 170+MB fp32 / 45MB int8 quantized, while they usually should be less than 10MB (for web) / 25MB (mobile app) for good user experience.

In the short-term it can be done via sacrificing model quality, but this contradicts our design philosophy - there should always be one model for any language
In the long-run we have set an aspirational goal to achieve ~20Mb
So it will be achieved in the long run

[snakers4]

Good news
In our research we discovered a way to make all models x2 smaller w/o sacrificing their quality
I am not sure about timelines yet
I have take this as a motivation to heavily refactor code as well

[snakers4]

The current aspirational targets for V2:

• Similar quality, maybe a slight degradation
• 20M params per model (now it's 50M)
• all models possible to quantize
• 2x shorter
• many minor optimizations
• all models unified (now they are a bit different)

[snakers4]

New experimental model is 25M quantized

[snakers4]

Most of these bullets are delivered
I hope I will be able to release V3 models eof this year

[snakers4]

#20 (comment)

Mostly approaching this goal
The smallest model is 40M (megabytes), though I was able to create a roughly equivalent model with only 15M params, but I could not quanitize it.

I am working on a model with 5-7M params as an experiment.

[snakers4]

All of these models already are very very fast.