Make Emotion2vec support onnx

[takipipo]

From the following feature requests

• ONNX? ddlBoJack/emotion2vec#55

• emotion2vec onnx #2291

• Amazing work did by thewh1teagle@586e81d

• Include LayerNorm to the Onnx model
  I have made emotion2vec exportable to onnx with dynamic audio sequence length. Sadly, I cannot include LayerNorm to the export_forward it produces this error

Error Log

python export.py
funasr version: 1.2.2.
Check update of funasr, and it would cost few times. You may disable it by set `disable_update=True` in AutoModel
You are using the latest version of funasr-1.2.2
Downloading Model to directory: /Users/kridtaphadsae-khow/.cache/modelscope/hub/iic/emotion2vec_base
2025-01-10 15:31:59,726 - modelscope - WARNING - Using branch: master as version is unstable, use with caution
Traceback (most recent call last):
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/export.py", line 9, in <module>
  res = model.export(type="onnx", quantize=False, opset_version=13, device='cpu')  # fp32 onnx-gpu
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/funasr/auto/auto_model.py", line 664, in export
  export_dir = export_utils.export(model=model, data_in=data_list, **kwargs)
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/funasr/utils/export_utils.py", line 22, in export
  _onnx(
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/funasr/utils/export_utils.py", line 80, in _onnx
  torch.onnx.export(
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/env/lib/python3.9/site-packages/torch/onnx/__init__.py", line 375, in export
  export(
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/env/lib/python3.9/site-packages/torch/onnx/utils.py", line 502, in export
  _export(
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/env/lib/python3.9/site-packages/torch/onnx/utils.py", line 1564, in _export
  graph, params_dict, torch_out = _model_to_graph(
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/env/lib/python3.9/site-packages/torch/onnx/utils.py", line 1117, in _model_to_graph
  graph = _optimize_graph(
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/env/lib/python3.9/site-packages/torch/onnx/utils.py", line 639, in _optimize_graph
  graph = _C._jit_pass_onnx(graph, operator_export_type)
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/env/lib/python3.9/site-packages/torch/onnx/utils.py", line 1836, in _run_symbolic_function
  return symbolic_fn(graph_context, *inputs, **attrs)
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/env/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 369, in wrapper
  return fn(g, *args, **kwargs)
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/env/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 264, in wrapper
  args = [
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/env/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 265, in <listcomp>
  _parse_arg(arg, arg_desc, arg_name, fn_name)  # type: ignore[method-assign]
File "/Users/kridtaphadsae-khow/Workspace/poc/FunASR/env/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 88, in _parse_arg
  raise errors.SymbolicValueError(
torch.onnx.errors.SymbolicValueError: Failed to export a node '%198 : Long(device=cpu) = onnx::Gather[axis=0](%195, %197), scope: funasr.models.emotion2vec.model.Emotion2vec:: # /Users/kridtaphadsae-khow/Workspace/poc/FunASR/funasr/models/emotion2vec/export_meta.py:33:0
' (in list node %204 : int[] = prim::ListConstruct(%198, %203), scope: funasr.models.emotion2vec.model.Emotion2vec::
) because it is not constant. Please try to make things (e.g. kernel sizes) static if possible.  [Caused by the value '204 defined in (%204 : int[] = prim::ListConstruct(%198, %203), scope: funasr.models.emotion2vec.model.Emotion2vec::
)' (type 'List[int]') in the TorchScript graph. The containing node has kind 'prim::ListConstruct'.] 

  Inputs:
      #0: 198 defined in (%198 : Long(device=cpu) = onnx::Gather[axis=0](%195, %197), scope: funasr.models.emotion2vec.model.Emotion2vec:: # /Users/kridtaphadsae-khow/Workspace/poc/FunASR/funasr/models/emotion2vec/export_meta.py:33:0
  )  (type 'Tensor')
      #1: 203 defined in (%203 : Long(device=cpu) = onnx::Gather[axis=0](%200, %202), scope: funasr.models.emotion2vec.model.Emotion2vec:: # /Users/kridtaphadsae-khow/Workspace/poc/FunASR/funasr/models/emotion2vec/export_meta.py:33:0
  )  (type 'Tensor')
  Outputs:
      #0: 204 defined in (%204 : int[] = prim::ListConstruct(%198, %203), scope: funasr.models.emotion2vec.model.Emotion2vec::
  )  (type 'List[int]')

How to export

from funasr import AutoModel
import onnxruntime as ort
import numpy as np

model = AutoModel(
    model="iic/emotion2vec_base",
    hub="ms"
)

export_dir = model.export(type="onnx", quantize=False, opset_version=13, device='cpu')
print(f"The onnx model is exported at {export_dir}")

session = ort.InferenceSession(f"{export_dir}/emotion2vec.onnx")
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name


x = np.random.randn(1, 32000).astype(np.float32)
y = session.run([output_name], {input_name: x})

print(y)

Feel free to review the code and give feedbacks 😁

[thewh1teagle]

@LauraGPT

We're almost done adding support for it. It looks like the only missing part is layer normalization during conversion.
Could you help with that, either by providing a fix or even just sharing some thoughts? It would really help! Thank you

[LauraGPT]

OK, I would have times to check it after 14 Jan.

[takipipo]

@LauraGPT @thewh1teagle In my assumption, I think the error occurs due to the dynamic computation from the layerNormalization and ONNX does not support this. I am thinking about replacing with the actual mathematical computation of the normalization since the following are equivalent.

import torch
import torch.nn.functional as F

x = torch.rand(1, 10)

x_norm_ref = F.layer_norm(x, x.shape)

mean = torch.mean(x, dim=1, keepdim=True)
var = torch.var(x, dim=1, keepdim=True, unbiased=False)
x_norm = (x - mean) / torch.sqrt(var + 1e-5)

torch.all(torch.isclose(x_norm, x_norm_ref))
>> tensor(True)

- x = F.layer_norm(x, x.shape)
+ mean = torch.mean(x, dim=1, keepdim=True)
+ var = torch.var(x, dim=1, keepdim=True, unbiased=False)
+ x = (x - mean) / torch.sqrt(var + 1e-5)
+ x = x.view(x.shape[0], -1)

[thewh1teagle]

OK, I would have times to check it after 14 Jan.

Did you able to check it? thanks

[LauraGPT]

I have tested it without any errors. Sorry, I do not know what the errors you ref to?

[thewh1teagle]

I have tested it without any errors. Sorry, I do not know what the errors you ref to?

I gave up on keep testing it. So thanks to @takipipo seems like it works now
Thanks

[LauraGPT]

I have tested it without any errors. Sorry, I do not know what the errors you ref to?

I gave up on keep testing it. So thanks to @takipipo seems like it works now Thanks

Thanks, I would check it and release a new version of funasr-1.2.3

[thewh1teagle]

Thanks, I would check it and release a new version of funasr-1.2.3

Not an issue in FunASR, but I will really appreciate if you could take a look on it
I exported onnx version of the emotion2vec model using export_onnx.py
And then I created emotion2onnx/init.py#L9
It's only 10-20 lines of code, but I have no idea how to map the extracted outputs correctly to the emotions. I tried to replicate the version from FunASR but still, I have no idea how exactly. the shape of the output is (1, 98, 768)
Any chance you can help with that? Or even give me direction where I maybe miss the important input/output code.
Thanks

https://github.com/thewh1teagle/emotion2onnx/blob/9e29972f93c74715940072fa0b4ac7b64acc871d/src/emotion2onnx/__init__.py#L9-L34