ONN runtime on all CPU cores

[talhaanwarch]

Greetings. My CPU has 12 cores, is there a way to multi thread or multiprocess the following code to speed up the inference.

session = InferenceSession("onnx/model-quantized.onnx")
tokens=tokenizer(list(val_df.tweet),max_length=token_size,padding='max_length',truncation=True, return_tensors="np")
outputs = session.run(output_names=["logits"], input_feed=dict(tokens))

[talhaanwarch]

I figured this out

sess_options = rt.SessionOptions()
sess_options.intra_op_num_threads = 24
sess_options.execution_mode = rt.ExecutionMode.ORT_PARALLEL
sess_options.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_ALL

session = rt.InferenceSession("onnx/model-quant.onnx",sess_options=sess_options)

[altair199797]

This can actually worsen latency, depending on your model.
So better check that.
As far as I know, there seems to be no way to utilize several cores with ONNX for many models (especially backbones and other one stream architectures).