rfc: support dynamic shape in graph api

[wuxun-zhang]

Description

Add dynamic shape support into Graph API.

Rendered version: link

[eralmual]

This will allow users to create a tensor at graph creation time without having any information of it's shape or even number of dimensions, so at execution stage the specific shape for the tensors can be defined. Is my understanding correct?

This would really help oneDNN Graph integration into ONNX Runtime for the oneDNN EP, since most models don't provide any shape information of a tensor at graph creation time, so we don't get a shape or the number of dimensions, just the data type. It is only at execution time when the inputs have been provided and we know their shapes that we can propagate the information throughout the graph.
This why we need to be able to specify the tensor shapes at execution in order to fully support dynamic shapes in ORT using oneDNN Graph.

[wuxun-zhang]

Hi @eralmual, thanks for the review.

This will allow users to create a tensor at graph creation time without having any information of it's shape or even number of dimensions, so at execution stage the specific shape for the tensors can be defined. Is my understanding correct?

DNNL_GRAPH_DYNAMIC_DIM is a placeholder to represent whether a dimension is dynamic or not. As mentioned in the rfc, number of dimensions (aka. ndims) is needed to allow check dynamic dimension at graph creation stage and skip partitioning if a backend cannot support dynamic shape compilation.

This would really help oneDNN Graph integration into ONNX Runtime for the oneDNN EP, since most models don't provide any shape information of a tensor at graph creation time, so we don't get a shape or the number of dimensions, just the data type. It is only at execution time when the inputs have been provided and we know their shapes that we can propagate the information throughout the graph. This why we need to be able to specify the tensor shapes at execution in order to fully support dynamic shapes in ORT using oneDNN Graph.

I see your points. With the existing graph API, there are two backends as of now: primitive based (DNNL) and graph compiler. Different supporting capabilities of dynamic shape are provided in each backend. oneDNN Graph wants to filter out those ops with dynamic shape inputs early if cannot support. Please also take a look at below section 3.2 Graph partitioning stage.

BTW, in ONNXRT, how dynamic is the input shape in typical scenarios? Will some certain dimension or all dimensions be dynamic? Will ndims also be dynamic?

[eralmual]

I see your points. With the existing graph API, there are two backends as of now: primitive based (DNNL) and graph compiler. Different supporting capabilities of dynamic shape are provided in each backend. oneDNN Graph wants to filter out those ops with dynamic shape inputs early if cannot support. Please also take a look at below section 3.2 Graph partitioning stage.

Currently we are using the primitive based backend but the graph backed has some advantages like automated op fusion and has the chance to simplify quite a lot of our code so we are very interested on the Graph backend.

BTW, in ONNXRT, how dynamic is the input shape in typical scenarios? Will some certain dimension or all dimensions be dynamic? Will ndims also be dynamic?

ORT has a few stages, the one where we claim parts of the graph does not provide any kind of information about the input shape, essentially they just ask something like: Do you support Softmax with fp32 I/O? If we do, we claim the op, at this stage that's all we know.
Later on when we are going to execute our part of the graph, ORT provides the inputs to the operators we claimed, at this point because we have the input we can get it's shape.
This means that at graph creation time we don't even know the ndims of the tensors, I might also mention that while we don't know the number of ndims this does not mean it is dynamic, it's just unknown util execution time, this happens to models with dynamic dimensions which is pretty much every model we work with.

[sanchitintel]

Hi @xiang1guo, we'd need this API in public oneDNN to support generic dynamic shape supporting kernels in oneDNN Graph.
We're fine with this initial implementation, but down the line, we would need a way to provide range information pertaining to possible input shapes.

For this API to be actually used, we'd need extensive op-coverage in backends, but backend support is orthogonal to this RFC.