Flexible shape input and output doesn't work at runtime

[h-shib]

I'm trying to run super resolution CoreML model which takes 50...300 x 50...300 image for input and 100...600 x 100...600 image for output.

I could convert .mlmodel as like above from the model which takes 100x100 for input and 200x200 for output with coremltools==2.0. However, I couldn't run the model for any other than 100x100 size of image on Xcode (and iPhone6s device).

this is error code.

2018-10-22 19:55:18.369748+0000 super-resolution-1[1885:331653] [espresso] [Espresso::handle_ex_plan] exception=Invalid X-dimension 100/200 status=-7
2018-10-22 19:55:18.369867+0000 super-resolution-1[1885:331653] [coreml] Error binding image input buffer input_image: -7
2018-10-22 19:55:18.369977+0000 super-resolution-1[1885:331653] [coreml] Failure in bindInputsAndOutputs.
CoreML Model Error: Error Domain=com.apple.CoreML Code=0 "Error binding image input buffer input_image." UserInfo={NSLocalizedDescription=Error binding image input buffer input_image.}

Xcode console shows the model has flexible input.

Also, spec.description.input and spec.description.output returns the results below.

What's wrong?

[aseemw]

Are you using the vision API or CoreML API directly ?
It would be great if you can file a bug at https://developer.apple.com/bug-reporting/ with the code to reproduce the issue. Thanks!

[h-shib]

Sorry for late reply. @aseemw

I found a cause of problem.
I use Pixel Shuffle layer in my model and CoreML implements it as combination of Reshape and Transpose as this documentation says.
https://github.com/onnx/onnx-coreml/blob/master/onnx_coreml/_transformers.py#L325

However, When I convert mlmodel to have flexible input shape, intermediate reshape layer crashes since it has static targetShape like this.

I tried to delete targetShape but it didn't help.
Is there any way to allow flexible targetShape?

[aseemw]

CoreML's reshape layer is a static layer, so the target shape must be fully specified.
I guess the other option I see is to use a custom CoreML layer to implement the shuffle Layer.

[h-shib]

OK, thank you.
I'll try to implement it.

[lgyStoic]

@h-shib do you fix that problem yet?

[h-shib]

@lgyStoic well, actually not yet.
I've decided to use Metal Performance Shader directly for my model after all.
Metal has MPSCNNSubPixelConvolutionDescriptor for pixel shuffle (or subpixel upsampling) layer. So I used it.

I think CoreML doesn't have implemented Pixel shuffle layer yet, so you still need to implement it in custom layer.

[lgyStoic]

@h-shib I also try MPSCNNSubPixelConvolutionDescriptor by using MPNNGraph, but got a uncorrect image , has a bit difference with pytorch result. can you have a look on this? https://github.com/lgyStoic/super_resolution_MPSNNGraph

[BorisKourt]

@lgyStoic, @h-shib and @aseemw I am having the same error in the issue linked above. But I do not appear to be using a Pixel Shuffle layer. Is there a way to diagnose which part of the model might be having this issue?

The specific runtime error looks exactly like yours:

Finalizing CVPixelBuffer 0x282f4c5a0 while lock count is 1.
[espresso] [Espresso::handle_ex_plan] exception=Invalid X-dimension 1/480 status=-7
[coreml] Error binding image input buffer input_image: -7
[coreml] Failure in bindInputsAndOutputs.

More detail here and here

[enoreyes]

Has anyone found a solution to this? I also have the same error as @h-shib. I also have a Pixel Shuffle layer. I'm also getting a similar error to @BorisKourt

[BorisKourt]

@enoreyes unfortunately I haven't been able to sort it out. If you take a look at the comments on the SO question there is some more info: Here

Have you had any luck yourself?