DPTForDepthEstimation class could not be matched

[pratos]

Hi, I was trying out depth estimation model via bumblebee (below is the code block):

{:ok, midas} = Bumblebee.load_model(
  {:hf, "Intel/dpt-hybrid-midas",
  cache_dir: bumblebee_cache_dir,
  auth_token: hf_token
  }
)

But I got an error:

** (RuntimeError) could not match the class name "DPTForDepthEstimation" to any of the supported models, please specify the :module and :architecture options
    (bumblebee 0.1.2) lib/bumblebee.ex:287: Bumblebee.load_spec/2
    (bumblebee 0.1.2) lib/bumblebee.ex:401: Bumblebee.load_model/2
    /Users/prthamesh/pratos-workbook.livemd#cell:vempkrw5zvjaonanodmtiiyi3xed4cza:1: (file)

I'm assuming this error is coz bumblebee is not on parity with huggingface libraries. Is there any way I can directly import .ckpt models or a way to contribute to bumblebee wrt the depth estimation model?

[josevalim]

Hi @pratos! I believe you are correct, we would need to implement the model in Bumblebee. You can look at how other models are implemented for guidance. Perhaps resnet as a starting point?

[jonatanklosko]

Correct, what's missing is the implementation of DPT (it is based on ViT, which we do have, so that's a good reference) and then a featurizer (matching DPTFeatureExtractor in hf/transformers) :)

[pratos]

Thanks for the info 😄! I'd see if I can get time to dive into this. I just started learning Elixir & don't have much experience writing pytorch arch from scratch! But would be an interesting exercise for me to go through 😀

Will checkout resnet & ViT in bumblebee first!

[jonatanklosko]

Moved to #188.