OWLv2 Evaluation Code Does Not Work!!

[rohit901]

Hello,

I'm following the evaluation instructions in https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit#evaluation

running the code, gives me this error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/rohit.bharadwaj/Projects/RNCDL/scenic/scenic/projects/owl_vit/evaluator.py", line 52, in <module>
    from scenic.projects.owl_vit import models
  File "/home/rohit.bharadwaj/Projects/RNCDL/scenic/scenic/projects/owl_vit/models.py", line 12, in <module>
    from scenic.projects.owl_vit import matching_base_models
  File "/home/rohit.bharadwaj/Projects/RNCDL/scenic/scenic/projects/owl_vit/matching_base_models.py", line 9, in <module>
    from scenic.model_lib import matchers
  File "/home/rohit.bharadwaj/Projects/RNCDL/scenic/scenic/model_lib/matchers/__init__.py", line 25, in <module>
    from scenic.model_lib.matchers.sinkhorn import sinkhorn_matcher
  File "/home/rohit.bharadwaj/Projects/RNCDL/scenic/scenic/model_lib/matchers/sinkhorn.py", line 24, in <module>
    from ott.geometry import geometry
  File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/ott/__init__.py", line 15, in <module>
    from . import geometry, initializers, math, problems, solvers, tools, utils
  File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/ott/initializers/__init__.py", line 1, in <module>
    from . import linear, nn, quadratic
  File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/ott/initializers/nn/__init__.py", line 1, in <module>
    from . import initializers
  File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/ott/initializers/nn/initializers.py", line 21, in <module>
    class MetaInitializer(initializers.DefaultInitializer):
  File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/ott/initializers/nn/initializers.py", line 64, in MetaInitializer
    rng: jax.random.PRNGKeyArray = jax.random.PRNGKey(0),
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/jax/_src/deprecations.py", line 54, in getattr
    raise AttributeError(f"module {module!r} has no attribute {name!r}")
AttributeError: module 'jax.random' has no attribute 'PRNGKeyArray'

I think it maybe because of wrong jax, jaxlib versions? The error originates from the "ott-jax" package.

from the requirements.txt:
ott-jax<0.4.0

so, do we have to downgrade jax, jaxlib? please give the correct jax versions to use to verify the results.

[rohit901]

@mjlm could you please help? I think this should be easy version fix, using the current requirements.txt doesn't seem to work :(

[rohit901]

we have to use jax, and jaxlib version of 0.4.23 for the evaluation code to work, maybe you can hardcode the version in requirements?

Now getting new error in evaluator.py :/

file scenic/scenic/projects/owl_vit/evaluator.py", line 675, in main

    compilation_cache.set_cache_dir('/tmp/jax_compilation_cache')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'jax.experimental.compilation_cache.compilation_cache' has no attribute 'set_cache_dir'

[rishabh-akridata]

@mjlm could you please help? I think this should be easy version fix, using the current requirements.txt doesn't seem to work :(

I fixed this issue by changing PRNGKeyArray to PRNGKEY in the initializers.py file.

[rohit901]

Thanks for the reply, I ended up using one of the custom evaluation code shared earlier here in the issues.

[Mukil07]

@rohit901 What evaluation code did you use ? when i tried to run the evaluation code using this command

python -m scenic.projects.owl_vit.evaluator --alsologtostderr=true --platform=gpu --config=clip_b32 --checkpoint_path=gs://scenic-bucket/owl_vit/checkpoints/clip_vit_b32_b0203fc --output_dir=/tmp/evaluator
its creating the tfds lvis dataset and its taking very long time >1 day and gets killed. Could you please let me know how to evaluate on LVIS dataset ?

[rohit901]

@Mukil07
i referred this: #762

[DishantMewada]

I solved it by changing PRNGKeyArray to PRNGKEY in the initializers.py file.
#1043
You can see the steps there to know where the file is located and what to change.