Bug Fixes for PTQ and ACQ based OpenVINO Model Export and Added Test Cases

[srikesh-07]

📝 Description

✅ Fixes and Refactors

• Fixed [Bug]: ValueError while exporting model in OpenVINO format with PTQ and ACQ #2592 by refactoring the access pattern of ImageBatch from subscriptable access (e.g., batch['image']) to attribute-style access (e.g., batch.image).

---

✅ Addressing #2593: Metric Instantiation Issue in Model Export

• [Bug]: AttributeError while exporting the model in OpenVINO format with ACQ #2593 was caused because the metric parameter was directly passed as a string (str) from the CLI into the Model.export function.
• This behavior required the following major modifications:

Modifications to Address the Issue

1. Added metric as a subclass argument in AnomalibCLI

• Previously, the metric parameter was passed directly as a str in the CLI, causing failures when trying to instantiate it in Model.export.
• To solve this, the metric parameter is now added as a subclass argument in the AnomalibCLI.

2. Migrated metric from torchmetrics.Metric to anomalib.metrics.AnomalibMetric

• This PR also aims to migrate the metric's base class from torchmetrics.Metric to anomalib.metrics.AnomalibMetric for seamless integration with the Anomalib ecosystem.

3. Deferred Metric Instantiation in AnomalibCLI

• Since AnomalibMetric requires a mandatory fields parameter, metric instantiation has been deferred from AnomalibCLI.
• The metric is now instantiated within the Model.export function using the helper functions provided.

4. Added Helper Functions for Metric Instantiation

• Introduced helper functions for AnomalibMetric which is similar to AnomalibModel that allow metric instantiation from Model.export via:

  • String (str):

    --metric F1Score

  • Dictionary (dict) input in Python:

    {
        "classpath": "F1Score",
        "init_args": {
            "fields": ["pred_scores", "gt_label"]
        }
    }

  • Namespace (Namespace) passed in the pipeline via CLI.

5. Allowed Dynamic Field Configuration via CLI

• Users can now pass custom fields via CLI to specify which fields should be used for metric calculation:

  --metric.fields "['pred_scores', 'gt_label']"

6. Handled Missing Fields Gracefully

• In cases where users do not pass fields explicitly, the metric will use a placeholder:

  fields = [""]

• The fields will later be modified depending on the model type (e.g., segmentation or classification).

7. Modified ACQ Validation Function

• The validation function utilized for ACQ is modified to support AnomalibMetric

---

✅ Test Coverage Enhancements

• Added unit test cases for helper functions to instantiate the AnomalibMetric.
• Added integration test cases for OpenVINO export covering all compression types from:

  from anomalib.export.deploy import CompressionType

• The compression types tested include:

  Compression Type	Tested
  FP16	✅ Yes
  INT8	✅ Yes
  INT8_PTQ	✅ Yes
  INT8_ACQ	✅ Yes

---

✅ PR Summary

• Unblocks Model.export functionality by allowing automatic metric instantiation in OpenVINO export.
• Improves user experience by allowing users to pass metric fields via CLI or Python without manual intervention.
• Fixed bugs related to ImageBatch access by switching from subscriptable access (batch['image']) to attribute-style access (batch.image).
• Ensures seamless compatibility with AnomalibMetric.

---

🛠️ Fixes

• [Bug]: AttributeError while exporting the model in OpenVINO format with ACQ #2593
• [Bug]: ValueError while exporting model in OpenVINO format with PTQ and ACQ #2592
• [Task]: Add unit tests for Engine.export #2273

The CHANGELOG.md will be updated once the changes have been approved by the respective authors of the repository.

✨ Changes

Select what type of change your PR is:

• 🐞 Bug fix (non-breaking change which fixes an issue)
• 🔨 Refactor (non-breaking change which refactors the code base)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update
• 🔒 Security update

✅ Checklist

Before you submit your pull request, please make sure you have completed the following steps:

• 📋 I have summarized my changes in the CHANGELOG and followed the guidelines for my type of change (skip for minor changes, documentation updates, and test enhancements).
• 📚 I have made the necessary updates to the documentation (if applicable).
• 🧪 I have written tests that support my changes and prove that my fix is effective or my feature works (if applicable).

For more information about code review checklists, see the Code Review Checklist.

PS: This is my first PR to the Anomalib repository, and I sincerely hope that my contributions align with the standards and expectations set by the repository and its authors. I appreciate the time and effort of the authors who will be reviewing my PR, and I thank you in advance for considering my changes.

@samet-akcay @djdameln @ashwinvaidya17

Thank You

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[srikesh-07]

@samet-akcay @djdameln @ashwinvaidya17

I see the build of ReadtheDocs failed due to a hash mismatch during the pip installation. Can you please check out the issue?

Whenever you get a chance, could you please review my PR? I appreciate your feedback.

Thank you!

[ashwinvaidya17]

Thanks for the efforts. It is a really thorough PR and I appreciate detailed inline documentation. I am fine with the OpenVINO stuff but for the metrics part let's wait for @djdameln 's review as he is the expert here.

[ashwinvaidya17]

Did you mean without a placeholder?

[srikesh-07]

What I’m trying to convey is this:

Let’s say the user passes a metric via the CLI using --metrics f1_score but does not specify any field parameters (i.e., --metrics.fields is not provided). In this case, F1Score is instantiated with a placeholder ([""]) in the model.export method. Later, the metric’s fields are updated based on the model type in export_mixin.py.

I also noticed a couple of grammatical mistakes in an inline comment and a logger statement:

• In L367:
  Change "if Metric is initialized with placeholder" to if Metric was initialized with a placeholder."

• In L370:
  Modify the logger statement from:

  "The fields of metric are initialized empty. Setting it to model fields {metric.fields}"
  to:
  "Since the fields of the metric were initialized as empty, they are now being set to the model fields: {metric.fields}."

I hope this clarifies your question. Let me know if you have any further doubts—I’d be happy to help!

Thank you.

[ashwinvaidya17]

Maybe something for a later PR but having a_u_p_i_m_o etc, with underscore might not look nice to the readers

[srikesh-07]

Got it, I had the same thought. Can we assume that for single-word metrics, we capitalize the entire word, and for those with underscores, we use PascalCase? For example:

• "aupro" → "AUPRO"
• "auroc" → "AUROC"
• "dice" → "DICE" (DICE may be implemented in the future)
• "f1_score" → "F1Score"
• "min_max" → "MinMax"

Alternatively, we could map each metric in a hashmap within the __init__.py file under anomalib.metrics.

Let me know your thoughts on these approaches.

Thank you.