[BUG] Model group doesn't not deleted automatically with the last model version in that group

[martin-gaievski]

What is the bug?
Model group remains in the system after the last model version in that group is deleted. This contradicts documentation that says:

When you delete the last model version in a model group, that model group is automatically deleted from the index.

https://docs.opensearch.org/latest/ml-commons-plugin/api/model-apis/delete-model/

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Setup remote connector using these steps:

• Create model group
• Create model and reference the new model group, deploy the model
• Create connector that points to the new model
  I used following API and deployed remote Cohere model using flow framework, but you can do same manually:

POST /_plugins/_flow_framework/workflow?use_case=cohere_embedding_model_deploy&provision=true
{
    "create_connector.credential.key" : "<YOUR_API_KEY"
}

Check the all three items exist

2. Undeploy model, delete connector, delete model

POST /_plugins/_ml/models/iKa8uJsBH57jLXRJEA7F/_undeploy

DELETE /_plugins/_ml/model_groups/gKa3r5sBH57jLXRJ5Q4B

DELETE /_plugins/_ml/models/iKa8uJsBH57jLXRJEA7F

3. Check model_group, it's still present in the system

GET /_plugins/_ml/model_groups/_search
{
  "query": {
    "match_all": {}
  }
}

            {
                "_index": ".plugins-ml-model-group",
                "_id": "hqa8uJsBH57jLXRJEA6Y",
                "_version": 2,
                "_seq_no": 9,
                "_primary_term": 1,
                "_score": 1.0,
                "_source": {
                    "created_time": 1768330891415,
                    "access": "public",
                    "latest_version": 1,
                    "last_updated_time": 1768330891442,
                    "name": "Cohere english embed model",
                    "description": "cohere-embedding-model"
                }
            }

get by id request return the same document, so it's not the stale/cached data problem:

GET /_plugins/_ml/model_groups/hqa8uJsBH57jLXRJEA6Y
{
  "name": "Cohere english embed model",
  "latest_version": 1,
  "description": "cohere-embedding-model",
  "access": "public",
  "created_time": 1768330891415,
  "last_updated_time": 1768330891442
}

What is the expected behavior?
Model group is delete together with the single model accossiated with it

What is your host/environment?

• Version 3.3
• Plugins: flow-framework (optional, you can do everything manually)

[brianf-aws]

Documentation may be outdated or miscommunication could have occurred

Latest change to the delete model API stating cascading delete is Nov 2023

When you delete the last model version in a model group, that model group is automatically deleted from the index.
https://github.com/opensearch-project/documentation-website/blame/d5f8006a40b924cb3279b339a23826c058440003/_ml-commons-plugin/api/model-apis/delete-model.md

While the code change you brought up that omits the functionality is Oct 2023
#1444

For maintainers @mingshl , @dhrubo-os , @Zhangxunmt , @b4sjoo , @ylwu-amzn Is the expected / designed functionality what was written in code or what we wanted in the documentation?

[dhrubo-os]

Yes, I believe there was a customer requirement behind this behavior. If a customer sets up a model group with all the role and user configurations, and then the model group gets deleted
automatically upon the last model deletion, the customer would have to recreate all those configurations for a new model group. This creates a poor user experience where customers lose their carefully configured access controls. I think we should Update our documentation to reflect the current behavior (model groups persist)

@ylwu-amzn please correct me if I am wrong about the customer requirement here.

[martin-gaievski]

I see scenarios when both options are needed. Wouldn't be better to have current behavior untouched and create a new optional flag/param that user can set at DELETE model API, and perform cascade deletion of the model_group. This will make system flexible and still satisfy existing customer requirement

[dbwiddis]

I think we should Update our documentation to reflect the current behavior (model groups persist)

Yes, the docs should be updated, but this is possibly a bigger issue, a (small) resource-leak.

When we create a model without specifying a model group, that group is automatically created with the model's name.

Automatically creating a document in an index and not automatically deleting it clutters up the model group index with unused model groups, without the user really even knowing they need to clean them up (or offering them an easy way to do so).