Tenant Aware Integration Tests #3516

dbwiddis · 2025-02-08T19:45:57Z

Description

Adds integration tests for multitenancy.

The test classes were squashed and cherry-picked from #2818.

Presently the search model and search agent calls are commented out, as MLSearchHandler and TransportSearchAgentAction classes are still using client.search() rather than SDKClient implementations.

Fixed in remaining sdk client changes for search operations #3522

Check List

New functionality includes testing.
New functionality has been documented.
~~[ ] API changes companion pull request created.~~
Commits are signed per the DCO using --signoff.
~~[ ] Public documentation issue/PR created.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2025-02-10T17:41:31Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.98%. Comparing base (ed4f09f) to head (bd40be1).
Report is 9 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3516      +/-   ##
============================================
- Coverage     78.20%   77.98%   -0.22%     
- Complexity     7162     7315     +153     
============================================
  Files           631      655      +24     
  Lines         32248    32997     +749     
  Branches       3666     3708      +42     
============================================
+ Hits          25219    25734     +515     
- Misses         5465     5676     +211     
- Partials       1564     1587      +23

Flag	Coverage Δ
ml-commons	`77.98% <ø> (-0.22%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dbwiddis · 2025-03-19T02:12:29Z

Rebased to current main (Lucene hits breaking change).

Note: commit 41353eb will currently fail integ tests, will be fixed by #3671.

Signed-off-by: Daniel Widdis <[email protected]>

dhrubo-os · 2025-05-01T17:34:43Z

I noticed these tests run for 1 hour! Is there any way to reduce time? I'm just wondering if this will add extra development headache or not.

dbwiddis · 2025-05-01T17:59:00Z

I noticed these tests run for 1 hour! Is there any way to reduce time? I'm just wondering if this will add extra development headache or not.

These tenant aware tests don't run that long. The one hour is just the existing CI tests; closer to 30 minutes but there's a built-in retry if/when it fails. Here are the timestamps for the Tenant-aware tests:

Initial Run:

RestMLAgentTenantAwareIT: 15:59:22,244 to 15:59:36,864 = 14.62 seconds
RestMLConnectorTenantAwareIT: 09:59:36,906 to 09:59:37,227 = 0.321 seconds
RestMLModelGroupTenantAwareIT: 00:07:42,160 to 00:07:47,810 = 5.65 seconds
RestMLModelTenantAwareIT: 17:07:47,833 to 17:07:48,460 = 0.627 seconds
RestMLModelUndeployTenantAwareIT: 23:07:48,487 to 23:07:48,947 = 0.46 seconds

2025-04-30T22:59:22.2870818Z RestMLAgentTenantAwareIT > testAgentCRUD STANDARD_OUT
2025-04-30T22:59:22.2871694Z     [2025-04-30T15:59:22,244][INFO ][o.o.m.r.RestMLAgentTenantAwareIT] [testAgentCRUD] before test
2025-04-30T22:59:22.2873806Z     [2025-04-30T15:59:22,251][INFO ][o.o.m.r.RestMLAgentTenantAwareIT] [testAgentCRUD] initializing REST clients against [http://[::1]:40731, http://127.0.0.1:37761]
2025-04-30T22:59:36.8868911Z     [2025-04-30T15:59:36,864][INFO ][o.o.m.r.RestMLAgentTenantAwareIT] [testAgentCRUD] after test

2025-04-30T22:59:36.9870522Z RestMLConnectorTenantAwareIT > testConnectorCRUD STANDARD_OUT
2025-04-30T22:59:36.9871828Z     [2025-05-01T09:59:36,906][INFO ][o.o.m.r.RestMLConnectorTenantAwareIT] [testConnectorCRUD] before test
2025-04-30T22:59:36.9874500Z     [2025-05-01T09:59:36,911][INFO ][o.o.m.r.RestMLConnectorTenantAwareIT] [testConnectorCRUD] initializing REST clients against [http://[::1]:40731, http://127.0.0.1:37761]
2025-04-30T22:59:37.2871387Z     [2025-05-01T09:59:37,227][INFO ][o.o.m.r.RestMLConnectorTenantAwareIT] [testConnectorCRUD] after test

2025-04-30T23:07:42.1874946Z RestMLModelGroupTenantAwareIT > testModelGroupCRUD STANDARD_OUT
2025-04-30T23:07:42.1875957Z     [2025-05-01T00:07:42,160][INFO ][o.o.m.r.RestMLModelGroupTenantAwareIT] [testModelGroupCRUD] before test
2025-04-30T23:07:42.1877547Z     [2025-05-01T00:07:42,163][INFO ][o.o.m.r.RestMLModelGroupTenantAwareIT] [testModelGroupCRUD] initializing REST clients against [http://[::1]:40731, http://127.0.0.1:37761]
2025-04-30T23:07:47.8869400Z     [2025-05-01T00:07:47,810][INFO ][o.o.m.r.RestMLModelGroupTenantAwareIT] [testModelGroupCRUD] after test

2025-04-30T23:07:47.8870311Z RestMLModelTenantAwareIT > testModelCRUD STANDARD_OUT
2025-04-30T23:07:47.8870858Z     [2025-04-30T17:07:47,833][INFO ][o.o.m.r.RestMLModelTenantAwareIT] [testModelCRUD] before test
2025-04-30T23:07:47.8871830Z     [2025-04-30T17:07:47,837][INFO ][o.o.m.r.RestMLModelTenantAwareIT] [testModelCRUD] initializing REST clients against [http://[::1]:40731, http://127.0.0.1:37761]
2025-04-30T23:07:48.4883341Z     [2025-04-30T17:07:48,460][INFO ][o.o.m.r.RestMLModelTenantAwareIT] [testModelCRUD] after test

2025-04-30T23:07:48.5868016Z RestMLModelUndeployTenantAwareIT > testModelDeployUndeploy STANDARD_OUT
2025-04-30T23:07:48.5868839Z     [2025-04-30T23:07:48,487][INFO ][o.o.m.r.RestMLModelUndeployTenantAwareIT] [testModelDeployUndeploy] before test
2025-04-30T23:07:48.5869920Z     [2025-04-30T23:07:48,490][INFO ][o.o.m.r.RestMLModelUndeployTenantAwareIT] [testModelDeployUndeploy] initializing REST clients against [http://[::1]:40731, http://127.0.0.1:37761]
2025-04-30T23:07:48.9869981Z     [2025-04-30T23:07:48,947][INFO ][o.o.m.r.RestMLModelUndeployTenantAwareIT] [testModelDeployUndeploy] after test

Retry Run:

RestMLAgentTenantAwareIT: 23:28:35,125 to 23:29:00,830 = 25.705 seconds
RestMLConnectorTenantAwareIT: 16:29:00,885 to 16:29:01,517 = 0.632 seconds
RestMLModelGroupTenantAwareIT: 20:37:29,420 to 20:37:35,441 = 6.021 seconds
RestMLModelTenantAwareIT: 19:37:35,469 to 19:37:36,706 = 1.237 seconds
RestMLModelUndeployTenantAwareIT: 00:37:36,729 to 00:37:37,600 = 0.871 seconds

2025-04-30T23:28:35.1393528Z RestMLAgentTenantAwareIT > testAgentCRUD STANDARD_OUT
2025-04-30T23:28:35.1394402Z     [2025-04-30T23:28:35,125][INFO ][o.o.m.r.RestMLAgentTenantAwareIT] [testAgentCRUD] before test
2025-04-30T23:28:35.1396159Z     [2025-04-30T23:28:35,131][INFO ][o.o.m.r.RestMLAgentTenantAwareIT] [testAgentCRUD] initializing REST clients against [http://[::1]:38517, http://127.0.0.1:43331, http://[::1]:34331, http://127.0.0.1:33387, http://[::1]:33917, http://127.0.0.1:37123]
2025-04-30T23:29:00.9390064Z     [2025-04-30T23:29:00,830][INFO ][o.o.m.r.RestMLAgentTenantAwareIT] [testAgentCRUD] after test

2025-04-30T23:29:00.9391072Z RestMLConnectorTenantAwareIT > testConnectorCRUD STANDARD_OUT
2025-04-30T23:29:00.9392056Z     [2025-04-30T16:29:00,885][INFO ][o.o.m.r.RestMLConnectorTenantAwareIT] [testConnectorCRUD] before test
2025-04-30T23:29:00.9396510Z     [2025-04-30T16:29:00,890][INFO ][o.o.m.r.RestMLConnectorTenantAwareIT] [testConnectorCRUD] initializing REST clients against [http://[::1]:38517, http://127.0.0.1:43331, http://[::1]:34331, http://127.0.0.1:33387, http://[::1]:33917, http://127.0.0.1:37123]
2025-04-30T23:29:01.5382588Z     [2025-04-30T16:29:01,517][INFO ][o.o.m.r.RestMLConnectorTenantAwareIT] [testConnectorCRUD] after test

2025-04-30T23:37:29.4396743Z RestMLModelGroupTenantAwareIT > testModelGroupCRUD STANDARD_OUT
2025-04-30T23:37:29.4397782Z     [2025-04-30T20:37:29,420][INFO ][o.o.m.r.RestMLModelGroupTenantAwareIT] [testModelGroupCRUD] before test
2025-04-30T23:37:29.4402482Z     [2025-04-30T20:37:29,424][INFO ][o.o.m.r.RestMLModelGroupTenantAwareIT] [testModelGroupCRUD] initializing REST clients against [http://[::1]:38517, http://127.0.0.1:43331, http://[::1]:34331, http://127.0.0.1:33387, http://[::1]:33917, http://127.0.0.1:37123]
2025-04-30T23:37:35.5383455Z     [2025-04-30T20:37:35,441][INFO ][o.o.m.r.RestMLModelGroupTenantAwareIT] [testModelGroupCRUD] after test

2025-04-30T23:37:35.5386998Z RestMLModelTenantAwareIT > testModelCRUD STANDARD_OUT
2025-04-30T23:37:35.5388229Z     [2025-04-30T19:37:35,469][INFO ][o.o.m.r.RestMLModelTenantAwareIT] [testModelCRUD] before test
2025-04-30T23:37:35.5390207Z     [2025-04-30T19:37:35,474][INFO ][o.o.m.r.RestMLModelTenantAwareIT] [testModelCRUD] initializing REST clients against [http://[::1]:38517, http://127.0.0.1:43331, http://[::1]:34331, http://127.0.0.1:33387, http://[::1]:33917, http://127.0.0.1:37123]
2025-04-30T23:37:36.7378495Z     [2025-04-30T19:37:36,706][INFO ][o.o.m.r.RestMLModelTenantAwareIT] [testModelCRUD] after test

2025-04-30T23:37:36.8380950Z RestMLModelUndeployTenantAwareIT > testModelDeployUndeploy STANDARD_OUT
2025-04-30T23:37:36.8382207Z     [2025-05-01T00:37:36,729][INFO ][o.o.m.r.RestMLModelUndeployTenantAwareIT] [testModelDeployUndeploy] before test
2025-04-30T23:37:36.8384553Z     [2025-05-01T00:37:36,731][INFO ][o.o.m.r.RestMLModelUndeployTenantAwareIT] [testModelDeployUndeploy] initializing REST clients against [http://[::1]:38517, http://127.0.0.1:43331, http://[::1]:34331, http://127.0.0.1:33387, http://[::1]:33917, http://127.0.0.1:37123]
2025-04-30T23:37:37.6384907Z     [2025-05-01T00:37:37,600][INFO ][o.o.m.r.RestMLModelUndeployTenantAwareIT] [testModelDeployUndeploy] after test

dbwiddis requested review from HenryL27, Zhangxunmt, austintlee, b4sjoo, dhrubo-os, jngz-es, mingshl, model-collapse, rbhavna, xinyual, ylwu-amzn and zane-neo as code owners February 8, 2025 19:45

opensearch-trigger-bot bot added the infra label Feb 8, 2025

dbwiddis had a problem deploying to ml-commons-cicd-env-require-approval February 8, 2025 19:47 — with GitHub Actions Failure

dbwiddis temporarily deployed to ml-commons-cicd-env-require-approval February 10, 2025 02:10 — with GitHub Actions Inactive

dbwiddis temporarily deployed to ml-commons-cicd-env-require-approval February 10, 2025 03:27 — with GitHub Actions Inactive

dbwiddis temporarily deployed to ml-commons-cicd-env-require-approval February 10, 2025 17:41 — with GitHub Actions Inactive

dbwiddis force-pushed the tenant-aware-integ branch from b5cddd2 to 99ae3ed Compare February 12, 2025 00:27

dbwiddis had a problem deploying to ml-commons-cicd-env-require-approval February 12, 2025 00:28 — with GitHub Actions Failure

dbwiddis temporarily deployed to ml-commons-cicd-env-require-approval February 12, 2025 00:28 — with GitHub Actions Inactive

dbwiddis mentioned this pull request Feb 12, 2025

[Feature/multi_tenancy] Tenant-aware integration tests for Connector, Model, Agent, Model Groups #2818

Closed

2 tasks

dbwiddis force-pushed the tenant-aware-integ branch 2 times, most recently from 98bf007 to 41353eb Compare March 19, 2025 02:12

dbwiddis mentioned this pull request Mar 19, 2025

Fix model search for deleting connector #3671

Merged

dbwiddis temporarily deployed to ml-commons-cicd-env-require-approval March 19, 2025 02:13 — with GitHub Actions Inactive

dbwiddis had a problem deploying to ml-commons-cicd-env-require-approval March 19, 2025 02:13 — with GitHub Actions Failure

dbwiddis temporarily deployed to ml-commons-cicd-env-require-approval March 19, 2025 02:13 — with GitHub Actions Inactive

dbwiddis mentioned this pull request Apr 22, 2025

Directly return responses from Local Cluster client opensearch-project/opensearch-remote-metadata-sdk#141

Merged

dbwiddis added 8 commits April 30, 2025 15:16

Tenant Aware Integ Tests

07a8aac

Signed-off-by: Daniel Widdis <[email protected]>

Fix compile differences from 2.x to main

b2717da

Signed-off-by: Daniel Widdis <[email protected]>

Extend timeout for deploying model test for flaky PARTIALLY_DEPLOYED

476385d

Signed-off-by: Daniel Widdis <[email protected]>

Update error message for missing tenant header

cc6e8f8

Signed-off-by: Daniel Widdis <[email protected]>

Temporarily disable Model and Agent tenant aware search tests

c3c6964

Signed-off-by: Daniel Widdis <[email protected]>

Uncomment search-related tests now that searches are tenant aware

dc20c46

Signed-off-by: Daniel Widdis <[email protected]>

Update hits value getter for Lucene breaking change

69995be

Signed-off-by: Daniel Widdis <[email protected]>

Add test for deleting connector not in a model

560f479

Signed-off-by: Daniel Widdis <[email protected]>

dbwiddis force-pushed the tenant-aware-integ branch from 0e10ea8 to f810d75 Compare April 30, 2025 22:20

dbwiddis had a problem deploying to ml-commons-cicd-env-require-approval April 30, 2025 22:22 — with GitHub Actions Failure

dbwiddis had a problem deploying to ml-commons-cicd-env-require-approval April 30, 2025 22:22 — with GitHub Actions Error

dbwiddis had a problem deploying to ml-commons-cicd-env-require-approval April 30, 2025 22:22 — with GitHub Actions Failure

Add an integ test for aggregations in search

bd40be1

Signed-off-by: Daniel Widdis <[email protected]>

dbwiddis force-pushed the tenant-aware-integ branch from f810d75 to bd40be1 Compare April 30, 2025 22:46

dbwiddis had a problem deploying to ml-commons-cicd-env-require-approval April 30, 2025 22:48 — with GitHub Actions Failure

dbwiddis had a problem deploying to ml-commons-cicd-env-require-approval April 30, 2025 22:48 — with GitHub Actions Error

dbwiddis temporarily deployed to ml-commons-cicd-env-require-approval April 30, 2025 22:48 — with GitHub Actions Inactive

dbwiddis had a problem deploying to ml-commons-cicd-env-require-approval April 30, 2025 22:48 — with GitHub Actions Failure

dbwiddis temporarily deployed to ml-commons-cicd-env-require-approval May 1, 2025 17:31 — with GitHub Actions Inactive

Merge branch 'main' into tenant-aware-integ

ca03c9f

dhrubo-os had a problem deploying to ml-commons-cicd-env-require-approval May 1, 2025 17:36 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Tenant Aware Integration Tests #3516

Tenant Aware Integration Tests #3516

Uh oh!

dbwiddis commented Feb 8, 2025 •

edited

Loading

Uh oh!

codecov bot commented Feb 10, 2025 •

edited

Loading

Uh oh!

dbwiddis commented Mar 19, 2025 •

edited

Loading

Uh oh!

dhrubo-os commented May 1, 2025 •

edited

Loading

Uh oh!

dbwiddis commented May 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Tenant Aware Integration Tests #3516

Are you sure you want to change the base?

Tenant Aware Integration Tests #3516

Uh oh!

Conversation

dbwiddis commented Feb 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Check List

Uh oh!

codecov bot commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dbwiddis commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhrubo-os commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dbwiddis commented May 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dbwiddis commented Feb 8, 2025 •

edited

Loading

codecov bot commented Feb 10, 2025 •

edited

Loading

dbwiddis commented Mar 19, 2025 •

edited

Loading

dhrubo-os commented May 1, 2025 •

edited

Loading