Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement index dynamic schema update on opensearch #18684

Conversation

nathansandi
Copy link
Contributor

@nathansandi nathansandi commented May 21, 2024

Description

  • Implement Validate index differences and Template differences for Open Search
  • Implement Update Schema on Elastic Search
  • Update Open search properties to bring the configuration
  • update createIndex and createTemplate methods from schema manager

Tests updates:

Related issues

closes #18487

@github-actions github-actions bot added the component/tasklist Related to the Tasklist component/team label May 21, 2024
@nathansandi nathansandi changed the base branch from main to tasklist/add-unit-tests-to-schema-implementation May 21, 2024 14:40
Copy link
Contributor

Tasklist Test Results

40 tests   - 509   40 ✅  - 504   28s ⏱️ - 1h 33m 55s
11 suites  - 132    0 💤  -   5 
11 files    - 132    0 ❌ ±  0 

Results for commit 5355976. ± Comparison against base commit c2e4ec0.

This pull request removes 509 tests.
io.camunda.tasklist.ProbesTestElasticSearchIT ‑ testIsNotReady
io.camunda.tasklist.ProbesTestElasticSearchIT ‑ testIsReady
io.camunda.tasklist.StartupIT ‑ testDockerWithNonRootUser
io.camunda.tasklist.config.GenerateOpenApiIT ‑ testSwaggerUIIsLoaded
io.camunda.tasklist.es.ElasticSearchSchemaManagementIT ‑ shouldChangeNumberOfReplicas
io.camunda.tasklist.es.ElasticsearchConnectorBasicAuthIT ‑ canConnect
io.camunda.tasklist.es.ElasticsearchConnectorSSLAuthIT ‑ canConnect
io.camunda.tasklist.es.ReindexElasticSearchIT ‑ reindexArchivedIndices
io.camunda.tasklist.es.ReindexElasticSearchIT ‑ resetIndexSettings
io.camunda.tasklist.es.SchemaCreationIT ‑ testDynamicMappingsOfIndices
…

Copy link
Contributor

github-actions bot commented May 21, 2024

Tasklist Test Results

144 files  ±0  144 suites  ±0   1h 24m 12s ⏱️ - 14m 7s
554 tests ±0  549 ✅ ±0  5 💤 ±0  0 ❌ ±0 
548 runs   - 4  543 ✅  - 4  5 💤 ±0  0 ❌ ±0 

Results for commit 7920ec3. ± Comparison against base commit 6c94589.

♻️ This comment has been updated with latest results.

Base automatically changed from tasklist/add-unit-tests-to-schema-implementation to Implement-dynamic-schema-update-usind-index-mapping-difference May 23, 2024 13:47
Base automatically changed from Implement-dynamic-schema-update-usind-index-mapping-difference to tasklist/implement-indexdiffmappingg May 23, 2024 13:47
@nathansandi nathansandi removed the request for review from houssain-barouni May 23, 2024 13:51
@nathansandi nathansandi marked this pull request as ready for review May 23, 2024 14:32
@nathansandi nathansandi changed the title feat: add index dynamic schema update on opensearch feat: implement index dynamic schema update on opensearch May 23, 2024
Copy link
Collaborator

@houssain-barouni houssain-barouni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment about reducing the duplicate code in IndexSchemaValidator implementations 🔧

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this class, I don't see anything that depends on OpenSearch besides org.opensearch.client.opensearch.indices.IndexSettings
Would it be possible to have a shared component that is used by IndexSchemaValidatorOpenSearch and IndexSchemaValidatoElasticSearch to avoid duplicated code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this @houssain-barouni (I am already working on that in a separate branch), I will work this on a different PR once involve refactor of methods that were previous implemented as well, and in order to make the review process easier, so I will do this out of scope of this Pull Request.

This work is located on this Pull request: #18896

…maStartup (#18812)

## Description

- Update of Migration Tests for the ElasticSearch 
- Remove .migrate() call from Schema Startup

Out of Scope of this PR: Delete Migration Code from the repository.
(Clean up)

## Related issues

closes #18488
## Description

- Create IndexSchemaValidatorUtil to store reusable methods 
- run lint
- add zeebe client as dependency to els-schema

## Related issues

closes #
…iff check (#18769)

## Description

- Apply integration tests for Open Search Dynamic schema updates

## Related issues

closes #18487
@nathansandi nathansandi merged commit fe133e3 into tasklist/implement-indexdiffmappingg May 31, 2024
10 of 15 checks passed
@nathansandi nathansandi deleted the tasklist/add-index-dynamic-schema-update-on-opensearch branch May 31, 2024 12:52
github-merge-queue bot pushed a commit that referenced this pull request Jun 4, 2024
## Description

Initial PR changes:
- Add the Index Schema Mapping Difference
- This PR Implements the Index Difference part for Elastic search
- It is a divided PR for the implementation of Elastic Search Update

The second PR will make this one functional
A third PR will be open for the tests

Complementary pull requests merged: (Elastic Search implementation) //
Already Approved
1) #18622
2) #18623

OpenSearch implementation: (To be merged/reviewed)
1) #18684
2) #18769
3) #18812

Refactor:
1) #18896

## Related issues

closes #18486
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/tasklist Related to the Tasklist component/team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants