Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add endpoint to enable exporter #19261

Merged
merged 8 commits into from
Jun 13, 2024
Merged

Conversation

deepthidevaki
Copy link
Contributor

@deepthidevaki deepthidevaki commented Jun 11, 2024

Description

This PR combines two things

  1. ExporterDirector does not close itself when no exporters are configured. Instead it goes to an idle state, where the actor is still running, but it is not actively doing any tasks. When a new exporter is enabled, it will become active again and starts all necessary services to resume exporting.
  2. Added endpoint in the gateway for enabling exporter. The endpoint allows to enable an exporter with and without another exporter to initialize from.

Related issues

closes #19022
closes #18752

@github-actions github-actions bot added component/zeebe Related to the Zeebe component/team component/operate Related to the Operate component/team component/tasklist Related to the Tasklist component/team labels Jun 11, 2024
@deepthidevaki deepthidevaki marked this pull request as ready for review June 12, 2024 06:28
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 Is there no test to check that the director is now idle? What would be the impact if it wasn't and we don't verify this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can test if the state is idle. But it is not adding much value. Testing whether it has stopped all services is a bit more tricky. The main requirement is that once we disable all exporters, we can enable new ones later. This is tested in the ExporterEnableTest.

The impact if it did not stop all services is that is unnecessarily consuming CPU cycles - it reads the log entry for every commit even if there are no exporters to export.

@deepthidevaki deepthidevaki added this pull request to the merge queue Jun 12, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 12, 2024
@deepthidevaki deepthidevaki added this pull request to the merge queue Jun 12, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 12, 2024
@deepthidevaki deepthidevaki added this pull request to the merge queue Jun 13, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 13, 2024
@deepthidevaki deepthidevaki added this pull request to the merge queue Jun 13, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 13, 2024
On idle, it stops receiving notification from logstream commits and also
stops the distribution service. When new exporters are configured, it will
restart exporting again.
ExporterDirector does not close if no exporters are configured. Instead it
just goes to an idle state. So the tests that wait for it to be closed is updated.
@deepthidevaki deepthidevaki force-pushed the dd-19022-gateway-enable-exporter branch from 1888509 to 4c30b88 Compare June 13, 2024 07:55
@deepthidevaki deepthidevaki added this pull request to the merge queue Jun 13, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 13, 2024
@deepthidevaki
Copy link
Contributor Author

Merge failed due to

Warning: Failed to save: Cache service responded with 429 during upload chunk.
/home/runner/_work/_actions/actions/setup-java/v[4](https://github.com/camunda/camunda/actions/runs/9497110638/job/26172950126#step:21:4)/dist/cleanup/index.js:475
                        throw new Error(`Cache upload failed because file read failed with ${error.message}`);
                        ^

Error: Cache upload failed because file read failed with EBADF: bad file descriptor, read

retrying.

@deepthidevaki deepthidevaki added this pull request to the merge queue Jun 13, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 13, 2024
@deepthidevaki
Copy link
Contributor Author

😭 I'm really unlucky today. Merge failed again due to "self hosted runner lost communication with the server". Retrying..

@deepthidevaki deepthidevaki added this pull request to the merge queue Jun 13, 2024
Merged via the queue into main with commit d291df7 Jun 13, 2024
38 checks passed
@deepthidevaki deepthidevaki deleted the dd-19022-gateway-enable-exporter branch June 13, 2024 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/operate Related to the Operate component/team component/tasklist Related to the Tasklist component/team component/zeebe Related to the Zeebe component/team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add gateway endpoints for exporter enable request Allow adding an exporter via an API
2 participants