Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance /datasets/data-status and /uploads/data-status endpoints #682

Open
yuanzhou opened this issue Jan 11, 2025 · 0 comments
Open

Enhance /datasets/data-status and /uploads/data-status endpoints #682

yuanzhou opened this issue Jan 11, 2025 · 0 comments
Assignees

Comments

@yuanzhou
Copy link
Member

yuanzhou commented Jan 11, 2025

The following enhancements to the /datasets/data-status and /uploads/data-status endpoints are needed in order to proceed with x-atlas-consortia/data-ingest-board#250:

  • Do not spawn new jobs when there's a running job
  • Response with 202 status if caching is in progress

Understand and port over the necessary implementation from SenNet: https://github.com/sennetconsortium/ingest-api/pull/625/files

According to @tjmadonna:

We refactored the caching from APscheduler to RQ. The only change to the data-status endpoints is that it will return a 202 if the scheduler is currently caching and there is no cache available. The same functionality can probably be done with APScheduler and a couple of flags in Redis.

The dashboard is so we can monitor jobs like entity/metadata validation and registration from users. I've been working on some changes that allows us to see the data-status caching jobs too. This is related to RQ so it shouldn't affect you.

@github-project-automation github-project-automation bot moved this to Backlog in Pitt HIVE Jan 11, 2025
@yuanzhou yuanzhou moved this from Backlog to Ready in Pitt HIVE Jan 21, 2025
@shirey shirey moved this from Ready to In Progress in Pitt HIVE Jan 21, 2025
@DerekFurstPitt DerekFurstPitt moved this from In Progress to Review/QA in Pitt HIVE Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Review/QA
Development

No branches or pull requests

2 participants