Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calls to scheduler are not asynchronous #217

Open
jbweston opened this issue Apr 5, 2024 · 1 comment
Open

Calls to scheduler are not asynchronous #217

jbweston opened this issue Apr 5, 2024 · 1 comment

Comments

@jbweston
Copy link
Collaborator

jbweston commented Apr 5, 2024

The Scheduler methods 'queue' and 'start_job' are blocking functions.

'start_new_jobs' is run in a ThreadPoolExecutor inside JobManager, but 'queue' is never run in a separate thread, and will block the event loop.

One fix would be to use 'asyncio.run_in_thread' everywhere these methods are called. An alternative is to make these methods async.

The first fix is easier to do, but also easier to footgun (forgetting to 'run_in_thread' impacts performance). Second fix would require more changes.

@basnijholt
Copy link
Owner

The reason I call start_new_jobs in a ThreadPoolExecutor is because when profiling, this part took a lot of time. queue seemed to be pretty fast.

Open to all improvements though!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants