Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[New Feature]: Make number of processes used DISP-S1 SCIFLO be settings.yaml-configurable #1076

Open
philipjyoon opened this issue Feb 5, 2025 · 0 comments
Labels
enhancement New feature or request needs triage Issue that requires triage

Comments

@philipjyoon
Copy link
Contributor

Checked for duplicates

Yes - I've already checked

Alternatives considered

Yes - and alternatives don't suffice

Related problems

No response

Describe the feature request

We currently use a hard-coded formula for determining how many processes to use in DISP-S1 SCIFLO processing. We should make this formula a bit more flexible by getting some of the parameters from settings.yaml so that change to this formula doesn't require a new PCM release and deployment. It's currently here:
https://github.com/nasa/opera-sds-pcm/blob/develop/opera_chimera/precondition_functions.py#L385
n_parallel_bursts = max(int(round(available_cores / 2)) + 1, 1)

Make the following changes to that function:

  1. Parameterize the division factor and the constant factor so that they are retrieved from settings.yaml. Have reasonable values if they are not found.
  2. Make the default value of the denominator be 4 instead of 2. The constant factor can still be 1.
  3. I'd also make change that formula so that it's a multiplication instead of division. This allows for maximum flexibility and the most straightforward. So the currently formula would look: n_parallel_bursts = max(int(round(available_cores * 0.5 )) + 1, 1)
  4. At this point might as well parameterize theads_per_worker too
    https://github.com/nasa/opera-sds-pcm/blob/develop/opera_chimera/precondition_functions.py#L380
    threads_per_worker = available_cores
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request needs triage Issue that requires triage
Projects
None yet
Development

No branches or pull requests

1 participant