Sending the following scan to a scancode.io instance triggers a scan with all the groups of the selected pipeline

Yes, this is the expected behavior. When no groups are provided, all the steps are included.
See https://github.com/aboutcode-org/scancode.io/blob/main/aboutcode/pipeline/__init__.py#L52

There is also no clear way to select groups when starting a project from the API

I'm adding support for this in #1427

We also do not see the selected groups correctly in the pipeline run modal when scan is triggered from the API, as in the above case, pipeline steps from the StaticResolver group is run, but this is not visible in the modal, but only in the log.

This is also expected. The current "selected group" implementation allows you to limit your pipeline steps to the specified groups. If no groups are provided, all the steps are run during the pipeline execution. We could improve the group feature but this will require design and refactoring.

@AyanSinhaMahapatra You can now provide multiple pipelines during project creation using the API.
Added in #1427
https://scancodeio.readthedocs.io/en/latest/rest-api.html#create-a-project

Thanks @tdruez for adding the support!

Yes, this is the expected behavior.

Ah right, this makes sense.

But if we have a way to select Groups in a pipeline and the default pipeline (when no groups are given) is all the groups included, then it seems that there are no way to select a pipeline without any optional groups?

Like the inspect_packages pipeline was structured more towards the default pipeline being package scanning + assembly and then an optional group is present where we resolve dependencies statically. Was this a wrong pipeline design then? As it is not possible to de-select the Optional steps through the API?

1. So if the default is to not select any options, and then if someone wants to select options they explicitly state it, we can run the pipeline with all possible combination of steps through the API, if the default is all optional steps enabled, then there is no way in the API to run the pipeline with just the mandatory steps.
2. In the UI too, the experience is that if no groups/options are selected, then the pipeline is run without those steps, and the default is without any optional steps. Optional steps are only enabled on explicit selection. This is different from the API experience too.

So do you think we should:

1. Make no optional steps the default for starting a project via the API
2. Restructure the pipelines such that users should ideally select one/multiple/all optional groups, and the default is all groups selected

If no groups are provided, all the steps are run during the pipeline execution.

Whether the groups are provided, or selected automatically by default, I think it would be consistent to show the selected groups in the modal, whenever they are run. Otherwise it is required to know which steps are from which options, and check the logs to determine the run. Again as the UI and API defaults are different, it makes sense to be more verbose

What do you think?

then it seems that there are no way to select a pipeline without any optional groups?

Right, this is a flaw of the current approach (which was initially design for the m2m pipeline)

Make no optional steps the default for starting a project via the API

We could make this the default, but that should be the same default regardless of using the UI, API, or CLI. The default behavior should be consistent.

For now, only 3 pipelines leverage the groups:

• inspect_packages
• map_deploy_to_develop
• resolve_dependencies

It's still time to review the group usage in those and see what would be the best refactoring.

@tdruez upon considering the above mentioned pipelines:

I think both the inspect_packages and map_deploy_to_develop pipelines have steps which are not part of any group which are quite valuable on their own.

In case of inspect_packages pipeline the dominant use case is actually without the optional group and even in case of map_deploy_to_develop the basic mapping/natching/ABOUT files steps can be used in an isolated manner too.

@pombredanne also had a suggestion sometime ago (I can't find the issue/comment) to rename groups to options which seems to favour these pipeline steps as optional add-ons in contrast to groups that can be disabled optionally.

It also makes less sense for example to run all the d2d steps for all groups by default in the API.

Even by design the UI/CLI rightly has the design that these pipeline steps or groups are optional, and enabled optionally upon explicit request, and I think this should be the case for the API as well.
This also gives us more flexibility in pipeline design with optional steps, and the option to run just the basic steps of the pipeline too, instead of being forced to select one/multiple groups or run all of them by default. The former can also run all the steps of a pipeline, but only by explicit choice.

What do you think?

@AyanSinhaMahapatra That makes sense, I can see this transition to "optional" steps.
Renaming to options and @optional_step would help to emphasize the "excluded by default" behavior.

@AyanSinhaMahapatra I've started the refactoring in #1442
Please have a look and let me know if anything else needs to be changed (I've tried to keep the changes to a minimum)
Also, the tests are failing following the behavior change, we need to ensure that the impact is expected.