Proposal: handling conflicting libraries in the same package

seps/sep-0002.md

@@ -0,0 +1,114 @@
+# Use-variants: for packages with conflicting libraries
+
+## The problem
+
+The `intel-oneapi-mkl` package is installed with many libraries. Some
+of these libraries define the same symbols but differ in their
+implementation. For example there is a set of libraries built with
+threading support, and another set which offers the same
+functionality but are built without threading support.
+
+Right now, the only mechanism Spack offers to choose between these
+libraries is a variant: you could add a `threading` variant and then
+choose which libraries to supply to dependents based on the value
+of that variant. The problem with a variant is that different
+variant values generate distinct installations of a package: we would
+install `intel-oneapi-mkl` once for each choice of `threading`, even
+though the installations would be the same.
+
+See also: https://github.com/spack/spack/discussions/22749
+
+## Proposed Changes
+
+* Modify variant declarations with a flag to indicate that the variant
+  determines how the package is used rather than how it is installed.
+* During concretization, this variant is treated like any other
+  variant (no changes to the concretization algorithm are required).
+* After concretization, the variant is stripped from the node and
+  stored on the edge connecting the node to its parent.
+
+## Example Use Cases
+
+### Intel oneAPI MKL
+
+Described in the introduction. This would be resolved by adding a
+`threading` variant:
+
+```
+variant('threading', values=['openmp', 'sequential'],
+        interface=True)
+```
+
+The new logic is the specification of `interface=True`: this
+indicates that this variant describes how the package is used rather
+than how it is installed.
+
+Dependents do not have to specify a value for `threading`: if they
+don't choose a value, then all dependents still need to "agree" on
+using the same libraries. Both the new and old concretizer already
+enforce consistency of variants amongst dependents.
+
+### Boost
+
+If users want to create a boost installation with many components,
+then dependents may only need a subset of the Boost libraries; this
+is a bigger issue if the unneeded Boost libraries define conflicting
+symbols (not with one another, but with symbols defined in the
+dependent or some sibling dependency of Boost).
+
+Because Boost component libraries don't conflict with each other,
+they do not need to be constrained like the dependents of
+`intel-oneapi-mkl`. However, when we call
+`root_spec['boost'].libs`, we want to know which libraries the
+dependent actually needs: to handle this, the dependent can
+recompute its dependencies based on its concrete spec, then pass the
+dependency variants it requires as parameters to the dependency when
+requesting `.libs`.
+
+We could avoid doing this universally by marking variants where this
+is a concern:
+
+```
+variant('regex', request=True)
+```
+
+This is not essential though: the dependent could universally
+recompute dependency information without any hints from dependents.
+For this reason, the Boost use case can be handled completely
+separately, since any variant modifications would only be used
+to speed up computations and are not required to achieve the
+semantics desired for the use case.
+
+### Intel Parallel Studio
+
+This provides multiple virtual packages: `mkl`, `blas`, `lapack`,
+`mpi`, and more. Each implementation has multiple sets of conflicting
+libraries. If a dependent is only using `intel-parallel-studio` for
+`mpi`, then we don't want use variants related to `mkl` to be
+recorded.
+
+This is murky pending a completed implementation of finer selection
+of virtual providers: Currently we cannot say that we depend on
+`intel-parallel-studio` *only* for `mpi` - if you ask for
+`intel-parallel-studio` it is assumed that you are using it for
+everything it provides (and there is no alternative `blas`
+implementation for example).
+
+Once there is a means of recording which virtuals are being used
+by a dependent, we will want to only propagate an interface variant
+to the dependent edge if the dependent actually uses the virtual
+associated with that variant. For that purpose, I think it would
+be suitable to name the associated virtual in the variant
+declaration:
+
+```
+variant('threading', interface=True, provider='mkl')
+```
+
+In other words, this has the same concretization semantics discussed
+for the Intel oneAPI MKL use case (i.e. that this is treated
+exactly the same as other variants during concretization), but
+adds controls on when the variant influences the hash of dependents:
+it only affects the dependent's hash when the dependent is actually
+using the package for the virtual named in the variant's `provider=`
+option.