RFC-3: drop custom config

[eed3si9n]

https://eed3si9n.com/sbt-drop-custom-config/

[hochgi]

@eed3si9n what about antlr or proto, and other types of "custom sources"?
I think there are 2 main types of custom configurations.

1. something that Compile depends on (like generated sources from src/main/antlr or custom schema dependencies with proto config etc'…)
2. Something that uses Compile output (Test also falls in this category), to use/build/generate/publish a different kind of result. e.g. I used a custom configuration to generate openapi schema from my main (Compile) scala code (using tapir), and had openapi tools generate typescript code from that schema to the frontend. The usage of a custom scope was more elegant than a new subproject.

What would be your recommendation to approach both these use-cases without new configuration scopes?

[eed3si9n]

1. This sounds like generated sources in the Compile configuration? https://www.scala-sbt.org/1.x/docs/Howto-Generating-Files.html scalaxb for example uses src/main/xsd/, and as the src/main/ indicates it should be in Compile configuration.
2. Probably just a task under Compile? If you think about current publish, it aggregates doc and source JARs which are also based on src/main/ Scala code.

[hochgi]

You're right, generating sources does makes sense to be scoped under Compile.
But what about pulling different types of dependencies?

lets say I have a server.
I can publish a jar to nexus, and the openapi contract to swaggerhub.
Would I need to write a custom task publishOpenAPI that is different than provided publish?
And what about plugins like native packager. Do we need a publishDocker, publishRPM, etc'…?

And what about consuming a dependency that is not a JAR?
I find it very appealing to just append % Configuration to the ModuleID to pull in a different kind of dependency.

Much like pulling proto files:
https://github.com/sbt/sbt-protobuf#declaring-dependencies

or custom code to pull in openapi schema, which is then used to generate sources.

How do you envision the syntax for pulling such dependencies without scopes?

[eed3si9n]

I can publish a jar to nexus, and the openapi contract to swaggerhub. Would I need to write a custom task publishOpenAPI that is different than provided publish?

No you can call addArtifact(...). See Publishing .war files.

And what about plugins like native packager. Do we need a publishDocker, publishRPM, etc…?

Yea. Why not?

And what about consuming a dependency that is not a JAR?

Maven doesn't have a concept of configuration. Anything that is integrated with Maven would need to be somehow encoded with classifier or something anyway.

How do you envision the syntax for pulling such dependencies without scopes?

libraryDependencies += ("org.testng" % "testng" % "5.7").classifier("jdk15")

[Baccata]

Maven doesn't have a concept of configuration. Anything that is integrated with Maven would need to be somehow encoded with classifier or something anyway.

Depends on what you mean by "integrated" really. Classifiers imply a certain coupling between how the artifact is produced/published and how it is consumed.

To keep the example of code-generation libraries such as proto, the smithy4s makes use of configuration ("org" % "artifact" % "version" % Smithy4s) to signal libraries that should be pulled for the code-generation process, but that should be omitted during the compilation process. Any library can be pulled like this, regardless of its classifier. These libraries can contain smithy files, but also some class files containing build-time validation logic (this is different from the paradigm implied by the protobug ecosystem, where the jars will contain both proto files and class files for consumption at run-time).

The reason we've used a "configuration" to encode this is mostly that it lets us gets automated updates from scala-steward for free. Now I can understand that it is arguably a hack, and that it shouldn't be SBT's problem, but considering how scala-steward is an inherent part of the Scala ecosystem, it'd be great if there was a way for users to signal from SBT that some settings/tasks are to be considered by steward. This would allow for the configuration usage to be substituted by a custom setting, like the following, without losing the benefit of auto-updates.

smithyLibraryDependencies ++= Seq(...) 

Additionally : it'd be great if SBT could make it easier to invoke its resolver layer manually, in order to turn an arbitrary List[ModuleID] into a List[File]. I think it would allow for plugins that currently use configuration to easily transition to custom settings.

[hochgi]

Thank you @Baccata, that's an excellent example.

Code (or resources) generation is currently making use of Configurations in various sbt plugins, and the way you put it made me better understand why my instinct is that dropping Configurations means we lose something quite unique.

Yes, most of the issues can be resolved with specialized tasks per Configuration, or multiple modules.

Trying to think of an example for how a Configurations give you something that isn't so easy to replace, is thinking of it in terms of another dimension, like @eed3si9n once wrote about in "traveling through the 4th dimension" blog post.

So the best example I currently have in mind, is actually an idea I'm toying with for a while now.
I've used @stryker-mutator's stryker4s for a while now.
I love the project, and the concept, but the sbt plugin is… meh… (see support for multi-project builds)

I was thinking about contributing better sbt support for this.
And I think that a new Configuration would be a natural fit.

the tl;dr:
the plugin should mutate your Compile sources, then compile them instead of the original sources, together with original Test sources. and finally run test, and check for failures.

This means you want to run same test, but replace the Compile dependency to something else, different from original sources.

Another "dimension" if you will.
So my thoughts were to run mutation, and store under target/scala-x.x/src_managed/NEW_CONFIGURATION/scala/…,
then define NEW_CONFIGURATION / compile to compile only these sources. and NEW_CONFIGURATION / test to run tests with modified classpath target/scala-x.x/NEW_CONFIGURATION-classes/** instead of target/scala-x.x/classes/** as usual.

So, I'm not sure how would I translate this idea to a world where we don't have Configurations.
Maybe this would mean we will need to redefine compile as (e.g.) compileMutated, and pretty much copy paste everything from original compile task, instead of a simple NEW_CONFIGURATION / sources := Seq.empty and NEW_CONFIGURATION / sourceManagedset to the proper dir, and using same compile task (in a different "dimension").

If we also thinking of dropping support for Test as well, this would break the "link" we now have between Compile sources and Test sources.
In terms of my example, the plugin would have to somehow know which project stores the files to mutate, and which project stores that test code to execute. Not an easy task for plugin authors.

Point is, that I think we're loosing something very unique if we drop support for custom Configurations.
WDYT?

[mdedetrich]

The shift in sbt 0.9 embraced configuration and enabled code reuse via inConfig(...)(...). However, generally the custom configuration often requires reimplementation of all tasks, and thus the complete knowledge of the internals.

At the risk of diverging away from the general point and rocking the boat a bit, I am getting an increasingly nagging feeling that the problem with tasks seem to stem from the fact that sbt doesn't really support inheritance. While it is definitely a true point that standard style inheritance is generally ill advised, as was shown by https://github.com/cvogt/cbt and https://www.youtube.com/watch?v=J0DrSaAqPSY, in essence build tools like sbt form a DAG where you can patch arbitrary nodes and this is one of the few cases where inheritance (and custom overriding of nodes) are useful, cbt just happened to embrace this fully using Scala's natural constructs (triat's, lazy val's/def's, etc etc along with a proper way of hiding internals, i.e. private).

It would be a very radical change, but maybe we should consider some inheritance style concept so you don't have to redefine tasks in a custom config? And maybe we should provide a mechanism so we can actually hide internals?

[eed3si9n]

The core notion of sbt is key-value pairs forming settings, and I think that's been working out ok for sbt.
Both cbt and Mill seem to be going towards trait stacking, and so did sbt 0.7, and I am not sure if sbt ecosystem following suit would be a great use of our energy esp if there are alternatives, vs what I am proposing here.

[mdedetrich]

Honestly the intent of my post wasn't necessarily to advocate for trait stacking (as you pointed out it would require a lot of effort) but rather just to highlight that we may be scapegoating the problem a bit, i.e. from what I can tell the problem isn't custom configs per say but rather limitations of sbt treating everything as a key/value (as you pointed rightly) and not having pretty basic abstractions (whether we solve that problem with trait stacking vs something else is another question).

I also don't know what the intent of sbt 2.0 should be, what I can judge from your post is its meant to be evolutionary rather than revolutionary. With this in mind removing custom configs seems like a very pragmatic solution but also a pretty blunt one and it may also have some pretty adverse effects on the ecosystem, while it can be argued that features like IntegrationTest can prob be solved better (as said here #7189 (comment)) we have cases of sbt-native-packager which uses configs for each of its distribution formats (i.e. there is a config for rpm, docker, etc etc) or sbt-paradox which uses configs for custom compiled web assets which do appear like more legitimate cases even though its been warned not to use custom configs for some time.

[SethTisue]

@muuki88 what would or could sbt-native-packager use instead?

[mdedetrich]

@SethTisue Made an issue at sbt/sbt-native-packager#1539 as well

[muuki88]

There are a few options. The easiest and most consistent one, would be to prefix all tasks/settings with the respective package. For example in docker there are

• dockerBaseRepository
• but Docker / publishLocal

The custom config is only used, when a setting is reused. So we don't reuse settings. dockerPublishLocal would be the new way to publish docker images. Even though this generates a lot more settings, it's easier to use IMHO.

What I'm sure will happen, that people start using Tasks to custom scope things, e.g.

val docker = settingKey[Unit]("this is now my config scope")

docker / publish := ???

However I would't want to go down that road with sbt-native-packager and use the opportunity to consistently rename all settings and do all the other breaking changes stacking up for years.

[SethTisue]

remove IntegrationTest, or deprecate for eventual removal? anytime anyone asks about it I always recommend just making a subproject instead

[eed3si9n]

Yea. IntegrationTest is a good example where it probably should've been a separate subproject to begin with.

[SethTisue]

could we rename to Compile to Main, or has that ship sailed? maybe Compile could be a deprecated alias for Main, to aid transition?

[eed3si9n]

The term Compile I think comes from compile scope in Maven, and I am not sure if changing to Main makes things easier. It's true that you can have compilation in either Compile or Test, but when someone invokes:

> foo/compile

they are building the Compile configuration, and when someone invokes

> foo/test

they are executing the code in Test configuration, so there's certain symmetry to the term.

I am curious how others, especially newcomers who are not steeped in the JVM build traditions think of this issue.

[adpi2]

I met quite a few Scala programmers that did not know that there was a foo/Test/compile task and they seemed surprised to discover it.

Those people do not really understand what a config is and that is I think because Compile is not a very intuitive name. Renaming Compile to Main would make things clearer: there is a Main config as in src/main and a Test config as in src/test.

[mdedetrich]

I met quite a few Scala programmers that did not know that there was a foo/Test/compile task and they seemed surprised to discover it.

Those people do not really understand what a config is and that is I think because Compile is not a very intuitive name. Renaming Compile to Main would make things cleared: there is a Main config as in src/main and a Test config as in src/test.

I can see both sides the coin here. The problem here is that sbt relies on maven/ivy concepts and the term Compile comes from that, so on the one hand while renaming the Compile scope to something else might make it less confusing for some newcomers, it would also make it extremely confusing for others (newcomers and otherwise).

For better or for worse, Scala (specifically sbt here) is piggybacking off the JVM ecosystem and because of this we use maven jars/dependency management for organizing project dependencies and compile/test scope is actually from Maven, its not something that really sbt defines itself and there is no such thing as a main scope in maven (I guess you could alias it by making a main config in ivy but that would make things even worse and the point of this discussion is to get rid of custom configs).

I can at least speak personally that when I started learning about sbt configs some time ago, I was quite confused because I had no idea whether it was some abstration that sbt came up with for better organising builds or something else. Later on I learnt that actually the whole concept of config's is taken from maven/ivy (ivy lets you create your own custom scopes called config's aside from compile/test) and thats why initially I just treated sbt as a fancy DSL over mvn/ivy builds.

This is the part that is most confusing to newcomers who start using Scala but are not familiar with JVM/Java ecosystem (I was sought of in that same boat like a decade ago now, I start using Scala in uni and didn't know too much about JVM ecosystem, just basic things about the language). If we started from scratch there may have been novel ways to do builds/dependency management (i.e. fury which explored source based dependency management) but that ship has sailed and it can also be argued that Scala wouldn't be where it is today if we didn't reuse parts of the JVM/Java ecosystem.

[lihaoyi]

Mill doesn't have configs for testing, and uses submodules. It works great, as not every module will have the same auxiliary submodules, and using submodules let you be precise in what kind of testing you need for each module:

1. Some have 0 tests (YOLO)
2. Some have 1 test submodule
3. Some have 2, e.g. Mill has its integration.local and integration.forked test suites which make use of a share integration module for support code
4. Some have more. for tests with different properties. e.g. upickle has upickle.jvm[...].test for most things, .testSlow for things that are slow and annoying to run but break infrequently, .testNonUtf8 for tests that require unusual JVM flags
5. You can have "orphaned" test suites not attached to any ScalaModule. e.g. an integration test suite that handles an entire project may be a standalone module, without any "parent" ScalaModule that it is a child of

One thing that is worth considering is how cheap modules are. In Mill, it's quite common to have dozens or hundreds of modules: e.g. Mill's own build has 84, while Ammonite's build has 440. But that's partly due to the fact that they are cheap to define (just an object foo extends Module), cheap to replicate (e.g. defining a module trait with N different submodules that are always instantiated together), and easy to manage (they're hierarchical, so you rarely ever see the "full list" of modules to get confused by).

SBT modules tend to be more heavyweight than that:

1. You can't easily define a "module group" of say 1 scala module and 2 test modules that are always instantiated together
2. All modules are in a flat namespace so you can't easily have a core.test module inside the core module the way we do in Mill, where it's common to have hierarchies like upickle.jvm[2.13.10].test as a submodule nested 4 layers deep in the hierarchy

But if they could be made more lightweight that would make it easier to use modules and sub-modules as a tool to replace a wide range of different bespoke build tool features

[YulawOne]

What about testFixtures? It's a common pattern in the Gradle worlds that we also use in SBT. Having {{name}}-test module for every compile module is IMO too much boilerplate and have harder discoverability. While we're having a global configuration for every module that also defines testFixtures .

[sjrd]

There are in fact 2 very different things that Configurations are used for: scoping tasks and settings, and namespacing library dependencies.

Clearly, there are good reasons to limit to Compile and Test for the former. IntegrationTest is the canonical example of this use case, and as has been mentioned, it is a pain to deal with. Two well-known example problems are:

• It does not really work with BSP-based IDEs.
• It is not supported by sbt plugins by default (because they cannot predict if your build is going to use it, and especially if it is going to use other custom configs like it).

On the other hand, for the second use case, there are plenty of known and lesser-known valid use cases:

• The Provided configuration namespaces dependencies that should be on the compilation classpath but not the run-time classpath. (there are more obscure ones in that category like CompileInternal, I believe).
• sbt plugins and user builds alike use them to download "meta-artifacts", which are used for something else than compilation or run-time:
  • The core functionality of some plugins, to be loaded by reflection.
  • Metadata such as the .proto files of Protobuf, which were mentioned elsewhere.
  • The Plugin configuration for compiler plugins actually belongs in this category as well.

All of these use cases are very relevant. They allow all those different namespaces to be resolved by update in a single pass, while being separated from each other. The plugin-tool configuration PR supposedly addresses that, but fails to bring the namespacing aspect with it: two plugins cannot use it together for two use cases without them colliding

---

I believe—and some colleagues at the Scala Center as well—that we should deprecate custom configs for task scoping, but keep them for namespacing library dependencies. Concretely, that would mean:

• Deprecate using a configuration other than Compile or Test in a Scope. We could make Compile and Test extend a marker trait for which it is OK to use as a scope without deprecation, and any other config would warn.
• Deprecate IntegrationTest, which is, I think, the only pre-defined configuration that is intended to be used as scope (as opposed to all the others, which are already intended and used for namespacing library dependencies).

[adpi2]

I also think custom configurations for resolving dependencies is a good use case that we should keep.

The issue that we are trying to fix is specific to task scoping: inheritance of tasks and settings on custom configs. Inheritance of dependencies over configs works as expected: if B extends A then B depends on all the dependencies of A transitively.

Custom configuration are used in three different ways:

1. To resolve dependencies only, e.g. Provided, ScalaTool, Plugin, Protobuf
2. To scope tasks only, e.g. Docker in sbt-native-packager
3. To create new compilation target (dep resolution and scoping of all the compilation/classpath related tasks), e.g. IntegrationTest, or Scalafix in sbt-scalafix.

The impact of deprecating custom configs for task scoping, on these 3 patterns, will be:

1. Still valid. The user can create its own configuration to resolve a different set of dependencies.
2. We replace the custom config by custom tasks: dockerPublish instead of Docker / publish
3. The user creates a new project, e.g. the integrationTests project or the scalafixRules project

[eed3si9n]

Thanks both for the thoughtful analysis. Focusing on namespacing library dependencies, I feel like these are feature leakage from Apache Ivy, which allowed custom configurations. Coursier may be able to emulate resolution for the simplest use cases, but I'd guess it would quickly run into issues with things like publishing to Maven or "protobuf->protobuf". In Bazel, a frequent pattern is to basically let the plugin define a subproject, and use it from the plugin, often referenced as tool.

sbt-equivalent might be document how to create a synthetic subproject (#2717), and use it? That would give dependency graph isolation (unlike my kitchen-sink proposal plugin-tool), and potentially a classpath isolation if you used run inside the synthetic subproject.

[mdedetrich]

This overview is very well put and explained in more clarity than what I was trying to express (which is basically that there seems to be a broad consensus that ivy style specific configs create more problems than they solve but we still need a way to deal with abstracting/name-spacing over library dependencies for cases such as web packaging/custom distribution formats and schema compilers etc etc).

This may also be an opportunity to solve the confusion problem that was expressed in #7189 (reply in thread), i.e. the fact that we are using config's as an abstraction method to solve namespacing issues which is complected with the fact that its also intertwined with maven/ivy. If we only support maven scopes then we can use something else thats not Config to solve the abstraction issue, making it clearer.

[sjrd]

sbt-equivalent might be document how to create a synthetic subproject (#2717), and use it?

If sbt projects were free, that might be an option. But creating an entire synthetic project (potentially one for every non-synthetic one) and all the associated settings, all to resolve some library dependencies, would be a significant overhead, IMO.

[eed3si9n]

Hi everyone. Thanks everyone for the feedback. After some pause (prepping ScalaMatsuri etc), I am back with fresh eyes, and wrote up feedback and outcome section in https://eed3si9n.com/sbt-drop-custom-config/, which I'll paste below:

feedback

I created a discussion thread #7189. In there, there were some interesting inputs (reminder: The goal of the RFC is to collectively explore the space of tradeoffs, and not everyone's opinion gets the same weight). In the thread Sébastien Doeraene wrote:

There are in fact 2 very different things that Configurations are used for: scoping tasks and settings, and namespacing library dependencies.

....

I believe that we should deprecate custom configs for task scoping, but keep them for namespacing library dependencies.

The library dependency use cases listed in the threads are:

1. The Provided configuration namespaces dependencies that should be on the compilation classpath but not the runtime classpath.
2. Sandbox configuration used for library dependency to download tooling.
3. Olivier Mélois mentioned tagging schema as "org" % "artifact" % "version" % Smithy4s, which allows Smithy graph to utilize dependency tooling like scala-steward.

These are interesting observations. Per Sébastien's suggestion, we would deprecate the scoping usage (for example Docker / publishLocal), but keep the % FooConfig usage. My counter argument is that the custom configuration resolution is a feature leakage from Apache Ivy days, and we'd be better off using synthetic subproject instead of creating a sandbox configuration.

To this Sébastien pointed out that creating synthetic subproject might incur overhead.

Since the main goal of RFC-3 proposal is to simplify the scoping, we could achieve that by limiting settings and task scoping to Compile and Test. At least for now, we should keep the library dependency namespacing via % FooConfig until the subproject performance is improved.

There was also a side suggestion to rename Compile to Main. I am undecided whether this is good idea or not. On on hand this makes it more consistent with src/main/. On the other than, Maven and Ivy both calls it compile scope, and it shows up in build.sbt as "compile->test" as well, so making this change would likely require long tail of Compile being alive.

outcome

• 2023-04-23: This RFC is partially accepted / put on-hold.
• IntegrationTest configuration should be deprecated.
• Setting and task scoping will be limited to Compile and Test configuration.
• Base on the feedback, we'll put the removal of custom configuration on hold until we can find suitable replacement for library dependency namespace.
• I am not sure renaming Compile to Main is good idea or going to be easy.