[MNG-5988] Dependency mediation should prioritize transitive dependencies based on scope.

[jira-importer]

Jostein Gogstad opened MNG-5988 and commented

The documentation states that dependency mediation only supports "nearest definition", regardless of the scope of the parent dependency.

If both compile- and test scoped dependencies shares the same transitive dependency, the test-scoped one will win if it has shallower depth. That in turn will lead to runtime exceptions since the transitive dependency is no longer on the classpath.

Take the following pom from a typical Spring Boot application. Since the camel-test-spring dependency also depends on spring, it wins and Spring is no longer available to the application at runtime.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>bugreport</artifactId>
    <packaging>jar</packaging>
    <version>1.0.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <version>1.3.2.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-test-spring</artifactId>
            <version>2.16.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

</project>

Now look for spring-beans or spring-context in the following dependency graphs:

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ bugreport ---
[INFO] com.example:bugreport:jar:1.0.0-SNAPSHOT
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:1.3.2.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:1.3.2.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:1.3.2.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:1.3.2.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:1.3.2.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.1.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.1.3:compile
[INFO] |  |  |  +- org.slf4j:jcl-over-slf4j:jar:1.7.13:compile
[INFO] |  |  |  +- org.slf4j:jul-to-slf4j:jar:1.7.13:compile
[INFO] |  |  |  \- org.slf4j:log4j-over-slf4j:jar:1.7.13:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.16:runtime
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:1.3.2.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.0.30:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.0.30:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-logging-juli:jar:8.0.30:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.0.30:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-validation:jar:1.3.2.RELEASE:compile
[INFO] |  |  \- org.hibernate:hibernate-validator:jar:5.2.2.Final:compile
[INFO] |  |     +- javax.validation:validation-api:jar:1.1.0.Final:compile
[INFO] |  |     +- org.jboss.logging:jboss-logging:jar:3.2.1.Final:compile
[INFO] |  |     \- com.fasterxml:classmate:jar:1.1.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.0:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.6.5:compile
[INFO] |  +- org.springframework:spring-web:jar:4.2.4.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:4.2.4.RELEASE:compile
[INFO] \- org.apache.camel:camel-test-spring:jar:2.16.2:test
[INFO]    +- org.apache.camel:camel-test:jar:2.16.2:test
[INFO]    |  +- org.apache.camel:camel-core:jar:2.16.2:test
[INFO]    |  |  \- org.slf4j:slf4j-api:jar:1.6.6:compile
[INFO]    |  \- junit:junit:jar:4.11:test
[INFO]    |     \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO]    +- org.apache.camel:camel-spring:jar:2.16.2:test
[INFO]    +- org.springframework:spring-test:jar:4.1.9.RELEASE:test
[INFO]    +- org.springframework:spring-context:jar:4.1.9.RELEASE:compile
[INFO]    +- org.springframework:spring-beans:jar:4.1.9.RELEASE:compile
[INFO]    +- org.springframework:spring-expression:jar:4.1.9.RELEASE:compile
[INFO]    +- org.springframework:spring-aop:jar:4.1.9.RELEASE:compile
[INFO]    |  \- aopalliance:aopalliance:jar:1.0:compile
[INFO]    +- org.springframework:spring-tx:jar:4.1.9.RELEASE:test
[INFO]    +- org.springframework:spring-core:jar:4.1.9.RELEASE:compile
[INFO]    |  \- commons-logging:commons-logging:jar:1.2:compile
[INFO]    +- com.sun.xml.bind:jaxb-core:jar:2.2.11:test
[INFO]    \- com.sun.xml.bind:jaxb-impl:jar:2.2.11:test

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ bugreport ---
[INFO] com.example:bugreport:jar:1.0.0-SNAPSHOT
[INFO] \- org.springframework.boot:spring-boot-starter-web:jar:1.3.2.RELEASE:compile
[INFO]    +- org.springframework.boot:spring-boot-starter:jar:1.3.2.RELEASE:compile
[INFO]    |  +- org.springframework.boot:spring-boot:jar:1.3.2.RELEASE:compile
[INFO]    |  +- org.springframework.boot:spring-boot-autoconfigure:jar:1.3.2.RELEASE:compile
[INFO]    |  +- org.springframework.boot:spring-boot-starter-logging:jar:1.3.2.RELEASE:compile
[INFO]    |  |  +- ch.qos.logback:logback-classic:jar:1.1.3:compile
[INFO]    |  |  |  +- ch.qos.logback:logback-core:jar:1.1.3:compile
[INFO]    |  |  |  \- org.slf4j:slf4j-api:jar:1.7.7:compile
[INFO]    |  |  +- org.slf4j:jcl-over-slf4j:jar:1.7.13:compile
[INFO]    |  |  +- org.slf4j:jul-to-slf4j:jar:1.7.13:compile
[INFO]    |  |  \- org.slf4j:log4j-over-slf4j:jar:1.7.13:compile
[INFO]    |  +- org.springframework:spring-core:jar:4.2.4.RELEASE:compile
[INFO]    |  \- org.yaml:snakeyaml:jar:1.16:runtime
[INFO]    +- org.springframework.boot:spring-boot-starter-tomcat:jar:1.3.2.RELEASE:compile
[INFO]    |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.0.30:compile
[INFO]    |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.0.30:compile
[INFO]    |  +- org.apache.tomcat.embed:tomcat-embed-logging-juli:jar:8.0.30:compile
[INFO]    |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.0.30:compile
[INFO]    +- org.springframework.boot:spring-boot-starter-validation:jar:1.3.2.RELEASE:compile
[INFO]    |  \- org.hibernate:hibernate-validator:jar:5.2.2.Final:compile
[INFO]    |     +- javax.validation:validation-api:jar:1.1.0.Final:compile
[INFO]    |     +- org.jboss.logging:jboss-logging:jar:3.2.1.Final:compile
[INFO]    |     \- com.fasterxml:classmate:jar:1.1.0:compile
[INFO]    +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5:compile
[INFO]    |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.0:compile
[INFO]    |  \- com.fasterxml.jackson.core:jackson-core:jar:2.6.5:compile
[INFO]    +- org.springframework:spring-web:jar:4.2.4.RELEASE:compile
[INFO]    |  +- org.springframework:spring-aop:jar:4.2.4.RELEASE:compile
[INFO]    |  |  \- aopalliance:aopalliance:jar:1.0:compile
[INFO]    |  +- org.springframework:spring-beans:jar:4.2.4.RELEASE:compile
[INFO]    |  \- org.springframework:spring-context:jar:4.2.4.RELEASE:compile
[INFO]    \- org.springframework:spring-webmvc:jar:4.2.4.RELEASE:compile
[INFO]       \- org.springframework:spring-expression:jar:4.2.4.RELEASE:compile

---

Affects: 3.2.3

Attachments:

• Collected.svg (6.65 kB)
• MNG-5988.zip (3.38 kB)
• PRE.svg (183.01 kB)
• Resolved.svg (5.85 kB)

Issue Links:

• MNG-6056 Implement Feature Toggle Module to handle Feature Toggles
  ("is blocked by")

• MNG-7852 Use all the versions for dependency resolution rather than "nearest first" or "declared first"

• MNG-6058 Test dependencies should override application dependencies only during testing
  ("is superceded by")

Remote Links:

• GitHub Pull Request #442
  

4 votes, 12 watchers

[jira-importer]

Christian Schulte commented

I think preferring runtime over test is the same use case.

[jira-importer]

Christian Schulte commented

This contradicts the way Maven performs dependency resolution. There are multiple solutions to this. The easiest is to just move the dependency in question up in the hierarchy so that it gets selected.

[jira-importer]

Jostein Gogstad commented

That's true, explicitly depending on some transitive dependency some library needs will force maven to use that version. It's the easiest solution, but the problem is unexpected and it is some times difficult to detect which library to depend on.

When a test-scoped dependency requires a different (not necessarily newer) version of a library at shallow depth, maven has two choices:

1. Package the test-scoped version, strictly adhering to nearest-definition and possibly downgrading the library that production code requires.
2. Package the compile/runtime scoped version, possibly downgrading the library that test code requires

Alternative 2 is the better option of these choices because errors as a result of conflicting dependencies are detected earlier and they don't affect production code. CI being fairly common among development teams these days will catch errors resulting from choosing the "wrong" library when the tests are run. On the other hand, it the tests gets to dictate which library are chosen, errors aren't visible until the application is run. Even though the transitive dependencies are compile scoped, it won't matter since the direct dependency is already compiled.

Maven should use "nearest definition" when choosing dependencies, but it consider the scope of the direct dependency when doing so.

[jira-importer]

Christian Schulte commented

Would it be possible to provide a stripped down example project just demonstrating the issue? There are quite a lot of conflicting dependencies on spring-beans for example and what is shown in the output may be misleading e.g. the version used by Maven is just another one of all those conflicting nodes not even introduced by the test scope dependency. It's quite hard to debug such a large dependency graph just to find out that there are way more than just the two conflicting nodes mentioned above. Maven never packages a test-scoped dependency, by the way and Maven does the prioritization you are requesting. spring-beans is introduced at multiple conflicting levels in the example you provided. I am not keen on debugging Springframework POMs. Please try to isolate the issue.

[jira-importer]

Christian Schulte commented

Take a look at this picture and see how many times spring-beans and spring-context appear.

[jira-importer]

Christian Schulte commented

Another solution would be to apply dependency management. Like:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>bugreport</artifactId>
  <packaging>jar</packaging>
  <version>1.0.0-SNAPSHOT</version>
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-framework-bom</artifactId>
        <version>4.3.0.RELEASE</version>
        <scope>import</scope>
        <type>pom</type>
      </dependency>
    </dependencies>
  </dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
      <version>1.3.2.RELEASE</version>
    </dependency>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-test-spring</artifactId>
      <version>2.16.2</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

[jira-importer]

Christian Schulte commented

Trying to create an example reproducing the issue. Please let me know if this really is demonstrating the issue. Unzip and run mng5988.sh. This will install various POMs to group localhost you can delete afterwards and then print the classpath in question.

[DEBUG] localhost:mng5988:jar:20160701
[DEBUG]    localhost:mng5988-direct-one:jar:20160701:compile
[DEBUG]       localhost:mng5988-transitive-one:jar:20160701:compile
[DEBUG]          localhost:mng5988-transitive-transitive-one:jar:20160701:compile
[DEBUG]    localhost:mng5988-direct-two:jar:20160701:test
[DEBUG]       localhost:mng5988-transitive-two:jar:20160701:test
[DEBUG]          commons-io:commons-io:jar:2.5:compile

See that 'commons-io:commons-io:jar:2.5:compile' has been pulled in from the test scope dependency instead of from the compile scope dependency. You are requesting to make Maven not use that one although it is the nearer but prioritize the one from the compile scope dependency although that is the farther. Is that correct?

[jira-importer]

Christian Schulte commented

Dependency graph of the example.

[jira-importer]

Christian Schulte commented

Dependency graph of the example after conflict resolution.

[jira-importer]

Jostein Gogstad commented

Good work on the test case.

That's correct. In your testcase maven should pick up commons-io from mng-transitive-transitive-one since it's is reached through a compile scoped dependency.

I wasn't aware that maven didn't package test-scoped dependencies. But as that is the case, it makes even more sense to prioritize transitive dependencies from compile scoped dependencies since the application won't work otherwise.

Regarding the horrible spring dependency graphs, I absolutely agree it's a pain to walk through and figure out where the dependency mediation fails.

Thanks for coming back on this issue so quickly!

[jira-importer]

Stephen Connolly commented

Re-opening as not actually fixed after 3.5.0 reset

[jira-importer]

Elliotte Rusty Harold commented

So is the bug that Maven is not doing what it claims to do, or that you would like a different dependency mediation rule?

[jira-importer]

Shannon Carey commented

Btw I see this happening still in 3.6.3.

is the bug that Maven is not doing what it claims to do

Probably not, but I would argue that Maven's document is not very clear about what it claims. It does explain how version conflicts are resolved, but it also explains how transitive scope resolution works. And specifically, it says that if you have a test-scope dependency, the transitive compile-scope dependencies will be test-scope. That implies that you wouldn't see a test dependency impacting the compile/runtime classpath as this Jira reports is happening. But, it's not super clear because the documentation doesn't explain how those interact.

IMO, an alteration to the test-scope dependencies should have zero impact on the production (runtime) classpath. It's counterintuitive and creates risk.

There are multiple solutions to this. The easiest is to just move the dependency in question up in the hierarchy so that it gets selected.

You're right that there are workarounds:

• create an explicit top-level dependency
• use dependency management
• add an <exclusion> to the test dependency (yuck)

However, those workarounds assume some unfriendly things, even from someone like myself who has been using Maven a long time:

1. You've noticed the problem in the first place
2. You have to figure out that this is the cause of the problem, which IMO is a bit mysterious/unexpected
3. Every time someone updates a dependency, they have to remember to go through and copy the transitive dependency versions up to the top-level POM
4. Every time someone updates a dependency, they have to remember to go through the transitive dependencies to figure out whether any should be removed from the top-level POM
5. Every time someone updates a dependency, you may get a sudden mysterious compilation failure because your <exclusion> now makes the needed library completely omitted

 

[jira-importer]

Tamas Cservenak commented

Bug is in Maven Core recorded as MNG-8041

Maven Resolver is fine, as explained here https://maven.apache.org/resolver-archives/resolver-2.0.0-alpha-7/common-misconceptions.html#misconception-no2-%E2%80%9Ctest-graph%E2%80%9D-is-superset-of-%E2%80%9Cruntime-graph%E2%80%9D

[jira-importer]

Elliotte Rusty Harold commented

I think what we need here is a merging of the test scoped and the compile scoped dependency. Assuming the test scoped dependency is nearer in the tree, then that is absolutely the version we should choose. However, we should be able to upgrade the scope of that dependency to compile while retaining the version.