Identify and remove unused dependencies

[epag]

I want to find and use a tool to identify any un-used dependencies and remove them from out project.

This will tangently help with #158 and also help keep a cleaner project overall

[HankHerr-NOAA]

Taking a look at this. I found this tool via web search:

https://github.com/autonomousapps/dependency-analysis-gradle-plugin

Its a plugin that, once added to build.gradle can be run with ./gradlew dependencyAnalysis. I just need to identify the latest version to use in the plugin line

      id("com.autonomousapps.dependency-analysis") version "<<latest_version>>"

Hank

[HankHerr-NOAA]

Looks like 2.7.0. Trying that. I'll report back if/when I get results.

Hank

[HankHerr-NOAA]

Well, I knew I only had a few minutes, and now my time is up.

Turns out that this is the line I needed to added to build.gradle:

    id 'com.autonomousapps.build-health' version '2.7.0' // Check for the latest version

With that, I ran ./gradlew buildHealth and got this outpout:

[hank.herr@owpal-d-ised01 wres]$ ./gradlew buildHealth                                                                                                                                                                                                       

FAILURE: Build failed with an exception.

* Where:
Build file '/ised/wres/hank.herr/git/wres/build.gradle' line: 90

* What went wrong:
An exception occurred applying plugin request [id: 'com.autonomousapps.build-health', version: '2.7.0']
> Failed to apply plugin 'com.autonomousapps.build-health'.
   > class org.gradle.api.internal.project.DefaultProject_Decorated cannot be cast to class org.gradle.api.initialization.Settings (org.gradle.api.internal.project.DefaultProject_Decorated and org.gradle.api.initialization.Settings are in unnamed module
of loader org.gradle.internal.classloader.VisitableURLClassLoader @61e717c2)

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 3s

I assume this is some sort of incompatibility between the gradle version we're using and the plugin, but I have a meeting with Jim followed by Arvin, so this will probably have to wait until tomorrow.

Hank

[HankHerr-NOAA]

Back to this...

When building, my gradle version appears to be 7.6.4:

------------------------------------------------------------
Gradle 7.6.4
------------------------------------------------------------

Build time:   2024-02-05 14:29:18 UTC
Revision:     e0bb3fc8cefad8432c9033cdfb12dc14facc9dd9

Kotlin:       1.7.10
Groovy:       3.0.13
Ant:          Apache Ant(TM) version 1.10.13 compiled on January 4 2023
JVM:          17.0.10 (Eclipse Adoptium 17.0.10+7)

Let me see if the buildHealth plugin is compatible with that.

Hank

[HankHerr-NOAA]

The wiki,

https://github.com/autonomousapps/dependency-analysis-gradle-plugin/wiki/Compatibilities-&-Limitations

points me here:

https://github.com/autonomousapps/dependency-analysis-gradle-plugin/blob/main/src/functionalTest/groovy/com/autonomousapps/AbstractFunctionalSpec.groovy#L24-L30

That has this code:

  protected static final GRADLE_7_5 = GradleVersion.version('7.5.1')
  protected static final GRADLE_7_6 = GradleVersion.version('7.6.2')
  protected static final GRADLE_8_0 = GradleVersion.version('8.0.2')
  protected static final GRADLE_8_4 = GradleVersion.version('8.4')
  protected static final GRADLE_8_9 = GradleVersion.version('8.9')
  protected static final GRADLE_8_10 = GradleVersion.version('8.10.2')
  protected static final GRADLE_8_11 = GradleVersion.version('8.11.1')

  protected static final GRADLE_LATEST = GRADLE_8_11

  // For faster CI times, we only test min + max. Testing all would be preferable, but we don't have till the heat death
  // of the universe.
  protected static final SUPPORTED_GRADLE_VERSIONS = [
    GradleVersions.minGradleVersion,
    GRADLE_LATEST,
    //GRADLE_8_11,
  ]

The earliest listed version is 7.5.1, which is after 7.4.2.

I need to reeducate myself on how to increase our Gradle version, and whether that requires ITSG intervention. I'll also check to see if we can get our hands on a version of the tool that lists 7.4 in the compatible versions.

Hank

[HankHerr-NOAA]

There is a Kotlin file in the repo that indicates the minimum version may be 7.4:

  /** Minimum supported version of Gradle. */
  @JvmField val minGradleVersion: GradleVersion = gradle74

Hmmm...

Hank

[HankHerr-NOAA]

I edited the previous comment. I pasted the wrong info from the clipboard.

Hank

[james-d-brown]

To be clear, our build is using 7.6.4. We should upgrade to 8 before too long, but it generally involves a non-trivial amount of work. The version of gradle is set in gradle-wrapper.properties. Your local version of gradle doesn't really matter in that regard.

[james-d-brown]

gradle-wrapper.properties

distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-7.6.4-bin.zip
networkTimeout=10000
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

[HankHerr-NOAA]

Thanks, James.

Not sure why I thought we were using 7.4.2. We're using 7.6.4. That should be compatible, and yet it seems that the tool is incorrectly casting a variable internal to Gradle: org.gradle.api.internal.project.DefaultProject_Decorated. Maybe that is actually in a plugin that needs a version update? Let me see if I can find it,

Hank

[HankHerr-NOAA]

That class, org.gradle.api.internal.project.DefaultProject_Decorated, is internal to gradle, at least based on a web search.

If I update the distributionUrl to point to the same version indicated in the GitHub project for healthCheck, 8.2.1, then I am able to get past the plugin issue, but end up with an error building a jar:

* What went wrong:
A problem occurred configuring root project 'wres'.
> java.util.concurrent.ExecutionException: org.gradle.api.GradleException: Failed to create Jar file /ised/wres/hank.herr/gradle/caches/jars-9/7004f7b06bdeb3169de40cb39f867fea/jackson-core-2.17.1.jar.

I think this goes back to James's comment about upgrading gradle involving a, "non-trivial amount of work". Let me first check if there are other tools available to do what we want to do. If not, I'll tackle the Gradle version update, first, and then revisit this ticket. I had proposed tackling updating dependency versions as my next big task (starting with #68 before the holidays), and this ties well into that.

Thanks,

Hank

[HankHerr-NOAA]

This may be an option:

https://www.baeldung.com/gradle-finding-unused-dependencies

It uses nebula.lint and I'm able to add it as a plugin pretty easily. Let me see if I can figure out how to use it,

Hank

[HankHerr-NOAA]

So I was able to get it to run, but it was consistently not reporting any issues. However, the lack of task output was disconcerting:

> Task :lintGradle

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6.4/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 24s

So I found a command to generate reports in https://www.baeldung.com/java-gradle-lint-intro:

./gradlew generateGradleLintReport

Upon running that an capturing screen output, I get a bunch of output like this:

> Task :generateGradleLintReport
Generated a report containing information about 0 lint violations in this project

> Task :wres-config:generateGradleLintReport
Generated a report containing information about 0 lint violations in this project

> Task :wres-datamodel:generateGradleLintReport
Generated a report containing information about 0 lint violations in this project
...

So either we have no lint violations, including no unused dependencies, or its just not checking/reporting them. Let me see if there is some sort of option I have to set in order to turn on the lint violations checking.

Hank

[HankHerr-NOAA]

From what I can tell, I don't think we have any unused dependencies. The lintGradle tools does not indicate any, and James let me know in the office hours that he would look for unused dependencies in the past, even if we haven't been lately. Dependencies aren't added often, so if he checked in the past, it would not be surprising that we would have no unused dependencies now.

One thing I want to do before closing this is insert an unused dependency artificially to see if lintGradle catches it. I'm working on something else at the moment, but will try to get that done before COB today.

Hank

[HankHerr-NOAA]

I artificially added this to wres-system:

        //TESTING LINT
        implementation 'com.google.guava:guava:33.3.1-jre'

lintGradle did not detect that that was an unused dependency in wres-system. I thought I had set it up so that all dependencies are checked, but perhaps I'm just not configuring it correctly. I'll keep investigating,

Hank

[HankHerr-NOAA]

So nebula.lint appears to not be finding unused dependencies when I artificially add one and com.autonomousapps.build-health requires a move to Gradle 8 in order to build (despite what their GitHub claims about it being tested using 7.4 and later).

Taking a step back, tomorrow I plan to check and see if the tools we've been using already recognized unused dependencies. To be honest, when updating dependencies and using the tools that we have to support it, which I used just a few weeks ago, I didn't really check to see if they noted unused dependencies somehow. I'll check that out tomorrow. I have other work to do for the rest of today.

Thanks,

Hank

[james-d-brown]

I think the tools we use for deployment are focused on "positive" inspection, i.e., listing dependencies, checking for updates, checking their CVEs etc. This is more like a "negative" task, i.e., list unused dependencies or dependencies that should be explicit and are instead transitive. I don't recall using tooling to check for unused dependencies, but I did often scan the build.gradle visually. A tool is obviously better.

[james-d-brown]

I mean, you could probably use the built-in tasks (e.g. dependencies) to list the dependencies and then cross-reference against the build.gradle, but that isn't really a tool.

[james-d-brown]

So nebula.lint appears to not be finding unused dependencies when I artificially add one and com.autonomousapps.build-health requires a move to Gradle 8 in order to build (despite what their GitHub claims about it being tested using 7.4 and later).

Taking a step back, tomorrow I plan to check and see if the tools we've been using already recognized unused dependencies. To be honest, when updating dependencies and using the tools that we have to support it, which I used just a few weeks ago, I didn't really check to see if they noted unused dependencies somehow. I'll check that out tomorrow. I have other work to do for the rest of today.

Thanks,

Hank

Did you add any rules? The linked article seems to suggest these should be made explicit, such as:

gradleLint { 
    rules= ['unused-dependency'] 
}

[HankHerr-NOAA]

Yes. The rule was,

    gradleLint.rules = ['all-dependency']

I would think that would capture unused-dependency, but let me test that independently to see if the behavior changes.

Hank

[HankHerr-NOAA]

It made no difference using 'unused-dependency'.

I'm almost certainly doing something wrong, given that I added guava to wres-system and nebula.lint didn't seem to notice it. I'm just not sure what yet.

I mean, you could probably use the built-in tasks (e.g. dependencies) to list the dependencies and then cross-reference against the build.gradle, but that isn't really a tool.

Brute force is an option for this if all else fails, so I might do just that, James. Or, as I mentioned above, I may tackle the Gradle 8 upgrade, first, since that has to happen eventually, and then tackle the unused dependencies later.

Thanks,

Hank

[james-d-brown]

Did you add it with global scope, i.e., to capture all sub-projects?

[james-d-brown]

It is probably something simple. More than likely, the tool isn't broken or unusable. Our build.gradle is fairly complex at this point too.

[HankHerr-NOAA]

Returning to this...

Did you add it with global scope, i.e., to capture all sub-projects?

Yes. I applied the plugin and declared the rule in the subprojects tag, as well as the wres-system portion just as a test.

Agree on it probably being something simple. Just have to figure out what,

Hank

[HankHerr-NOAA]

This is probably a stupid question, but look at the source... Are we using groovy or kotlin in our build.gradle?

Hank

[james-d-brown]

Groovy, the default :) Yeah, it won't work with Kotlin, I spotted that too.

[HankHerr-NOAA]

Thanks, James. Just noting here that I'm going to try to up the logging somehow to see if I can get detailed information. Looking in the github repo, https://github.com/nebula-plugins/gradle-lint-plugin, its not obvious to me how to do that or if it can be done.

First, however, I have about 90 minutes of meetings,

Hank

[HankHerr-NOAA]

Looks like log4j, and, if this properties file is correct and used (no idea), it should be debug by default:

https://github.com/nebula-plugins/gradle-lint-plugin/blob/main/src/main/resources/log4j.properties

Hank

[HankHerr-NOAA]

At this point, it might just be easier to do a manual check. Let me see what would be involved, starting with the command to obtain the dependencies.

Hank

[HankHerr-NOAA]

I started with dependencyCheckAnalyze which didn't really give me what I had hoped: a list of used dependencies. I then ran just dependencies. That gives me a tree showing transitive dependencies, as well, but doesn't actually do any checking to see if something is used; it appears to be solely based on the build.gradle. I say that because I added Guava to wres-system, which doesn't use Guava, and saw this in the tree for wres-system:

|    \--- com.google.guava:guava:33.3.1-jre
|         +--- com.google.guava:failureaccess:1.0.2
|         +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
|         +--- com.google.errorprone:error_prone_annotations:2.28.0
|         \--- com.google.j2objc:j2objc-annotations:3.0.0

So I think it is actually a tool that I need, and that means fighting through the problems of the two tools I've found so far, or finding another one.

I'm going to put this task on hold and revisit it next week, hopefully. I want to take a look at the Podman stuff Evan has been working on.

Hank