Automatically download pods for tasks that are defined in a dependency

[logseq-cldwalker]

Is your feature request related to a problem? Please describe.
When I pull in a bb task via a gitlib dependency that depends on a pod, I would like for the pod to automatically download like clojure dependencies do. The specific use case I have is https://github.com/logseq/bb-tasks which provides tasks that depend on the clj-kondo and fswatcher pods.

Describe the solution you'd like
Tasks can declare the pods they depend on via deps.edn or some other declarative means.

Describe alternatives you've considered
For now, I tell users of my library to explicitly declare the pods in their bb.edn. Another workaround @borkdude mentioned is explicitly using load-pod in the library task

Additional context
n/a

[borkdude]

I thought more about this and I think a good approach would be that every pod would have a little wrapper library in their repo which loads the pod for you. Then you would just include a specific version of that library in your bb.edn and the tools.deps stuff would automatically pick the newest one.

By doing it that way, we wouldn't even need the :pods entry in bb.edn anymore. /cc @cap10morgan

[cap10morgan]

I like your proposal, @borkdude. It would also help solve an issue I'm running into where I need to package up code & pods for a different platform than the one I'm building on (darwin/arm64 vs. linux/arm64). This wrapper library approach could be helpful for that as the dependency itself would be platform-neutral. But it could then load and cache whichever pod binary was appropriate for the platform it was running on.

I'm going to start exploring this approach if this still sounds good to you.

[cap10morgan]

One thought I had for a potential implementation: The "wrapper library" could contain nothing more than the manifest.edn file in a well-known location (e.g. resources). Babashka itself could look for these on the classpath on startup, populate the pod-namespaces collection from them, and then load the pods similarly to the way it does now. This might be nice for non-Clojure pods b/c it minimizes the amount of Clojure-y stuff they need to learn to publish their pods under this new approach, and it eliminates the need for the centralized pod registry. You would publish the wrapper lib anywhere tools.deps can get it from and then the manifest.edn points to where to download the binaries.

Am I missing something that makes that a bad / non-viable approach?

[borkdude]

Sounds good!

[borkdude]

@cap10morgan While this issue is in hammock state, I have another nice trick for libraries wrapping pods:

https://github.com/babashka/instaparse.bb

That library only has .bb files so it can safely be included alongside JVM dependencies without conflicting.

It currently does the load-pod call so the user doesn't have to, but once we have a better grasp on the above issue, we could include the manifest directly in the library, without even having to publish the manifest to the registry.

[cap10morgan]

Cool! I need to re-gather the issues I ran into exploring the manifest-only approach and keep thinking about potential solutions to them. I think you and I discussed them in Slack, so I should maybe go find that conversation and paste some of the outstanding issues in here.

[borkdude]

I think we could maybe support (load-pod 'foo/bar) in the wrapper library and then it will search the classpath for manifests for foo/bar which can be present in the wrapper library, e.g. in META-INF/babashka/pods/manifests/foo/bar/manifest.edn. Namespaces that depend on the pod being already loaded should load that one namespace. Users of the pod use only the wrapper library and never call load-pod themselves.

The prepare command can search the classpath for all META-INF/babashka/pods/manifests manifests and pre-download them.

This we way we could also avoid scanning the classpath pre-maturely on startup of bb.

[cap10morgan]

Ran into a need for this today, so I'm pondering circling back to working on it. I was trying to share some bb tasks across a few different projects as a library, but realized this and the fact that tasks aren't merged from libraries kind of made it awkward.

In addition to the wrapper library approach for pods, would it also make sense to support something like a :load-tasks true option on deps in bb.edn so that a project can pull in tasks from a bb library? Or maybe something like :tasks-prefix foo to avoid task naming conflicts?

[borkdude]

It might be interesting to see this decision matrix that Jude Payne made:

https://docs.google.com/spreadsheets/d/1kM0eDfQRPOYyL4ov5R8z2rA8DXu8JJoHrj7XQpcqrp4/edit#gid=0

I think the wrapper library makes most sense, at least that's what worked for me with clj-kondo-bb and instaparse-bb.

Tasks were never meant to be exported as part of a library, they should be mere short-hands for what is relevant inside of a project. If you want to share stuff, just write a regular library I'd say?

[cap10morgan]

That is interesting, thanks for linking. And yeah wrapper library seems like the way to go for pod deps.

As far as a regular library for my tasks use case, I might be misunderstanding what you mean, but for my case, babashka is critical. I have a code linter that runs cljfmt and clj-kondo that I also setup as a git pre-commit hook that only operates on staged changes (not the current working directory like all examples of this kind of thing show). So it's enough code that sharing it as a library is very useful, and babashka makes it performant enough to do all in Clojure. But the big stumbling block is the repetitive boilerplate to install the clj-kondo pod and setup the two tasks (one to lint the codebase on demand and another to install the git hook) in each repo. It would be great if I could just include the library as a dep in bb.edn and then tell it to pull in the tasks exposed by it. I'm sure there are other ways to do something similar, though.

[borkdude]

It would be great if I could just include the library as a dep in bb.edn and then tell it to pull in the tasks exposed by it.

You could perhaps just use bb -x foo.bar/baz for this?