Ability to specify a non standard bb.edn file path

[lispyclouds]

Context

Currently for using a bb.edn file, the canonical location for it is the same PWD where we are launching the bb command. This works out well as the paths referred in it are all relative to the location of it.

Issue

For any case bb needs to be invoked from any other location, its not straightforward to specify the location of bb.edn and also execute the script(s) relative to it.

Proposal 1

We model it like docker in which we can specify a path for a non standard location of the Dockefile via -f and it takes the context path. All the referrals in the Dockerfile are relative to the context path not the Dockerfile path.

We could introduce --bb-edn opt to specify the path, defaulting to . and --context or --project-root or something similar to specify the context. The defaults to . too. All the paths in the bb.edn are relative to this.

[borkdude]

My initial response is: just write a bash wrapper script that cds into the right directory and then calls bb? But perhaps I'm missing something.

[lispyclouds]

My thoughts are stemming from wanting to treat bb as a task runner in the same spirit as gradle, make etc which allow a way to specify the path to the makefile/build.gradle etc when the command isn't invoked from the same path as the build file.

[thiagokokada]

My initial response is: just write a bash wrapper script that cds into the right directory and then calls bb? But perhaps I'm missing something.

This is what I did initially, but sometimes you can't cd to the directory since this may break some other assumptions.

For example, let's say you have a script that receives a directory as parameter and creates a file in the directory passed as parameter. Now let's say the directory that is passed to this script is the . (or any relative directory). If you do a cd before calling this script, now it will create a file in the directory of the script instead of the directory where the script is called, and this is unexpected.

Sadly since, AFAIK, you can't change the directory inside Clojure/Babashka too (without using clojure.java.shell or babashka.process), it means that you can't simple switch back to the original directory, so now you need to create some code to get the path where the script is called and write some code to correctly handle the paths.

[thiagokokada]

BTW, the first proposal is a (possible) breaking change, and while it is more user friendly I think it looks kinda magic.

I prefer the second proposal of a explicitly flag setting the context. I really liked the --project-root for that parameter, it is very clear what it is doing and does not look like magic as the --bb-edn would be.

[thiagokokada]

Just to get an inspiration from a popular tool that provides something similar: git.

Git has the --git-dir that allows you to pass the path to the .git repository from a project and run commands from them. There is also --work-dir that allows you to change the path of the working directory (a.k.a. PWD) for the git command.

Since most of the time on git you expect to run the commands inside the directory that the repo is container (that is, it is the opposite use case that I described here), you generally need to use both flags together (something like --git-dir=project/.git --work-dir=project). But not in our case.

So the flag could be called --bb-edn-file maybe, inspired by --git-dir.

[borkdude]

We cannot change the PWD in babashka once it is started. Does this affect any of the above?

[thiagokokada]

I can't think of a use case where not having the ability to change PWD would affect the issue described above.

Like the case of setting environment variables, changing directories is possible in other scripting languages (Bash, Python), and would be a nice to have, but I think in most cases it is workable (since clojure.java.shell and babashka.process both seems to support arbitrary directories).

[borkdude]

Are you suggesting that we should tweak those libs to take into account the --bb-edn-file directory? I'm very confused right now.

[thiagokokada]

No, the libraries works perfectly fine as it is.

[borkdude]

Some new insight, while I was working on nbb (https://github.com/borkdude/nbb). It seems that in the Node.js ecosystem, the file location from the script is somehow also playing a role in dependency resolution: the relative node_modules directory of the script is taking into account, no matter where you invoke the script from.

I don't think the clojure CLI does this.
E.g. foo/deps.edn + foo/script.clj + clojure foo/script.clj will not use foo/deps.edn but only the local deps.edn
But I think it would make sense if it did.

[smessems]

In order to support both the ability to run tasks from different locations and to be able to maintain multiple configuration files ( like admin-tasks.edn and dev-tasks.edn ), I think we need the ability to specify the following independently:

• Configuration file: Like --conf my-tasks.edn -> default to bb.edn
• Project root: Like --project-root -> default to configuration file location
• Working dir: is just where the user is running the task ( no way to control that )

The configuration file could also specify internally the project-root as a key/val.
We might need to provide a (project-root) so tasks could build paths on top of that.

So, if I have an admin-tasks.edn in the project root - I could run it from anywhere with the --conf option. The project root would be the location of the admin-tasks.edn file. I could do things on files on PWD or use the (project-root)
And if I have a "scripts/admin-tasks.edn", I might put a {:project-root "../"} inside

Would that make sense?

[smessems]

BTW, for the reference, a way to get the tasks always to run from the root, one can use git behavior:
git config --global alias.bb '!bb '

The git's bang invoke is always from the repo root. Then you can invoke it like `git bb tasks'

[adam-stokes]

Just to add on to this discussion, I attempted to do something similar here: https://clojureverse.org/t/help-utilizing-babashka-tasks-globally/8026. The result was having to cd into the directory housing bb.edn so that the third party dependencies would resolve properly. Probably something to think about here as well when thinking about the implementation details

[borkdude]

I think we could make bb:

• pick up on the bb.edn relative to a script
• :paths should be resolved relative to the bb.edn

So given:

/foo/bar/bb.edn
/foo/bar/script.clj

this invocation

/tmp $ bb /foo/bar/script.clj

will use /foo/bar/bb.edn and :paths ["src"] will resolve to /tmpfoo/bar/src.

What are some things that can break if we introduce this behavior?

[adam-stokes]

I would have to go back and re-run but I think I ran into issues when I attempted to load third parties deps that weren't part of the bb distribution

[borkdude]

We could also make an API to define tasks in a standalone script

(babashka.tasks/deftasks '{a ... b ...})

How would one use this if it was possible?

[borkdude]

I wonder how you would use that if you did this in global_scripts/script.clj.
a and then (babashka.tasks/run (symbol (first *command-line-args*))) to run a?

[borkdude]

I think (currently) you would still have the PWD + relative paths problem
2:05 PM
so we might as well go with the "use bb.edn relative to script" solution
2:08 PM
This would allow you to make a global bb.edn with multiple global scripts that can use it
2:08 PM
and then put that dir on the path

[borkdude]

node behaves like this:

/foo/node_modules
/foo/script.js
/bar # no node_modules

/bar $ node ../foo/script.js # will use /foo/node_modules

[KDr2]

Any update on this topic?

How about referring to $HOME/.bb.edn after it fails to find a bb.edn in the project?

I need this global bb.edn feature no matter how it is designed/implemented. Thank you.

[lispyclouds]

Its implemented for some time now with the --config and the --deps-root options.

How about referring to $HOME/.bb.edn after it fails to find a bb.edn in the project?

Do you mean bb should be looking for a bb.edn recursively upwards from the the pwd too?

[KDr2]

Oh, great! thank you.

I meant that try $PWD/bb.edn first, if that file doesn't exist, then try $HOME/.bb.edn. Of course, this behavior is expected when no --config or --deps-root options are provided.

[KDr2]

Or use ~/.babashka/bb.edn instead of ~/.bb.edn. This will be very convenient coz it will save us from typing --config ....

[KDr2]

In other words, if there's no bb.edn in the current dir and no (--config and --deps-root) options are given, make it behave like bb --config ~/.babashka/bb.edn --deps-root ~/.babashka.

[borkdude]

There's already enough flags and features and I don't want to complicate it further by looking at a home folder. People will expect things to be merged etc and I don't think it's worth the confusion. Just write a tiny bash wrapper script if you want to do this.

We do have something new though: when you have a script foo.clj and there is a bb.edn adjacent to it (in the same directory), when you add foo.clj to the path and invoke foo.clj from anywhere on your system, it will respect that bb.edn so your script can have dependencies.