Idea: neil add shadow-cljs

[rads]

Rationale

I recently converted a project from figwheel-main to shadow-cljs and now have a pretty good understanding of the challenges when using these libraries in 2022. Though figwheel-main was working fine once I set it up, I ended up switching to shadow-cljs for more community support. That said, when setting up both build tools, I had to parse a lot of information to build a minimal but complete dev workflow that included a ClojureScript REPL, tests, CI, etc...

One option to make this easier now that neil new supports external templates would be to create a configurable shadow-cljs template, which is similar to the approach of frameworks like kit-clj. That's a totally valid approach and it's why I'm also working on a PR for kit-clj to add deps-new compatibility which will enable this top-down workflow with neil.

That said, the template approach doesn't work as well for adding shadow-cljs to an existing project. This is where neil has an opportunity to shine, providing a bottom-up workflow in addition to the top-down one. Beyond that, if we could have a tool that knows how to add shadow-cljs features incrementally with sane, modern defaults, this would be a boon to ClojureScript developers everywhere.

Example

# Create empty project
$ neil scratch my-project
$ cd my-project
$ neil add shadow-cljs

# Start a watcher for app and tests
$ clj -M:shadow-cljs watch app test
shadow-cljs - HTTP server available at http://localhost:9500
shadow-cljs - server version: 2.19.8 running at http://localhost:9630
shadow-cljs - nREPL server started on port 51484
shadow-cljs - watching build :app
shadow-cljs - watching build :test
[:app] Configuring build.
[:test] Configuring build.
[:app] Compiling ...
[:test] Compiling ...
[:test] Build completed. (147 files, 1 compiled, 0 warnings, 5.38s)
[:app] Build completed. (417 files, 0 compiled, 0 warnings, 5.55s)

# Auto-run and hot-reload CLJS tests
$ open http://localhost:9500

# Generate an optimized bundle for production
$ clj -T:build bundle
added 652 packages, and audited 653 packages in 6s
[:app] Compiling ...
[:app] Build completed. (384 files, 0 compiled, 0 warnings, 22.52s)

# Generate an uberjar with the bundle step included
$ clj -T:build uber
$ cd target && jar tf *.jar | grep main_bundle.js
public/js/main_bundle.js

Requirements

Note: Due to the number of features in shadow-cljs, we may consider options within neil add shadow-cljs to generate only a subset of the features below (e.g. --karma-tests false if you don't want karma.conf.js).

• neil add shadow-cljs should fill in the following files as needed:
  • deps.edn
    • Add :shadow-cljs alias using example from shadow-cljs tools.deps integration docs
    • Use :extra-deps to add shadow-cljs dependency
  • shadow-cljs.edn
    • Add :deps true to enable tools.deps integration
    • Add :builds :app with :target :browser
    • Add :builds :test with :target :browser-test
    • Add :dev-http to run tests from :builds :test
    • Add :builds :ci with :target :karma
  • build.clj
    • Add bundle function
    • Update uber function to include bundle
  • package.json and package-lock.json
    • Includes dependencies for :target :karma
    • Used for adding NPM deps with shadow-cljs
    • Can also be used to enable npx shadow-cljs command, but this is optional with tools.deps integration
  • karma.conf.js
    • Used for :test build in shadow-cljs.edn
  • .gitignore
    • Ignore .shadow-cljs directory
  • .github/workflows/cljs-tests.yml
    • Auto-run :ci build on push to main

[borkdude]

I'm using shadow-cljs in a couple of projects myself and I find neil add deps thheller/shadow-cljs sufficient for now. Taking on all the options that shadow-cljs provides is imo too broad of a scope for this project. Also it's not always the case that you want to bundle the assets that shadow produces into your uberjar, if you create an uberjar at all.

[borkdude]

I feel it's more the job of shadow itself to create such .edn files based on command line options.

[borkdude]

To add, I'm using shadow-cljs mostly for testing or producing libraries and publish them to NPM. So that doesn't align at all with the above proposal for example.

[rads]

@borkdude: I totally get where you're coming from. I should clarify that I want to solve a specific problem here: I believe ClojureScript is too hard to set up and use to build web apps in 2022. Disclaimer: This is just my opinion and I'm incredibly grateful for all the work that the community has provided to us for free.

Nonetheless, there's glue that's often missing and my dissatisfaction with the developer experience motivates me to solve the problem however I can, whether that's through neil or shadow-cljs or something else.

For example, understanding what goes in all those files above just to get started was an annoying process for me even though I've been writing ClojureScript professionally for 5+ years now. The steps need to be performed manually if you want to add these things to an existing project instead of starting from a template. I know people who could really enjoy ClojureScript but may not be able to get past all these little steps on their own.

Anyways, rant over. This project is awesome and I want it to stay that way. With all this in mind, I see a few possible approaches for me to move forward:

1. Document usage of neil add deps thheller/shadow-cljs and the above files
   • Not enough of a solution from my point of view since it requires a lot of manual understanding to build a simple web app, but it could be a good starting point for the following options
2. Add improvements to shadow-cljs itself so the wrapping isn't needed
   • Probably the best in the long-term but will likely require a lot more time and effort to get changes inside shadow-cljs itself than it would to get into a wrapper tool
3. Create a separate tool from neil to implement what I described above
   • This seems like a good way to experiment for now, though it means there would be another tool for people to install and learn

Since this isn't really relevant to neil anymore, I can go ahead and close the issue.

[borkdude]

@rads I think a solution to the above could be a neil new shadow-browser-app template or so.

[borkdude]

And perhaps host it with shadow-cljs so it can be used as neil new thheller/shadow-cljs ... or so?

[borkdude]

Or we could still have a neil add shadow-cljs --browser thing perhaps. I just found the above a bit too much. E.g. adding things to build.clj could be a bit too interfering.

[rads]

Thanks for moving this to a discussion. Happy to keep the conversation going while I get a better understanding of the problems here. No rush either, just glad to to be involved.

[borkdude]

I think it's better to start adding a small feature and then see how it behaves and then add more it over time. I don't think neil has to fully "manage" your project by adding things consistently. So generating a shadow-cljs.edn file with a browser config based one a name, based on some flags, seems fine to me. And then we'll see what else we should do. Some people host assets on S3 or via nginx instead of an uberjar, so that is one of the things that could get in the way if we would add that to build.clj.

So: better start very minimally, then refine.

[rads]

Or we could still have a neil add shadow-cljs --browser thing perhaps. I just found the above a bit too much. E.g. adding things to build.clj could be a bit too interfering.

Yeah, my original intent in the proposal was just to be thorough in listing everything needed for a real app. I can come up with something more limited that doesn't mess with build.clj. Given the original list of files above, do you have any guidelines for which files are fair game to update vs. which ones we should leave alone?

[rads]

Just saw your post above. Will use that as a starting point.

[borkdude]

@rads After reading this, I think it makes sense to have neil emit some bare-bones shadow-cljs setup for browser app development. Let's just start with the most minimal thing and then let's build it out.

neil add shadow-cljs --browser

or so?

[rads]

@borkdude: I want to get things started for the neil add shadow-cljs command. Do you have any specific requirements here? Otherwise I can make a minimal PR and we can go from there.

[borkdude]

Yes, let's just do it. Minimal PR sounds great.