Rework website build

[aepfli]

JUnit Pioneer was initially designed as a basis for multiple project and till the recent past, there was only JUnit Pioneer. This has changed with Test-Reports. Now we actually do have multiple repositories which should trigger the website build.

We can not rely on an input parameter, because the trigger can only provide insight to the project it used, and not to the other project. Our fast workaround is just relying on the latest tag for the build as @Michael1993 did in his PR.

Our future goal, or the long term plan is to change the general workflow of the website build. Instead of fetching what is needed from the projects, the projects will build and commit their changes to the website. The website itself will be pretty "stupid" build wise, and will just trigger a new page build, without any fancy logic of fetching other repositories etc. (at least that is what we talked about).

To summarize, the basic problem lies within our project structure, and our basic "need" to build the page for multiple repositories. The current solution is not fit to that (as we could update the documentation to show unreleased features). Therefor fast mitigation is looking for tags within the website build. Long term mitigation is committing prepared data to the website, and not fetching anything from outside at all. (this should also make the build way easier to understand).

Additionally we might be able to remove Gradle from the build at all with this step, and reduce complexity

[nipafx]

I thought about the site setup's inarguable complexity as well. Some drivers behind that:

• the need to pull in documentation from "various subprojects"
• GitHub's (old?) org page setup with the website's HTML source required in master
• Jekyll's limited design which only reads files and requires Liquid and/or Ruby for any kind of customization

The first can indeed be addressed by pushing documentation to this repo, but doesn't that just move the complexity elsewhere? Then every subproject is needs to clone a repo, know where to put files (.adoc and now also demos), and push a commit. And the logic how to combine those files still needs to be expressed either in Liquid/Ruby or in an external process (currently in Gradle build files).

I think we can get further by uprooting the entire stack. Gatsby has the concept of sources, which pull in content from anywhere. The most obvious one is a file source (searching the file system), but it should be possible to pull in files from the most recent v...-tag of a GitHub repo as well. Gatsby also makes it trivial to include custom logic in how to build a site. If we also move away from publishing the final HTML in master, we can greatly simplify this setup:

• no more Gradle (all build logic and combination of sources in Gatsby)
• no more worktrees (files are pulled via Gatsby source)
• all sources in main with build published to gh-pages (as in other projects I recently worked on)

[beatngu13]

As already mentioned in Discord, may I suggest another alternative: Antora

The multi-repository documentation site generator for tech writers who ❤️ writing in AsciiDoc.

This may have some limitations in terms of design, but personally I'm OK with that.

[nipafx]

Antora

Yes, that's definitely a good idea. I just read a bit about it and it looks promising - this simple playbook example is pretty much what I was hoping for. Would be nice if branches can be set to something like latest, but if that doesn't work that wouldn't be a deal breaker to me. Will also be interesting to see how this works with external snippets.

Another aspect is that I considered taking this revamping opportunity to self-host our Javadoc. That would mean downloading each release's JAR and unpacking that into a target folfer.

Gatsby

Before seeing the Antora recommendation, I played a bit with how this could work with Gatsby. There's a GitHub source, which works with GraphQL by forwarding it to the GitHub v4 GraphQL API. GitHub hosts a pretty handy explorer, but I couldn't create a query that returns a single folder from the latest release, but it would work in two queries:

{
  rateLimit {
    cost
    remaining
  }
  repository(name: "junit-pioneer", owner: "junit-pioneer") {
    latestRelease {
      # returns, e.g. 1.6.1, which we then feed into a second query...
      tagName
    }
    # ... here
    object(expression: "v1.6.1:docs") {
      ... on Tree {
        entries {
          name
          object {
            ... on Blob {
              text
            }
          }
        }
      }
    }
  }
}

Links:

• https://gist.github.com/MichaelCurrin/6777b91e6374cdb5662b64b8249070ea
• https://mohamedsahbi.com/2019/12/24/fix-a-gap-in-github-api-v4/
• https://git-scm.com/docs/git-rev-parse (for object(expression: ---))

Gatsby sources pull in content (e.g. /docs/*.adoc files from our repos' latest releases) and puts it into the GraphQL store. Then, transformers transform (e.g. from Asciidoc to HTML). Here, again, I'm curious to see how that would work with external snippets because my impression of gatsby-transformer-asciidoc is that it just uses AsciidoctorJS, which operates on files / file paths. Hm...