Best way to stay up-to-date with the template

[DevSecNinja]

Hey,

Thanks again for this amazing project! I see you’re making lots of great improvements to the template and I wonder what the best way is of rebasing your changes. I copied the template as suggested in your guide, but it seems that I should have just forked the project so that I can easily rebase. It would also be easier to propose new changes with a PR.

What do you think? Am I missing anything?

Thanks!

[onedr0p]

It's not a easy thing but you can add this repo as a remote and then use git cherry-pick to bring in changes. Also if you wanted to be part of the discussion on the direction this is going we have #flux-cluster-template channel in the kubernetes@home discord.

[DevSecNinja]

Thanks, will give that a shot!

[joryirving]

git remote add upstream [email protected]:onedr0p/flux-cluster-template.git
git fetch upstream
git cherry-pick <commit>
git push

[LeonStoldt]

Hi 🙂
I've taken an approach that's quite similar to what @LilDrunkenSmurf suggested. I've created a new task in the taskfile to streamline updates from this remote template repository:

update-from-remote:
    desc: Update from remote template repository
    cmds:
      - git remote get-url template >/dev/null 2>&1 || git remote add template [email protected]:onedr0p/flux-cluster-template
      - git fetch --all
      - git cherry-pick --no-commit --allow-empty $(cat latest-template.sha)..template/main
      - git ls-remote template HEAD | awk '{ print $1}' > latest-template.sha

update via:

task update-from-remote

Before you run this task for the first time, ensure that there's a latest-template.sha file at the root of your repository (can be created empty). This file is essential as it records the SHA of the last commit from the template. The task will cherry-pick all subsequent commits from the remote template, starting right after the commit SHA noted in latest-template.sha.

After successful cherry-picking, the task will update the latest-template.sha file with the SHA of the latest commit from the template. It enables you to review each code change in detail and make any necessary adjustments if the updates conflict with your custom modifications.

[onedr0p]

Woah nice! Would you mind me adding this to the template's taskfile?

[onedr0p]

@LeonStoldt I have added and changed it a bit, feel free to adapt or open a PR to fix anything I missed
c749ff8

[LeonStoldt]

I'm glad to hear you like it! It's great to see this task added to the template.

Just a heads-up, though... there are some issues to be aware of, particularly when dealing with merge conflicts. If a merge conflict arises, you'll need to manually resolve it. After fixing the conflict, you won't see the template sha automatically written to the file. So, you'll have to manually commit, execute git cherry-pick --continue until everything is fully resolved, and finally, manually write the SHA to the file.

Despite this, I believe it's a good start and provides good value considering the low complexity of the task.

[giannoug]

Since the "Use this template" doesn't keep history and authors, I'm using the following solution which just takes changes and applies them (in a single commit) to my repo:

git remote add upstream [email protected]:onedr0p/flux-cluster-template.git
git merge --squash --allow-unrelated-histories upstream/main

This usually creates a lot of merge conflicts, but since they all are in files not "managed" by me, I just accept all incoming changes, stage and commit them in my own commit. If no conflicts are found, git stops and asks you to finish the merge.

I'm not sure how to apply incoming changes to my cluster though. For example, in PR #1110 some renames are introduced in Jinja templates. To my understanding I need to rerun task configure and commit changes for flux to pickup, is that needed and/or am I missing something?

[giannoug]

The template could just depend (or maybe make it optional somehow) to an object storage that lives "outside" the cluster, much like it requires a DNS server for split horizon. Also, since we are talking about backup, it could be AWS or CF! Lastly, most NAS solutions offer block storage somehow, so that also fits the home server idea of this template.

Maybe it's my software engineer background that wants to treat the template as a blackbox dependency and completely trust it and use it as-is, but having a single source of truth somehow makes sense to me

[joryirving]

R2 is CloudFlare, and Minio is block storage typically living on a NAS.
But that's not necessarily. I run a test cluster that only uses local-path storage, with no external storage dependencies, so it's based on your use case.

You need DNS to make a network work, you don't need R2/Minio/NAS to make a homelab work. Adding dependencies like that is a lot of assumptions.

[giannoug]

But that's not necessarily. I run a test cluster that only uses local-path storage, with no external storage dependencies, so it's based on your use case.

So you used the template, added your own apps, defined volsync and then you could rebuild the cluster (even in other machines) from scratch restoring (selected) PVs? That's what I have in mind, which sounds feasible from what I've learned so far

[joryirving]

Nothing in my test cluster is stateful, so there's no data to lose. That's just one use-case.
On my primary cluster, I've setup rook-ceph with volsync and volume populator, sure.

I'm just saying that it's possible to have clusters using this template that don't require an external backup source.

[giannoug]

The point of the discussion is to have state and be able to update to the latest template "version" though. I took it a bit further and brought up backup-restore to a new cluster (which is practically the same thing). I'll try my luck with volsync though, its a start