Suggestion: split out fidelity tester, maybe move to Khronos Group org, adopt Git LFS

[bhouston]

Description

The fidelity test app is phenomenal. I share it constantly around. I am not sure if it is best in this repository. The repository has a lot of stuff in it. It makes it hard to find the fidelity tester. The fidelity tester also seems to be oriented around model-viewer at first pass, when really it is about the Khronos PBR material model.

Also the fidelity tester makes this repository excessively huge. Even if it doesn't move over to Khronos Group separating it into its own repository would make model viewer more manageable.

We have done similar test suites in the past and we used Git LFS for the files. It works quite well. It is a way to keep the repository relatively small and fast while also storing the history. It would be a perfect fit for this use case.

[elalish]

Thank you! Agreed, and in fact I proposed to 3D Commerce years ago to move it to Khronos as an alternative/helper to binary certification, but they weren't interested 🤷. The main reason we haven't moved it out (other than laziness) is that it's part of our CI regression testing, which in turn serves as three.js PBR regression testing. And yes, Git LFS would be a good choice here, though cloning with depth=1 has been a decent work-around. If you're interested in helping out here (or finding someone who is), I'm open to ideas on how to restructure things.

[bhouston]

We could separate this out to a separate project and use Git LFS. I could set it so that it auto-deploys to a separate website via Google Cloud Run on successful main merge.

We could publish it as an independent website or ask to associate it with Khronos Group, and maybe move it into that org - as it is mostly a Khronos glTF / PBR fidelity test?

We could ensure that it is structured so that model-viewer can continue to use it for fidelity testing and as its unit test framework? Or you could just leave it duplicated if that is easier.

[bhouston]

Wrongly closed.

[elalish]

As I say, if you're willing to take it on, I'll be supportive. I'd prefer to avoid duplication if possible. How would you structure the separate repo so that we can still use it in our CI here?

[elalish]

To be fair, our CI only uses the model-viewer regression test part of this. It would be great to make the renderer harness its own package that we could simply depend on. Likewise, the most valuable thing for us would be getting updated configs and such if more people are invested in keeping up with the sample model updates.

[elalish]

Update: The Khronos 3D Commerce group sounds like they may be interested in taking this over after all! Upon further reflection, I'm less worried about duplication, and I may even simplify what our CI uses once the rest of this is gone.

Of course none of this will fix MV's gigantic repo clone size, since those images are still in our history (and we'll keep updating the MV goldens for our CI tests anyway). In a perfect world I'd use git-filter-repo and and git LFS to clean this up properly, but it sounds like I'd need to move it to a new Github repo to do that properly, which seems problematic.

[bhouston]

I think you can just move it to a new repo re-create it using GitLFS and then use git submodules to bring it into this repo when you want to run CI.

[bhouston]

@elalish can you connect me with the people who want to take it over? I would like to be involved.

[elalish]

Yes, I've been talking with the Khronos 3D Commerce group about this, though they said it would probably work via 3D Commerce putting budget into the Khronos Tools TSG so that they can own it. Very early stages, but so far we're discussing on the Commerce calls, Thursdays at 8:30am Pacific time.

[elalish]

@bhouston Good news from 3D Commerce! They want to move forward with taking ownership of this render fidelity repo. I recommended that we merge it into the new glTf-Sample-Assets repo, since that will make it easier to keep them in sync. There is a Tools TSG meeting tomorrow (the 3rd) to discuss this, but it's at 7am pacific so I can't make it. Are you in a better time zone for that? Would be great to have perspective from someone who knows the code. Keep an eye out for an email from Leonard with the invite/agenda.

[bhouston]

I will make an effort to attend!