Consider supporting zome only repositories

[guillemcordoba]

Currently the scaffolding tool only supports repositories with at least one happ.yaml and one dna.yaml. This covers the majority of projects, which are just web-happs that need to produce a .webhapp package to be published somewhere.

A lot of the repositories I work on don't fulfill that criteria, as they are just modules, so they only output a single zome. In these kinds of repositories, running hc-scaffold entry-type results in Error: No app manifest (happ.yaml) was found in this directory tree, because the scaffolding tool first looks for the happ and then the DNA to discover all the zomes in which the entry-type could be created.

This means that I need to keep a dna.yaml and a happ.yaml files in my repositories which are kind of "fake", as in they are only used to allow the scaffolding tool to discover the only zome in my repository. Maybe if the scaffolding tool could first scan for happs, fall back to find DNAs, and lastly fall back to finding zomes, it would improve the developer experience in these kinds of repositores.

What do you think @c12i ?

[c12i]

Hi @guillemcordoba

I would like some clarification just to better understand your proposal. At the moment, the repositories scaffolded to be used as modules rather than webhapps still contain a happ.yaml and dna.yaml despite not really needing one, since the consuming happ would import them as rust dependencies from my interactions with the profiles and file storage zomes.

Maybe if the scaffolding tool could first scan for happs, fall back to find DNAs, and lastly fall back to finding zome

I am not sure about this, since we would still need to keep the "fake" happ.yaml and dna.yaml files, which I wonder would really be needed in such a case.

What do you think about having a separate command i.e hc-scaffold module?

[guillemcordoba]

Mmm yes you are mostly right. But why would we need to keep the dna.yaml and happ.yaml files? I have an alternative way in which I produce the DNA and happ bundles without them.

What would hc scaffold module do? What I really need is to be able to do hc scaffold entry-type in that kind of repository where there are only zomes.

[c12i]

What would hc scaffold module do?

I'm seeking a better understanding on whether it's still appropriate to use hc scaffold web-app to scaffold a module instead of a web app

[c12i]

But why would we need to keep the dna.yaml and happ.yaml files?

I agree with the decision to no longer require the dna.yaml and happ.yaml files for these kind of repositories. I'm curious about the process for generating the DNA and hApp bundles. I'm assuming the bundles would be produced by the web-hApp consuming this module, right?

[guillemcordoba]

Yeah that's right. I have a method of generating DNA manifests in nix https://github.com/holochain-open-dev/infrastructure/blob/97a15932e38a3a0db5ce5b48b4989508bf91bb46/nix/fixtures/module-repo/zome/zome.nix#L18 that allows me to generate fake DNAs as needed without needing a dna.yaml file. So in the repository where the zome lives, there is no happ.yaml or dna.yaml but you can still test your app.

In the consuming repositories, there you include the zomes into your dna.yaml to actually build your app, like this. Do you have any other questions about the pattern?

[c12i]

I see, this is quite new to me

I am curious to know how the proposed method is different from what I have used before from this guide. Do they both achieve the same goal?

[c12i]

So in the repository where the zome lives, there is no happ.yaml or dna.yaml but you can still test your app.

Also another question regarding this; the fake manifests getting generated are only useful for testing within the zome's repository context, right?

[guillemcordoba]

Well, in the proposed method, you don't need actual dna.yaml or happ.yaml files, you can "fake them" in nix files so that you decide what to include and when. In zome only sweettest, I can create a DNA bundle only with the zome I want to test, which means I can test only that given zome with its integrity.

Also another question regarding this; the fake manifests getting generated are only useful for testing within the zome's repository context, right?

Yes exactly, that's the main purpose.

[c12i]

I see, so for such repos, the scaffolding tool should not be generating these manifest files, this is the only change required from the cli as far as I understand.

Or should "faking" the manifests in nix be handled by scaffolding as well?

[guillemcordoba]

Close, but not really what I mean.

If the command hc scaffold dna is run, it's totally fine that a dna.yaml file gets created. It is in fact necessary. The thing is that command will never be run in the kinds of repositories we are talking about.

What I'd like to change is that in repos where there's no dna.yaml, only zomes, I can't run hc scaffold entry-type or hc scaffold link-type. This is because the scaffolding tool does not find any happs or DNAs, so it just errors out. I'd like for it to be able to find the zomes that do exist in the repository and choose them as the options where to scaffold the entry or link-type.

I don't think the faking the manifests in nix should be handled by the scaffolding tool no. As a matter of fact I'm scaffolding them myself in the custom template for holochain-open-dev.

[c12i]

What I'd like to change is that in repos where there's no dna.yaml, only zomes, I can't run hc scaffold entry-type or hc scaffold link-type.

I see, so for such repos, the workflow would remain the same except for the fact that no dna would be scaffolded. Therefore the zome, entry-type and link-type commands should still be able to run without producing this error Error: No DNAs were found

How about the happ.yaml and web-happ.yaml manifests, they would still get generated in this case, should these too be skipped?

[guillemcordoba]

Yes same, they will never get created.

[c12i]

Great, I can add this functionality, probably as an optional flag passed to hc scaffold web-app to indicate this is a module.

[guillemcordoba]

Hmmm how is that going to change the behavior of hc scaffold entry-type? The problematic code is here.

[c12i]

The problematic code is here.

Yes, this change will also be made,

I am just considering the user who scaffolds a web-app that is intended to be used as a module, the project that gets generated should not have these manifest files by default and there shouldn't be any instruction to run hc scaffold dna

Do you think this is not necessary and they can manually delete these files to turn their hApp into a module?

[guillemcordoba]

Ah no that is already covered with the module template, which already scaffolds only the zomes and tells you not to scaffold DNAs. It's cool that this can be done without needing to change the code for the scaffolding tool!

[c12i]

Ahh I see, no need to handle this on scaffolding then

[c12i]

@guillemcordoba, I've conducted an in-depth review of the issue and would like to share my findings:

The generation of happ.yaml occurs upon execution of the command nix run github:holochain-open-dev/templates#hc-scaffold-module-template -- web-app. Consequently, the absence of this file should theoretically result in the error No app manifest (happ.yaml) was found in this directory tree when attempting to scaffold an entry or link type. This discrepancy appears to stem from how the entry-type scaffolding process attempts to "get or choose" an AppFileTree at this point in the code as well as this. These areas may contain problematic code that warrants further investigation.

A significant dependency exists between the ZomeFileTree and DnaFileTree structures. This coupling is particularly evident in functions tasked with scaffolding entry, link types, and collections, where the DNA manifest is referenced. Decoupling these components, or at least providing a default implementation for DnaFileTree in the absence of a DNA manifest, would likely require extensive refactoring. I'm curious if such an approach has been considered in your proposal, especially since addressing the initial issue you identified might not fully resolve the problem, given the presence of similar issues elsewhere in the code.

I'm keen to hear your thoughts on this analysis.