Excluding files/folders with `dhall-docs`

[SiriusStarr]

(Apologies if there's a way to do this already that I'm just unaware of.)

The current behavior of dhall-docs is to produce documentation for every file with the *.dhall extension in its input path, including those that have no valid header comment in the dhall-docs fashion (the docs generated in such cases including just the literal source of the file).

In a current project, I have a large number of Dhall files that I don't want documentation generated from, as they just hold data (and have no header comment), but a number of files I absolutely do want documentation for. It would seem useful to have a way to exclude such files from doc generation.

A couple of potential options:

• Opt-in: Ignore files that do not begin with a comment in the form {-| or --| (This, to me, seems like it should be the default behavior, but could have a command-line flag to turn it back on.)
• Opt-out: Have a special header comment, like {-# NO_DHALL_DOCS #-}, that instructs dhall-docs to ignore the file (not a huge fan of this option, personall).
• Explicit exclusion: Provide for an exclude list of some sort, whether via a file (.gitignore) or a command-line option (--exclude Data/*, probably the better choice)

[TristanCacqueray]

Supporting .gitignore sounds like a good thing to do by default, are your Data files already listed in .gitignore?

[SiriusStarr]

Supporting .gitignore sounds like a good thing to do by default, are your Data files already listed in .gitignore?

No, definitely not, as they need to be in the VCS; I was just saying a separate, .gitignore-esque file (potentially). Though I definitely agree ignoring .gitignore would seem to make a certain amount of sense (though maybe not a good idea to not be VCS-agnostic?).

Opt-in behavior truly makes the most sense to me as I don't see the value added by just reproducing the source for every Dhall file in the docs (and if you wanted that behavior, it's pretty trivial to put an empty {-| -} at the first line of the file, or to have a flag to turn it off).

[Gabriella439]

@SiriusStarr: Part of the reason we generate documentation for all source files (even those without documentation) is so that the user can jump between them by clicking on imports. If we were to not generate anything for them then "jump to import" would no longer work for files missing documentation.

[SiriusStarr]

@SiriusStarr: Part of the reason we generate documentation for all source files (even those without documentation) is so that the user can jump between them by clicking on imports. If we were to not generate anything for them then "jump to import" would no longer work for files missing documentation.

I guess I don't see the value for jumping to non-existent documentation, since it's just viewing the source code with worse formatting/syntax-highlighting/etc. It seems like a good compromise might be to generate documentation for all files but not add them to the visible index? So you can still follow clicked imports, but only files with actual documentationt show up as "Exported Files/Packages?"

Regardless, if we want to keep the current default behavior exactly as is (understandable), I would be good with just any way to opt out of generation (e.g. exclude directories or whatnot), since I'm going to end up generating docs for ~8000 files when only like 20 have any documentation (and don't link to any of the others), which makes the documentation really confusing to read for others.

[Gabriella439]

@SiriusStarr: So I've implemented this on a branch, but I suspect that this will be a highly breaking/surprising change for most users, so I want to better understand the motivation for this to see if there's a different solution.

What is the main reason that you want to omit the files without documentation from the index? Is it to improve the performance of dhall-docs or to make it easier to navigate the documented files?

[SiriusStarr]

@Gabriel439

What is the main reason that you want to omit the files without documentation from the index? Is it to improve the performance of dhall-docs or to make it easier to navigate the documented files?

More the latter than the former, though I am concerned about having 10,000+ extra files. The repo that prompted this has a layout like this:

.
├── buildJsonDb.dhall  -- *docs
├── buildJsonFiles.dhall  -- *docs
├── A
│   ├── ...several subfolders with ~3000 data files
│   └── db.dhall
├── db.dhall  -- *docs
├── Internal  -- *docs
│   ├── TypeA.dhall  -- *docs
│   ├── TypeB.dhall -- *docs
│   ├── ... etc
│   ├── Subfolder -- *docs
│   │   ├── TypeC.dhall -- *docs
│   │   └── ... etc
│   └── TypeZ.dhall -- *docs
├── B
│   ├── db.dhall
│   └── ... ~200 data file
└── C
    ├── ...several subfolders with ~8000 data files
    └── db.dhall

Basically, there are ~10,000 Dhall files that are just data, constructed using the types/functions in ./Internal/. Everything in that folder as well as some top-level stuff (like ./buildJsonDB.dhall) has documentation to be used with dhall-docs, but the ~10,000 other files have no documentation and no reason to be documented.

So currently, dhall-docs generates many, many, many more HTML files (and links to them) than are necessary, which makes it unintuitive to navigate as well as just having a bunch of extra files in the docs folder.

Any of the three approaches I described previously (opt-in, opt-out, or ignore) could work in this case, though I'd prefer to avoid opt-out, since it would require adding whatever we decided the {-# NO_DHALL_DOCS #-} comment would be to 10,000 files.

Which is to say that if people feel like not generating documentation for files without doc comments is too unintuitive, then it's no skin off my nose to use one of the other approaches (or to make that behavior a flag?). Though I'm still a little confused what the value is to generating docs for files with no docs (not judging, just hasn't come up for me 😆).

[SiriusStarr]

This is off-topic, apologies, but thinking back on:

Part of the reason we generate documentation for all source files (even those without documentation) is so that the user can jump between them by clicking on imports.

If we're going to keep generating documentation for undocumented files, then it would be really neat to automatically create an import section (sort of like how "examples" are automatically pulled out right now). Sort of like neuron's generation of uplinks/backlinks (see e.g. the bottom of this page).

[Gabriella439]

@SiriusStarr: dhall-kubernetes is a good example of a package that is useful for users to browse (e.g. to discern the expected type of a resource) even though basically very module lacks a comment header. Another common example is the ./package.dhall file that most packages have, which is commonly a plain record without comments, and we'd definitely want users to be able to navigate to that (if only to transitively navigate to its fields).

One possible way we could address the signal-to-noise ratio is having the documentation generator split the index into two sections: documented modules (at the top) and undocumented modules (at the bottom). That would make it easier for users to easily discover the modules that are documented.

I also really like the idea of automatically lifting imports to the top of the file, just like we do for tests

[SiriusStarr]

One possible way we could address the signal-to-noise ratio is having the documentation generator split the index into two sections: documented modules (at the top) and undocumented modules (at the bottom). That would make it easier for users to easily discover the modules that are documented.

That would definitely help cut down on visual clutter, in which case it isn't too much of a problem if there's a bunch of non-documented files included.

Another common example is the ./package.dhall file that most packages have, which is commonly a plain record without comments, and we'd definitely want users to be able to navigate to that (if only to transitively navigate to its fields).

See, to me it almost seems as if it would be useful to intelligently exclude package.dhall's (if they don't have documentation). I could imagine a case where (if there's a package.dhall) for a folder, dhall-docs would generate a special heading for the folder like:

Exported by package.dhall:

* Type : Type
* array : List JSON → JSON
* bool : Bool → JSON

Other files:

* internalRenderingFunc.dhall : Text → JSON

But that might be a lot of work for not much benefit.

Just brainstorming ideas, sorry, haha.