generate grammer documentation

[droundy]

It would be lovely to be able to auto-generate documentation of the grammar. It seems like at least in principle this should be possible, and would be really convenient, either when adopting (and learning) someone else's grammar, or when developing changes to the grammar. I'm imagining something that generates markdown, html or latex.

[drmfinlay]

I agree it would be nice to have this. I am familiar with the Sphinx documentation generator, which can generate HTML, LaTeX and other formats. Dragonfly's documentation uses a Sphinx extension (sphinx-argparse) to document CLI commands. It should be possible to write a custom Sphinx extension to generate documentation for grammars and rules.

@LexiconCode I recall you have brought this sort of thing up before. Maybe you have done something like this with Caster's documentation?

[LexiconCode]

This would be an awesome feature to have. Generating documentation from grammars has not been implemented in Caster. The challenges not to so much generating docs it's more of extracting the information out of the grammar in a way that is useful.

As a side note in case dragonfly implements CCR in the future. CCR adds a extra challenge of managing grammars. It's important to expose grammars before they become part of the CCR merged state. That would allow for displaying commands as individual grammars or generating grammar documentation.

In Caster we currently are working on the UI to display commands that are in grammars. This PR #219 was meant to format action classes for visualization.

The work that utilizes the above issue is to display commands via the GUI
dictation-toolbox/Caster#773 which is a work in progress.

The challenging bit is going to retrieve information that's hidden behind a Function or
Choice. For instance the BringMe commands above store keys in that contain the the spec and the data to return. Saying bring me google is represented by google = "https://www.google.com" for <website>. What are the options for website category? Another instance "(<capitalization> <spacing> | <capitalization> | <spacing>) [(bow|bowel)] <textnv> [brunt]": Function(textformat.master_format_text),

Looking at function def master_format_text(capitalization, spacing, textnv): what of the choice specs for capitalization and spacing?

Another thing we have in Caster is to include rdescript which allows us to embed a description of a command. This is great for when a spec is short and phonetically distinct not descriptive of its action.

        "dredge [<nnavi10>]":
            R(Key("alt:down, tab/20:%(nnavi10)d, alt:up"),
              rdescript="Core: switch to most recent Windows"),

When a command does not have a rdescript defined its spec is dynamically created as a rdescript. Thus displayed as Navigation: shackle

       "shackle":
            R(Key("home/5, s-end"), rspec="shackle"),

I think if dragonfly had its own version of rdescript that may be of help. That way if commands are self descriptive they can go without and if more description is needed include that within the grammar.

All in all the UI above is a wonderful leap forward. The same issues we experience working with displaying commands on the UI will be an issue for anyone implementing methods for generating documentation.

[drmfinlay]

Sorry for the late response!

Thanks for this very detailed post! The Caster command visualisation UI is very impressive. Given that you have already solved some issues you mentioned with this, it might be a good idea to see if I can get Sphinx building documentation from it somehow.

[daanzu]

I think if dragonfly had its own version of rdescript that may be of help. That way if commands are self descriptive they can go without and if more description is needed include that within the grammar.

I think this makes a lot of sense. May be as simple as making it an optional argument to ActionBase()?

Regarding how to handle documenting helper functions, have you considered using their docstrings to describe what they are doing, and then displaying that?

[LexiconCode]

Regarding how to handle documenting helper functions, have you considered using their docstrings to describe what they are doing, and then displaying that?

That's an interesting thought! In terms of space not sure how that would fit on the GUI depending on the doc string. Perhaps the command could unfold with the doc string.

A major point of clarification on my part regarding CCR. With CCR I believe the reason choice and maybe the function hard to get at is because these are evaluated before the grammar is merged and loaded into dragonfly. With casters CCR implementation it loads a 2 CCR merged grammars app and global. If we were able to pseudo-load these grammars before merger in a way that we get at choice and function from the engine's perspective.

[drmfinlay]

Documenting grammars or command modules can already be done with the Configuration Toolkit. Christo Butcher, the author of Dragonfly, wrote in his dragonfly-modules repository many command modules and .txt files using this toolkit.

The following is an example configuration for commands that type letters A, B, C and numbers 1, 2, 3.

from dragonfly import Config, Section, Item

config                     = Config("Configuration")
config.cmd                 = Section("Command section")
config.cmd.type_letter     = Item("<letter>", doc="Type the <letter> key.")
config.cmd.type_number     = Item("<letter>", doc="Type the <number> key.")

config.extras              = Section("Command extras section")

config.extras.letter       = Section("Extra letter section")
config.extras.letter.a     = Item("alpha", doc="Letter A.")
config.extras.letter.b     = Item("bravo", doc="Letter B.")
config.extras.letter.c     = Item("charlie", doc="Letter C.")

config.extras.number       = Section("Extra number section")
config.extras.number.one   = Item("one", doc="Number 1.")
config.extras.number.two   = Item("two", doc="Number 2.")
config.extras.number.three = Item("three", doc="Number 3.")

By default, invoking Config.generate_config_file() results in a neatly formatted text document like this one. Custom Config subclasses could be written to output information in different formats.

A Config hierarchy like the above could be generated from grammar, rule or element objects. Using the above example, Item objects for each letter could be generated from the relevant dict object:

letter_names = {
    "Alpha":     "a",
    "Bravo":     "b",
    "Charlie":   "c",
    "Delta":     "d",
    "Echo":      "e",
    "Foxtrot":   "f",
    # Et cetera.
}

for k, v in letter_names.items():
    item = Item(k, doc="Letter %s." % v.upper())
    # config.extras.letter.<v> = item
    setattr(config.extras.letter, v, item)

The only thing missing here, in my opinion, is a method for outputting code similar to that in my first code block above. With that added, one could do the following:

1. Create Config, Section and Item objects with data from a current grammar, using code similar to my second code block.
2. Generate clean configuration toolkit code and copy it back into the grammar file(s).
3. Replace strings in the grammar file(s) with items, e.g. "alpha": "a" becomes config.extras.letter.a: "a".
4. Load/generate regular config .txt files for command documentation and user customisation.

My apologies for not suggesting all this in the first place.

[drmfinlay]

It appears there is little interest in this issue. Adding new functionality to Dragonfly for this is unnecessary; the configuration toolkit serves this purpose well. It is really not difficult to use in command modules. There are also many examples demonstrating use of the toolkit.

What I'll do is add a new section to the configuration toolkit documentation on its utility for documenting grammars, fix some minor formatting problems with the output of Config.generate_config_file(), and then close this issue.