More advanced comment/documentation DSL, documentation generation

[homonoidian]

First of all, I think all the hard work should be done in Novika or Crystal, not in JS as is currently the case. The json-docs.nk script simply generates a shit ton of { "name": "...", "desc": "..." } objects, which then get processed in several steps by JS. The processing extracts stack argument names etc.

There are three big problems with all of this.

1. The current algorithm for extracting names is not perfect. It yields some false positives which mess up the docs. That said, it already is pretty complex (for what it does), and even uses some NLP stuff, for instance favoring nouns over other parts of speech! We also shouldn't forget it's in Markdown, so we first have to parse Markdown and only then do the NLP and argument search stuff. I think the hardest work in this sense should be shifted to Crystal code and e.g. some comment:... words in the comment capability, enabled by default (for instance comment:arguments could leave arguments [ [ '<short-name>' '<long-name>' ] [ ... ] [ ... ] ])
2. There is a big problem with leaving result blocks in Novika. Imagine x y newPoint word that leaves a point object for the X and Y coordinate. Even if you write the docs on the block newPoint leaves, these docs are only going to be visible at runtime, not at doc collection time.
3. This whole machinery is barely portable to Novika projects. Ideally we should have something like crystal docs.

Now to outline the solutions:

• I think a scoring system would be better, with the score being computed from the number of occurrences in docstring AND the noun-ness, adverb-ness and so on. Whether this is possible in Novika I do not know, perhaps there is a C library that can do something like this which we can have bindings to.

2. I already started (conventionally so to speak) to use a system where ::markName:: (::point:: in our newPoint example) creates a "link". Think of it like holes in the code linked by wires. Saying that the effect of newPoint is ( X Y -- ::point:: ) means that it leaves the point block, i.e. the block whose comment is ::point:: -- and nothing else besides that, at least for now. Let's not over-complicate things. Each word interested in being documented should also say the following in its comment: ::point/<name-of-word>::. The UI should then show a link or an expansion of sorts with documentation for the block. See the code snippet at the very bottom for an example.
3. I don't think generating outright HTML is the best choice, rather, I personally think a pluggable JSON payload is more favorable. This JSON payload is then read by (or embedded in) a script in the browser (or loaded as a resource), and the UI is built appropriately. Again, most of the hard work should be done outside of JS. The payload should contain enough information to build the UI. The payload could be generated for the current project with something like novika docs, docs being an app found in env

Finally, the UI should be redesigned. It's too plastic-looking.

[ "( X Y -- ::point:: )"
  ${ x y }

  orphan dup extend: [ "::point::"
     [ """( ::point P:: -- ::point C:: ): fooizes Point.

       ::point/foo::
       """
       drop this
     ] @: foo

    [ """( ::point A:: ::point B:: -- ::point C:: ): barizes Point.

      ::point/bar::
      """
      2drop this
    ] @: bar
  ]
] @: newPoint