Add basic jsdoc tag types

[sandersn]

This PR adds basic support for the JSDoc tags @type, @param, @returns, @typedef and @template. The implementation is quite different from Strada, but Corsa will now have examples of all 3 major approaches we've planned for JS inference we've planned for JS inference in Corsa, meaning that the rest is just filling out support.

Functionality

@type applies to variable declarations and property declarations:

/** @type {number} */
const x = 1
class C {
  /** @type {1 | 2 | 3} */
  p = 1
}

It also works as a cast on parenthesised expressions:

let x = /** @type {12} */(12)

And it behaves as a cast on property assignments, export assignments, and return statements:

const o = {
  /** @type {number} */
  p: 2
}
/** @type {Level} */
export = { e: 1, m: 1 }
function f() {
  /** @type {1 | 2 | 3} */
  return 1
}

I ported this support from Strada, and it's useful for @param/@returns (see next), but it doesn't really make sense for @type. I might cut it, or perhaps add a real Type field to property assignment/export assignment/return statement; the checker currently acts as if they do because of porting from Strada.

@param and @returns apply to parameters and return types of functions. They work on anything function-like, including in lower positions:

/**
 * @param {string} s
 * @returns {never}
 */
function fail(s) { throw new Error(s) }
/**
 * @param {*} input
 * @returns {string}
 */
const log = input => input.toString()
class C {
  /** @returns {string} */
  f = () => ""
}
const o = {
  /** @returns {string} */
  f: () => ""
}
/** @returns {string} */
export = () => ""
function f() {
  /** @returns {string} */
  return () => ""
}

@typedef supports both single-line types and multi-line object types with @param. Nested object types are not yet supported. As in Strada, @typedefs are implicitly exported, but this PR is missing the fanciness to make that work right with global scripts or CommonJS modules. Currently, @typedef always turns a file into a module.

/** @typedef {'a' | 'b' | 'c'} ABC */
/** @typedef {Object} Composite
 * @property {number} one - some documentation
 * @property {string} two - more documentation
 */

@template tags work with @typedef, function-likes and classes.

/**
 * @template T
 * @typedef {Object} Box
 * @property {T} value
 */
/**
 * @template T
 * @param {T} x
 * @returns T
 */
function id(x) { 
}

Architecture

Corsa now uses 3 different approaches to support JS inference:

• expando assignments (eg f.p = expandable), including to class properties, are (or will be) handled in the binder, just like Strada. Unfortunately, this means that every BinaryExpression includes DeclarationBase.
• TS-equivalent type and modifier tags (eg @type @template @private) produce synthetic nodes during parsing. After a node is parsed and its JSDoc is parsed, missing fields on the node can be set to nodes from the tag instead.
• TS-equivalent statements (eg @typedef) produces synthetic statements, which the parser inserts right before the just-parsed node. These statements have their own node kinds and type to allow for JS-specific semantics, although this PR doesn't have any @typedef-specific semantics vs type aliases.
  Worth noting: I have no ability yet to support API users like TSDoc, who in Strada could request all tags for a variable declaration and get all tags on the variable declaration statement, for example. I expect eventually we will have to port the old tag lookup code just for uses like this.

Expando representation

For an expando assignment f.p = expandable, I could emit a synthetic jsnamespace f { jsexport var p = expandable }. This might be more efficient than treating BinaryExpression as a declaration in the binder. Class properties would need an equivalent jsprop p = expandable, and this declaration would either need to be lifted to the class body or retain special support in the binder/checker to allow (JS) property declarations to occur anywhere inside a class.

More information on the synthetic nodes:

Hosts

Corsa searches for a host similar to the way Strada does, but the search is (1) top-down (2) at parse-time instead of repeated each time on-demand. In other words, in Strada, hosts would search up the tree for applicable tags. In Corsa, tags search down the tree for applicable hosts. However, this makes it harder to track multiple hosts, so for this PR, the search stops at the first host, which is by far the most common case.
The bottom-up on-demand nature of Strada's tag-to-host attachment meant that if multiple hosts find a tag, the tag applies to multiple hosts. In Corsa, tags look for hosts, and they apply to the first host only.
This should be much more efficient. This means, at least currently, that each tag can only have one host.

Also note that 'host' generally means the entire signature in Strada, but is a specific parameter type annotation, for example, in Corsa.

Clones

Currently, only the top node of a hosted type is cloned into its new position. It's not a deep clone, so children of the type still retain their original positions and parents. That means when walking the tree top-down, the checker may suddenly be looking at JSDoc nodes that originated from a different place in the tree.

This is important because if the checker needs to walk back up the tree, the parent pointers have not been changed. To fix this, I had to track a type's host on JSDocTypeExpression:

JSDocTypeExpression

JSDocTypeExpression is the JSDoc node kind just above all the types that are cloned. Because of this, when there is a bottom-up lookup, JSDocTypeExpression is the signal that a type was cloned from JSDoc. JSDocTypeExpressions store a pointer to the host node of its type, and name resolution, for example, uses it to jump from a JSDoc type to its host.

Overall, a deep clone with parent pointer fixup might be better. But I looked at DeepClone and it doesn't change the parent or provide hooks to change it.

Implementation

• ClassLikeBase now has LocalsContainerBase because @typedef can be nested in a class body. Questionable.
• I simplified the representation of JSDocTemplateTag, which previously tracked its single constraint per-tag, but now copies the constraint down to each type parameter. Thinking about this, this probably breaks visits and needs to be undone.
• JSDocTypedefTag can now only have an Identifier as a name, not QualifiedName, so I simplified that in this PR too.
• I simplified other places that no longer need to check for param/return/type/template tags. A good example is name resolution, which is much simpler, although it now has to follow a JSDocTypeExpression's host pointer when walking up the tree.
• JSTypeAliasDeclaration is exempt from the grammar checks that apply to TypeAliasDeclaration. Otherwise, a lot of the diffs are from pasting JSTypeAliasDeclaration wherever TypeAliasDeclaration occurred.
• Statement parsing uses parseListIndex, not parseDelimitedList, so I only needed to change the former. After parsing, I check whether a new parser property reparseList contains elements and add them before the just-returned element if so. Then I empty reparseList.
• JSTypeAliasDeclaration emits as jstype T = U. I have no idea if that's right, but normal JSDoc emit should already emit the @typedef (and JSTypeAliasDeclaration should be erased by the type eraser.)
• I had to change the baseline writer to skip synthetic nodes, as far as I remember.

Limits and Future Work

• Host search stops at the first host. I need to figure out whether tags should even support multiple hosts, then figure out to make it work.
• Make host search recursive, if it makes sense to support that.
• Over 2000 baseline diffs still remaining.
• Find out to how permanently accept intentional diffs (or normalise them with Strada's baselines).
• CommonJS
• Class property expando assignments
• Limited constructor function support

[sandersn]

Unlike type, @typedef is allowed as a local inside classes.

I haven't checked to see how common this is so I added support for now as a way of fixing a nil panic. But I will remove it if it's not common.

[jakebailey]

Every time we have a KindTypeAliasDeclaration, we should have a KindJSTypeAliasDeclaration? How does that scale for getting things right long-term? Hopefully it's not too much worse than the old code...

[sandersn]

Yes, except for the places where we shouldn't. I looked at the current codebase and I'm confident that it's 95% right and that the tests catch the rest.

The real value of having two kinds is in commonjs, where module.exports= is going to need to allow a bunch of things that export= doesn't.
Still, it's ultimately a matter of whether jstype should work the same as type by default or not, not expressiveness, because you can always have isJS := node.Flags & FlagsReparsed != 0. That way jstype works like type by default.

[sandersn]

In other words, as an exercise for me, when developing, having two kinds is GREAT, because it forces me to go port every single TS construct to JS and consider the semantics. And if I forget something, the tests won't work.

However, long-term it might make more sense to have everything work the same by default.

[sandersn]

After discussing this in the design meeting: Anders prefers having two separate kinds and nobody else spoke up.

[jakebailey]

How likely are we to hit this bug elsehwere when working with positions on synthetic nodes? IIRC we would never have to do this previously, but now there's a reason to check for each error.

[sandersn]

The actual problem is that typeParameters could never be parsed at the very beginning of the file, but now they can:

/** @template T */
function f() { }

opposed to TS, which has no way to have a type parameter list that starts at position 0.

function f<T>() { }

[sandersn]

This might indicate that the synthetic type parameter list Pos should not be 0, though.