Smarter behavior for 'input currently matches' in parse errors

[tedinski]

I remembered and wanted to write down this issue today.

Right now, whenever there is a syntax error in a Silver source file, we get this as part of the parse error:

   Input currently matches:
   [silver:extension:templating:syntax:QuoteWater,
    silver:extension:templating:syntax:SingleLineQuoteWater]

These terminals are pretty much [^"]+ regexes, so naturally maximal-munch prefers them, almost always, over anything else, despite these terminals only being valid in rare contexts. Nearly always, something more specific should be reported.

I'm not sure if there's a perfect solution to this problem. Possibly a simple heuristic would be to resort to using an "all terminals are valid" lex for error reporting ONLY if there are NO matches for the set of terminals valid in the current parser state. But this may not be enough?

I'm not sure if there's any sensible way to try to enlarge the scope of valid terminals without going all the way to recognizing everything.

[schwerdf]

The real challenge is how to restrict the scope of the union scans in a composable way.

One could manually assign some kind of "priority" to terminals and, when a syntax error occurs, run the union scan on each priority class in descending order until a match is found. It might even be possible to use smart means to assign a priority when the user does not provide one explicitly. This, however, would create composition issues if extension writers did not coordinate on a priority scheme.

Another possibility is to eliminate the use of maximal munch when running union scans -- collecting the longest match of each terminal instead of the longest match, period. This is composition-safe but might result in somewhat match lists.