feat: use oniguruma-parser/optimizer for all grammars

[slevithan]

Closes #125.

Introduces a fancy Oniguruma regex optimizer that not only minifies more robustly (e.g., C++ drops more than 35,000 characters compared to the existing minification), but also improves performance of some regexes (for both the Oniguruma and JS engines in Shiki). C++ is probably the biggest beneficiary of these performance improvements because of some of the monster regexes it includes that are able to be significantly improved by the optimizer.

Changes in the PR:

• Updated scripts/grammars/cleanup.ts.
  • Removed compressRegExp.
  • Added oniguruma-parser/optimizer, behind a try/catch.
  • Added injections to traversal of grammars, to match a similar update @antfu made in Shiki that seems like it's also appropriate here (see shikijs/shiki@3722d2f).
• Ran pnpm run fetch:force.
• Deleted packages/tm-grammars/grammar-syntaxes.md, which was outdated.

Also, it seems like the the previous minification was inappropriately modifying some regexes. E.g., with the ActionScript 3 grammar...

Raw version:

"class_declaration": {
      "begin": "(?x) \\b(class)\\b \\s+ ([A-Za-z0-9_\\$\\.]+|\\*)",

Old minified version:

"class_declaration": {
      "begin": "\\b(class)\\b\\s+([\\.\\w]+|\\*)",

With the new optimizer:

"class_declaration": {
      "begin": "\\b(class)\\b\\s+([A-Za-z0-9_$.]+|\\*)",

The old version changed [A-Za-z0-9_$] to \w, which is inappropriate since Oniguruma's \w is Unicode-based by default and doesn't include $. Or maybe this was legacy processing that has already been removed but the grammar hasn't since been force-updated?

Regardless, the old version prevents correctly highlighting class names that include "$". So this PR will also fix highlighting bugs like this that were the result of prior, incorrect minification.

Aside: Oniguruma's \w is equivalent to [\p{L}\p{M}\p{N}\p{Pc}] (different than POSIX \p{Word}/[:word:], which are equivalent to [\p{Alpha}\p{M}\p{Nd}\p{Pc}]!). If the new optimizer sees that first set of Unicode categories in a character class, it indeed collapses them to \w.

[netlify]

❌ Deploy Preview for textmate-grammars-themes failed.

Name	Link
🔨 Latest commit	51d7a8d
🔍 Latest deploy log	https://app.netlify.com/sites/textmate-grammars-themes/deploys/67cd160454644100080b0c63

[slevithan]

(I'm assuming the deploy is failing for the same reason that #132 fixes.)

Ideally, textmate-grammars-themes would force-update all grammars whenever the oniguruma-parser dependency updates, so that grammars are reoptimized with any improvements/fixes. If you think that would be appropriate, I'm assuming the easiest solution would be to change the package.json bump script to add && pnpm run fetch:force. But maybe there's a better way to do it that limits it to running only when oniguruma-parser updates, rather than prior to every release? Or maybe you'd prefer to avoid that and only forcibly update all grammars on demand. But if so, I'd recommend doing it every now and then, if there have been updates to oniguruma-parser.