Define scope of this project

[nzakas]

To aid in communicating with participants, it would be helpful to come up with a statement of scope so everyone knows what is and is not a part of ESTree. There has traditionally been some confusion around this:

• Do we want to define expectations around source locations?
• Do we want to incorporate a concrete syntax tree?
• Do we want to include common extensions like JSX?
• Do we want to include comment types?

From the response (or lack thereof) to these various issues, my guess is the answer to each is "no," but it would be nice to get some consensus and then narrowly define the scope of the project so we can more easily respond to issues and PRs as they come up.

[michaelficarra]

Do we want to define expectations around source locations?

I think this is reasonable. I agree with what @RReverser expressed in the linked thread, that we should have an agreement on what portion of source text is being represented by an AST node, regardless of how concrete syntax information is being conveyed.

Do we want to incorporate a concrete syntax tree?

There's many ways to convey concrete syntax information, and each has its benefits and drawbacks depending on the use case. I don't think there should be single blessed way to represent and pass around concrete syntax info. For most cases, I prefer a side table style approach, but I can also appreciate the convenience (and difficulties) of inlining concrete syntax info into the AST.

Do we want to include common extensions like JSX?

Please no. The format is extensible, let them extend it. The most I could see the estree project doing is coordinating reserved prefixes or something for type names to avoid extensions stepping on each others' toes.

Do we want to include comment types?

No, for the same reason we don't include concrete syntax information like whitespace, position, and unnecessary grouping. Comment representations are also varied and have the same tradeoffs I mentioned with concrete info above. I'd be interested to see either this project or a similar project try to standardise the various comment "attachment" rules, though, defining how comments associate with adjacent nodes. In our comment-based templating system, we ran in to some tricky cases and disambiguated them with optional node types included in the comment.

[RReverser]

I agree that answer is "no" in general to most of these questions.

If we ever have enough people and interest to tackle problem of CST again, we could create it as a separate project under the estree org and design it in a way compatible with ESTree.

However, I think it might make sense to follow the precedent of "extensions" like @nzakas added for base type annotation definitions here https://github.com/estree/estree/tree/master/extensions and perhaps define existing common non-standard extensions used by parsers - e.g. range or comments.

Not sure if it brings much value, but at least it would be documented somewhere as a point of reference for such optional extensions.

P.S.

I agree with what @RReverser expressed in the linked thread, that we should have an agreement on what portion of source text is being represented by an AST node, regardless of how concrete syntax information is being conveyed.

I forgot about these by now, but damn, that looks like a reasonable suggestion. Past me was smarter than today 😅

[nzakas]

Okay, so it seems like we have a general agreement that:

1. We want to define what portion of source text is being represented in the AST (define logic of range and loc)
2. We do not want to work on a CST in the context of this repo
3. We might define extensions based on their popularity or prevalence in the ecosystem, but this would more documenting what has been implemented rather than defining what should be implemented.
4. We do not want to formally define comment types.

Is that a good summary? @RReverser @JLHwung

[RReverser]

Sounds good. Just few clarifications:

1. define logic of range and loc

Just the general logic, not the actual properties, right? We can still define the properties under (3) though, just want to make sure we have a clear scope.

4. We do not want to formally define comment types.

I think these can similarly be merged under (3) - that is, like other popular extensions, we might document how they work without making them part of the official standard.

[michaelficarra]

Just the general logic, not the actual properties, right?

I came here to say the same thing. We should define what portion of source text corresponds to what AST node(s) in a parse, but not define how the ranges from a parse of a particular source text are represented, either within the AST or outside it.

[nzakas]

@RReverser @michaelficarra just to make sure I'm understanding about range and loc, you're saying that we should define the range that nodes represent, but not say what the values in range and loc should be?

[michaelficarra]

@nzakas Yes, this includes at least

• clarifying what syntax constructs are represented by each AST node
• clarifying what each property of each node means (the property names alone are insufficient)
• clarifying whether adjacent whitespace/comments/grouping/commas (anything not represented in the AST) are considered to be part of each node

For the first two, imagine the AST nodes were named "A", "B", "C", etc. and their properties were named "a", "b", "c", etc. What would you need to document to still be able to implement tooling based on the spec?

[RReverser]

Yeah, I think a good example here would be something like ExpressionStatement where ; is outside of what is directly covered by properties, but logically still covered by the node span.

Or loop statements where properties describe conditions and the body, but node covers keyword and punctuation as well.

It should be fairly simple / logical in most cases, but needs adding examples to each node.

[nzakas]

Okay, gotcha, thanks for explaining. I'll try to put together a scope statement on the README soon.