RFC: openapi-typescript v7 built on top of Redocly

[drwpow]

Proposal

Over the course of v6 (3.1 support), this library has had to solve for quite a few challenges when it comes to complex schemas. Especially with the increased usage, more bugs have been uncovered. Which is great for stability, but it’s taken on a maintenance impact.

If I had to characterize the biggest challenge facing this library, it’s divided concerns. At its core, this library should be focused on converting an OpenAPI document to TypeScript. However, an unexpected challenge has been dealing with complex schema compositions (resolving $refs) and a lot of time, energy, and bugfixes have gone into that. If the latter were able to be offloaded to a more reliable solution, it would be a huge win in stability and maintenance for this library.

I’ve been using the Redocly CLI in other projects for over a year. And for almost as long, I‘ve been recommending the Redocly CLI in this library’s docs as the best validator for OpenAPI schemas. It’s an amazing OpenAPI toolkit for any OpenAPI spec, and it recently released its 1.0 stable release in August 2023. It does all of the following, but is not limited to:

• Validation
• Linting (e.g. enforce your schema is formatted according to your org’s standards)
• Bundling / schema resolution
• (and of course) docs generation

While people may be familiar with Redocly’s doc generation (“doc” is in the name, after all), I’m more interested in the validation and bundling capabilities of the CLI. Validation has been a sticking point of this library as openapi-typescript requires valid schemas but does not validate itself. This leads to many issues being opened reporting a bug when really it’s a malformed schema. Similarly, bundling would solve many resolution bugs from this library’s custom resolvers that Redocly CLI frankly handles better.

Pros

• 🔍 Validation errors! 🎉 This would result in a better UX for everyone
• ✈️ Redocly’s bundler would solve resolution problems with remote schemas
• 🤝 Alignment in project goals: Redocly is also purely Node.js, and is lightweight, well-built, and performant
• 🐛 Offloading resolution to the Redocly bundler would also reduce a ton of code in the codebase, resulting in better reliability, easier maintenance, and reducing bug surface area
  • This would also result in more reliable discriminator support when mapping is complex / nested
  • This would also fix a lot of historic composition bugs* (see below) where oneOf/allOf/anyOf are deeply-nested and reach across paths and components. Though many have been fixed, they’ve been fixed at the cost of codebase complexity and after being an open issue for a while. There are likely more bugs unreported.
• 🚀 No foreseeable impact on performance (probably won’t be faster, but shouldn’t slow it down, either)
• 🧹 For existing users of Redocly CLI, config could be simplified

Cons

• This would incur the weight of @redocly/openapi-core (which is pretty lightweight, but just pointing out)
• Breaking changes for auth for remote schemas. The current options for auth would be replaced with Redocly’s auth config. Though most common usecases should be handled, minus some config change annoyance.
• Minor breaking changes to types (e.g. the overall structure wouldn’t change, but the external interface will likely go away, along with some other minor changes)

Additional Info

Alternatives

• The alternative to continue without Redocly will likely result in slower-to-fix bugs from a larger codebase. It not only poses a barrier to new contributors, but for existing maintainers, we’ve reached another tipping point in complexity where removing code would benefit this project and all maintainers.
• A related approach to build our own bundler is something I’m against; why create a worse version of something that exists already?
• I am strongly against using the Swagger CLI as this library exists to reduce the complexity / frustration with consuming official Swagger tools. Plus, Swagger’s validation leaves much to be desired (it ignores most new 3.x features as well as ignores many errors it should catch) when compared to Redocly’s fantastic validator.
• I’m not aware of another bundling solution (please suggest one if you know one / describe how it may differ from Redocly if you’re able to). While validation is a “nice-to-have,” bundling is a requirement for this change.

*Why use the Redocly bundler?

There are quite a few historic bugs with composition and complex references that have been a burden on this library since its inception. They all revolve around resolution and interpreting OpenAPI’s flexible, loose format into explicit TypeScript types.

In between the OpenAPI source schema and the final TypeScript types lies a third, undocumented “intermediary” format (that mostly exists in ctx). While some may think of this as mere “context” needed to generate types, it really is more complex than just arbitrary metadata, and involves building quite a few ad-hoc structures through deep scanning that become its own format, in a sense. This library tries to do as little as possible in building this intermediary layer and instead rely on predictable TS references. In simple cases, we can bypass this layer by doing simple string conversion (e.g. $ref: "#/components/schemas/foo" → components["schemas"]["foo"]). This is not only performant, but also reduces bugs by relying on TypeScript to resolve things for us.

However, for complex examples, we need this intermediary context layer, which is built upon hacky implementations such as deep parameter and discriminator scanning and hinting (which was a mistake). For example, when using discriminator.propertyName + oneOf, its child oneOf objects need to have a propertyName key added when there is no hint on the object itself it should be there. So we can’t simply deal with that when we come to it; we have to build the context mapping ahead-of-time on schema load. Every time we add another pass of deep-scanning, it’s slow at best, and buggy at worst.

Bundling flattens this and makes deep-scanning not only cheaper but more predictable, as we only have a single schema with a predictable structure to deal with and not myriad complex, nested, partial schemas that are either linked through brittle string references or hacky ad-hoc context objects. Redocly also does a ton of work in handling conflicts, normalizing values, and simplifying the structure to make this task as easy as it can be.

---

🙏 Welcoming any thoughts or strong opinions on this topic

[drwpow]

Update on this: started writing some prototype code this weekend consuming @redocly/openapi-core and already I’m blown away how good it is. This is still an RFC, after all, and I’d love to source opinions. But am starting to really like this direction—it seems like a huge net win. 🎉

[drwpow]

I’m also experimenting with an AST approach suggested by @rxliuli. It‘s too early to make the call, but trying to use TypeScript’s official AST has been frustrating (not my first time using it, but first time in a while, and it hasn’t gotten better).

However, ast-types/recast is extremely promising. If we’re able to wipe out syntax bugs without introducing an incomprehensible abstraction over the code, I might make the switch over the string-mashing this library’s always done.

[drwpow]

Update on AST transformation: ast-types is great, unfortunately, comments are throwing a wrench into everything. Comments are not very easy to manage at all in the esprima parser syntax (which ast-types is built for), so any cleanliness it is bringing is quickly flying out the window having to build an entire abstraction layer just for JSDoc comments.

Further, digging into the AST land, it’s pretty common for things to get out of date quickly. Even TypeScript AST helpers like tsutils are 3 years out-of-date and don’t support any TypeScript version newer than 2.8. It’s looking like if we want to use an AST, the only reliable thing would be the official TS compiler itself. Coming back around to it a 2nd time it’s not as bad as I originally thought.

[drwpow]

Update 3 on AST transformation: I was just “holding it wrong.” The native TS compiler is actually pretty great (or at least, is today). Comments are a breeze 🎉. This is cleaner & safer than string mashing, and it’s looking promising that v7 will drop string mashing in favor of AST compilation.

I feel better about using the native TS compiler because it has the highest guarantees of accuracy, maintenance, and performance improvements over time compared to alternatives.

My initial hesitation, again, was never “can the maintainers accomplish it.” It was, instead, “can the average developer understand it.” Which, I think it meets that goal. While writing a new AST from scratch is still a struggle (and probably always will be), editing existing ones is easier and more straightforward than I anticipated. The code is also much cleaner.

Speed-wise it’s too early to tell, but the actual building of the AST is fast—roughly the same as string mashing. However, using an AST introduces a new step we could skip before: compilation (string mashing is compiled as-you-go so there is no final step). TS’s compilation of the AST into a final document does add a noticeable lag. So the AST route will probably slow down this library a bit. But for the sake of accuracy and maintenance.

Further, I’m positive there are ways to pay down any potential performance losses (again, too early to call it a loss). One key example is parallelism—by leaning into an AST, we could parallelize it if we wanted to, since we couldn’t before (though TBD if that buys us anything).

[drwpow]

Last note on AST transformation vs string mashing: it’s not significantly slower! It’s a little slower, but most people probably won’t notice.

Unexpectedly, the biggest performance gain was Redocly’s bundling. By flattening the schema into one document, it improved multi-schema performance by several orders of magnitude. We used to have to skip the DigitalOcean test in CI, but no longer 🚀. Now it doesn’t take any longer than a single-file schema.

So again, overall basically a wash with AST, but v7 is a surprise performance speedup because of adding Redocly.

[adamaltman]

This has my vote. (I'm the founder of Redocly and definitely biased and I appreciate your kind words above.)

[drwpow]

v7 is now in beta! And baking Redocly’s CLI core into the project was even easier than expected! Would love to know if you have any thoughts: #1368