feat: AOT compilation with icu-to-json (experiment)

[amannn]

Fixes #962

(WIP)

This is an experiment to use icu-to-json for AOT compilation of messages on the server side / at build time and to reduce the bundle size in Client Components.

Current progress: For now, both compilation as well as evaluation happens in the same module (and therefore on the client side). This is mostly intended to evaluate initial compatibility. Later, we could move the compilation step to the server.

Questions:

• How to support t.raw or t.markup? We need to decide if a message needs to be parsed before it's used. Do we need to analyze the usages? Or could we revert the parsing when t.raw is called?

TODO

• Can this help with feat: Support ISO 8601 date strings with full precision for all formatting functions where dates can be passed #758 (comment)?

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
next-intl-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 11, 2023 9:04pm
next-intl-example-app-router	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 11, 2023 9:04pm

[amannn]

Looks good!

Currently skipped tests:

• Error handling (will be moved to a different place)
• Nested rich text

[amannn]

Needs to be addressed on the icu-to-json side I think.

[jantimon]

this "should" work 😄

you might have to define a tag and/or a baseTag formatter

[amannn]

Could these two properties be removed from the parsed result? Or should they be used by the consumer?

[jantimon]

Not sure what you mean - Formatters are called here with (value, locale, formatOptions)

https://github.com/jantimon/icu-to-json/blob/83f24322bfd4ab66402e70d9727a650720501925/src/runtime.ts#L150-L152

the format options are passed through from MessageFormat

[amannn]

I mean e.g. these cases:

// [["price",4,"numberFmt",{"type":0,"tokens":[{"stem":"currency","options":["EUR"]}],"parsedOptions":{"style":"currency","currency":"EUR"}}]]
'{price, number, ::currency/EUR}'

// [["value",4,"numberFmt",{"type":0,"tokens":[{"stem":".#","options":[]}],"parsedOptions":{"maximumFractionDigits":1}}]]
'{value, number, ::.#}'

Is tokens necessary here? In regard to minimizing the size of the AST, could this be simplified to something like this?

[["value",4,"numberFmt",{"maximumFractionDigits":1}]]

Similarly, date skeletons seem to be a bit verbose:

// [["now",4,"date",{"type":1,"pattern":"yyyyMdHm","parsedOptions":{"year":"numeric","month":"numeric","day":"numeric","hourCycle":"h23","hour":"numeric","minute":"numeric"}}]]
'{now, date, ::yyyyMdHm}'

Could the parsedOptions be unwrapped and replace the object they're placed within?

[amannn]

Could this be removed from the parsed result? See also https://github.com/amannn/next-intl/pull/705/files#r1418720864

[jantimon]

use compileToJson instead it's the same as compile just without args

args are helpful to generate types or to optimize which formatters should be included to a bundle.
e.g. if date is used you might want to add a date formatter

[amannn]

Got it, thanks!

[amannn]

Should I pass the values from the user or args from the parsed result here?

[jantimon]

the args are not needed for formatters however the args can tell you during compile time which formatters need to be available to run the code.

in your case there is a very special formatter wich you should add for full react support

it allows processing the children before they are passed to the given tag function:

{
   tag: (children: Array<string | ReactNode>, locale: string) => 
         children.map((value, i) => typeof value === "string"
            ? value 
            : <Fragment key={`f-${i}`}>{value}</Fragment>
}

e.g. for a translation like <foo>Hello <b>{name}</b></foo>

[jantimon]

there is also the baseTag formatter which is a fallback which kicks in if the end-user did not define a tag.

by default the tag is converted to a string but you might also wrap in a fragment or even allow some tags to be used as html:

// for a compiled message "<b>Hello {name}</b>" 
const formatters = {
  baseTag: (Tag, chilren) => {
      // allowlist:
      if (["b", "strong", "p"].include(Tag)) { 
        return <Tag>{children}</Tag>
      }
      return <>{children}</>;
  }
};
run(message, lang, { name: "Joe" }, formatters)

[amannn]

I tried updating your example here to use nested rich text: https://codesandbox.io/p/sandbox/icu-to-json-demo-forked-2jzngr?file=%2Fsrc%2Ficu.tsx%3A19%2C1-20%2C1

Somehow the inner part is missing. Am I doing something wrong?

[jantimon]

there are two different ways to run the precompiled icu messages:

1. run
2. evaluateAst

the idea is that run will always return a string and evaluateAst will always return an array.

const t = (key, args) => run(messages[key], lang, args);

icu-to-json is framework independent so it would require you to wrap a react Fragment around the result:

t.rich = (key, args) => <>{evaluateAst(messages[key], lang, args)}</>;

[amannn]

Oh, that helps with the fragment—thanks!

I was wondering about this case:

It seems like the args determine the return value. However, the function b has already been called at this point (having returned a React.JSX.Element) and date should be turned into a string.

Therefore the returned value should have the type (string | React.JSX.Element)[], right?

Or am I missing something?