Internationalization in Bevy

[TimJentzsch]

This discussion aims to collect everything we have to do to provide high-quality internationalization in Bevy and other ideas and thoughts around the topic.
If you think anything else is important, let me know and I'll add it to the list :)

Internationalization and Localization

First, let's clarify some terms.

Internationalization (i18n) is the process of designing software in a way that it can easily be adapted to other languages and regions without bigger changes to the software architecture and code.
This is what we have to provide as engine developers.

Localization (L10n) is the process of using the features provided by internationalization to adapt the software to a region or language, e.g. by translating texts and adapting assets.
This is what the game developers/publishers have to do.

Note that i18n and L10n are common "numeronyms" for these terms, the numbers signify the number of letters that are left out in the middle.

Motivation

As we are going to see below, a substantial effort has to be put into providing proper internationalization. So why should we go through this effort?

The first idea is the one of universal design, making a product accessible to all people, no matter their region, language, disabilities or other factors. Only providing a game in English severely limits that goal: Mandarin Chinese alone has 929 million native speakers alone and 1.118 billion native speakers. Arabic (and its varieties) has another 450 million native speakers. The latter is one big example of a language that is not properly supported by Bevy right now.

If you are developing a commercial product, this also plays into the financial considerations. Localization can open up your game to entirely new markets and potentially big profits. Not having the opportunity to do so may lead you to choose a different game engine for your app.

Localization Scope

Okay, so let's take a look at what you might want to localize, this will give us a better grasp on what we need to do for internationalization.
Some examples include:

• Translating any text of your game to a different language
  • Simple strings in your UI
  • Changing formatting of numbers, dates, etc. for a different region
  • Adjusting time-synced text such as captions and subtitles
• Localizing other assets such as images/textures, sounds (e.g. in-game dialogue) and videos (e.g. cut-scenes).

This might sound easy, but due to big cultural and linguistic differences it presents many complex challenges for us to solve.

Internationalization Scope

Now that we know what we the use-cases are, we can take a look at what we need to support in Bevy (mostly in bevy_ui) to provide the proper support for game developers.

Writing Modes

One big challenge are different writing modes. In Latin scripts (e.g. English), we write lines from left to right and add new lines from top to bottom. However, this is not the case for all scripts: In Arabic scripts, the text is written from right to left; Japanese can be written with lines from top to bottom and new lines going from right to left; Mongolian scripts also go from top to bottom, but new lines are added from left to right.

These script types should not only influence how the text is rendered, but also the layout of the UI. The text direction and writing modes strongly affect how users parse content, so while a native English speaker typically looks at UI from left to right, and Arabic user will do so from right to left. Therefore, the UI layout should also be "mirrored".

CSS Flexbox has support for this type of layouting, with the writing-mode, direction and text-orientation properties.
Bevy also uses Flexbox via taffy, but doesn't have to support for writing modes yet.

Relevant Issues/PRs/Projects

• Writing mode support: Support direction property (right-to-left layout) DioxusLabs/taffy#213

Resources

• Languages using right-to-left scripts: https://www.w3.org/International/questions/qa-scripts.en
• CSS Writing Modes Specification: https://drafts.csswg.org/css-writing-modes/

Complex Text Layout

Complex text layout (CTL) is typesetting where the shape or positioning of characters is dependent on others:

This example (borrowed from Wikipedia, licensed CC BY-SA 3.0) shows the word العربية al-arabiyyah, "the Arabic [language]" in Arabic, in multiple stages of CTL. The first one shows the letters one-by-one, rendered from left to right. This is how it might look in Bevy right now. The second stage applies bidirectional text rendering, so now the letters are rendered from right to left. The third stage applies glyph-shaping to combine the characters according to their context.

Relevant Issues/PRs/Projects

• Bidirectional text rendering: Simple bidi integration #4614 and Bidi as components #4672

Font Support

Not all fonts support all characters that are necessary for a given language. The user needs to be able to easily change the font to properly render the languages they need.

To my knowledge, Bevy already has sufficient support for this.

However, we should also consider to make it possible to provide font fallbacks if a glyph is not available, for example via a system font. font-kit might provide such functionality.

Localization Tools

We need a good ecosystem for developers to localize their game ergonomically. Due to grammatical difference and region-specific formatting, simple key-value text substitution is not sufficient here.

First and foremost, we need proper locale management, i.e. a way to store and change the current language and region of the user. Support for the system language would also be nice here.
We should also support some system for fallbacks, often translations can be incomplete, so another language could be substituted (e.g. American English for British English).

Then we need support to localize any assets, to be able to swap out e.g. textures and audio based on the active locale.

Furthermore, text strings in the UI deserve proper attention, as this might be the most common use-case for localization.

Relevant Issues/PRs/Projects

• bevy_fluent: Support for text localization via Project Fluent
• bevy_mod_localization (WIP): Support for locale management and text localization via Project Fluent; planned support for asset localization and time-synced subtitles and captions via WebVTT

Keyboard Layouts

Many different keyboard layouts are available around the world. For example, controls like W/A/S/D don't make sense on an AZERTY keyboard as they are used in France.

Here, developers need to be able to easily change keybindings based on the locale.
Another nice thing would be to support keybinds that depend on the position on the keyboard instead of the character that is produced (this is for example supported in Dota 2).

Closing Thoughts

I most likely missed a lot of things that are equally important as what I have mentioned here. If you know of any, please leave a comment on this discussion and I will add it to the list!

Let's make Bevy a shining example of proper internationalization, which might be a good selling point in the future.

[wilk10]

Thanks a lot for the write up!

I totally agree that internazionalisation is a very important topic and I really do hope that it will get first class support in Bevy.
I could honestly see this topic as a top 5 arguments when evaluating Bevy for commercial projects, because, simply put, it directly affects the bottom line.
Being able to localise a complete (or near-complete) game to key languages and increasing the target market size by two or three times that of English speakers, is a must-have in a modern engine. Being able to do that easily and without much refactoring, would be a very strong selling point when choosing an engine.
It’s also a topic that applies across genres, and goes beyond games.

I definitely agree with the requirements listed already. I would like to add a couple of wishes, that might be a tiny bit more advanced, but would really make a difference, in my opinion:

• Out of the box support for conditional formatting within a localised sentence. Very often in games numbers, personal nouns or verbs are formatted differently. When those sentences are localised, these elements can be found in different positions, or split up, or declinated, etc. As a user, I would love to be able to define the formatting of a sentence in English (or the “core” language) together with its text, and once the locale changes have the engine automatically keep the defined formatting (without having to manually slice a string etc) in the new locale.
• Built-in tool for managing localisation files (CLI maybe?). This can still be a pain point with current more featureful engines, and it might hinder the localisation efforts. A simple to use tool that checks the files of the core language during development, groups them so that the developer knows exactly what still needs to be localised, or what has changed etc, and once the localisation files are updated, the tool makes it easy to update the existing files, would be awesome. I understand it feel very advanced, but it would give Bevy some competitive advantage in this area.

[TimJentzsch]

Regarding your wishes:

• The conditional formatting would be provided by Project Fluent, if we choose to integrate this (otherwise it can be done via a crate). They provide region-specific date, time and number formatting as well as support for custom formatters and proper pluralization. The harder part is to properly integrate it into Bevy. bevy_fluent already does this, but IMO it's not very ergonomic right now. You have to write quite a lot of boilerplate code and deal with the raw asset handles. With bevy_mod_localization I started experimenting myself, for example with just inserting components to localize a text entity. But there are still a couple of open questions, e.g. how to properly do the component approach with components/resources as inserted arguments and how to provide custom formatters. I'm still trying to prototype solutions on that front.
• I agree that the CLI tool would be great. Questions like "Which localization keys are missing in which languages?" should be easy to answer. We also need to consider that the people who do the localizations might not have any dev experience and might be completely external. Crowd-sourced localization is also becoming more popular (to my knowledge). I wonder if it would be possible to facilitate the integration with external services, but I don't know enough about that to provide any insights.

[wilk10]

Thanks for the answer!

• I feel I havent’t expressed properly my first point, I don’t mean only date formatting etc, but for example being able to have the number in a sentence of a different colour, or a character name with different text size or a verb in bold etc. Once localised, those will be in a different position, or declinated etc. As far as I understand Fluent receives a key and arguments and returns a string, but does track those variables internally. I guess Bevy would need some higher level API to handle tracking the position of those variables for the user, and applying the style once the sentence is localised via Fluent. The alternative is for the user to have to look for the value of the variable in the returned string. With numbers it's easy, but it gets messy very fast with nouns, verbs, adjectives etc. As far as I understand this is not supported today, but if it is, I'd love to look into it. By the way, great work with bevy_mod_localisation!
• I agree the CLI would be more of a tool for the dev team than the localisation teams, where the devs would still have to do some manual work managing the files with the localisation teams. Having a UI for that would be absolutely fantastic. Having some integration with an existing tool would be great too, but maybe the CLI is something achivable in the mid term? I do see how this is more of an advanced need though.

[Tsudico]

My current thoughts on conditional formatting would be to use a specialized syntax (probably Markdown) inside Fluent messages with special tags (Github Formatted Markdown allows for custom HTML tags) for marking styling sections. When the localized Fluent message is returned, that would be passed to another parser, like markdown-rs, to create the necessary AST that can be used to style the translation.

[igorhoogerwoord]

Font Fallback

We're building a project where any user can name objects in any language (chinese, english, arabic ...). We stumbled on an issue where font fallback is not yet supported, leading us to load 1 single large unicode font of over ~20mb.

Font fallback is important to seamlessly support multiple languages in a single block of text.
I want to chip in with helping develop and/or test font fallback for Rust Bevy.

Writing this here to find if other people are interested in this and whether we can work together.

[khassar00]

I think this should have a higher priority, at least than the editor. More languages mean more players. As a player, I would love to be able to dynamically modify localized text while the game is running, which is very good for translators, even if the developer can't afford to support more languages because of the cost, so the player can translate for themselves.