[CI4] Language fallback improvements

[BudsieBuds]

Following best practices for language fallback in CI4, I've made the following changes.

• Changed en-US to en for fallback purposes. See CI4 docs.
• All other language variants will fall back to English. This is the only language that is consistently and officially updated.
• Changed some names and codes, following the BCP47 recommended by CI4.
• Norwegian and Urdu languages are working again.
• Sort languages by alphabet in config for easier navigation.

NOTE: Language fallback to English is currently broken in OSPOS because of empty strings, see #3994.

[jekkos]

Great work sorting this out @BudsieBuds . The issue in german with empty strings is not resolved then by renaming the en-US to en?
I was going through the CI4 code and indeed, if the language line does not yield any content, it falls back to english.

[jekkos]

how do we decide whether we use a base language folder like de or es? In my local version I see this as a login screen when selecting german. Also CI4 docs seem to suggest that one should use a base language folder, and then only translate strings that deviate from this in the more specific variant. The idea sounds good as it means less maintenance, if a translation is the same you do not need to duplicate and maintain it.

But I'm not sure whether weblate would be able to handle this setup.

[BudsieBuds]

Great work sorting this out @BudsieBuds . The issue in german with empty strings is not resolved then by renaming the en-US to en? I was going through the CI4 code and indeed, if the language line does not yield any content, it falls back to english.

That's what I thought, but unfortunately no. It is resolved by removing the keys though. It's also an option to remove all empty keys during build or something, but I think coding in that empty values should be ignored would be better and less prone to error.

how do we decide whether we use a base language folder like de or es? In my local version I see this as a login screen when selecting german. Also CI4 docs seem to suggest that one should use a base language folder, and then only translate strings that deviate from this in the more specific variant. The idea sounds good as it means less maintenance, if a translation is the same you do not need to duplicate and maintain it.

But I'm not sure whether weblate would be able to handle this setup.

I thought about this before this PR and think it's better to have everything fall back to English. Mainly because that's the only one being consistently updated, like I mentioned. I think the fact that this is a POS means that seeing the wrong translation (f.e. from de when you have selected de-CH), could really confuse the end user. I don't know the differences between de-DE and de-CH for example, but they could be big. Especially with more technical terms for taxes etc. A better example would be nl-NL and nl-BE, mainly because those are our native languages, and I noticed pretty big differences. How it was set up, everything that isn't translated in nl-BE would fall back to nl and then to en (if renamed and empty string ignored).

The only way I see this being an option is when Weblate can handle showing what it falls back to, in what order, and with multiple languages. But from what I could find in the docs this isn't possible with PHP strings.

[jekkos]

@BudsieBuds one last question, why do we convert de to de-DE, while for hu-HU we convert it to hu ?

Would like to understand the rationale behind either going for language name with or without localization suffix (but then excluding english)

[BudsieBuds]

Ask away. I'm not the best at describing my PRs.

Among other resources, I've referred to this file in the repo. I removed all ISO 3166-1 region codes from languages that are only mentioned once. However, I didn’t do the opposite. For instance, with French, the variant is unknown (fr-BE, fr-CA, fr-CH, fr-FR, fr-LU, fr-MC). It's likely fr-FR, but that only really matters when another region is added.

Therefore, I changed de to de-DE because there are two different German translations available; de-DE and de-CH. If de isn't renamed, de-CH will fall back to de before en, which could lead to confusion. It goes the other way around for hu-HU, there's only one Hungarian translation available and no mention of another region in the aforementioned list. Following this logic, I kept pt-BR as is for example.

There's only one assumption: changing es to es-ES assuming it's Spanish (Spain).

Following the BCP 47 as recommended by CI4, I've checked the official resources for the shortest ISO 639 (language) code and official English name and the ISO 3166-1 (region) code. Adjusted them accordingly.

Yes, I went down the rabbit hole. 👀

[jekkos]

thanks for the clarification and research @BudsieBuds . Looks all fine to me.

[BudsieBuds]

@jekkos Should a database migration script be written for updating the languages on upgrade?

[jekkos]

selected language might indeed be in the Config table. So indeed this config value needs to be migrated.