Use Erlang string case folding. Add word mappings to normalize

[mworrell]

This pull request introduces a new module, z_string_normalize, which provides comprehensive Unicode string normalization and transliteration to ASCII, supporting multiple languages and custom word mappings. It also adds a test to verify specific word normalization behavior. This module has been split from z_string.

In z_string the functions to_lower/1 and to_upper/1 now use string:foldcase/1 and string:uppercase/1 instead of their own mappings.

String normalization and transliteration:

• Added the new module z_string_normalize with a normalize/1 function that lowercases, sanitizes, and transliterates Unicode strings to ASCII, including support for Cyrillic, Ukrainian, Polish, Turkish, and Hebrew scripts. The normalization also handles HTML entities and various accented characters.
• Implemented a mechanism to load and cache custom word mappings from a CSV file using persistent_term, allowing for efficient and customizable normalization of specific words (e.g., language-specific transliterations).

Testing:

• Added a test normalize_map_words_test to ensure that the normalization correctly maps "Одесса" (in Cyrillic) to "odesa" using the custom word mapping.

[Copilot]

Pull Request Overview

This pull request introduces a new module for comprehensive Unicode string normalization and refactors the existing case conversion functions to use Erlang's built-in Unicode-aware string functions.

Key Changes:

• Created z_string_normalize module with normalize/1 function that performs lowercasing via string:casefold/1, sanitization, and transliteration to ASCII for multiple language scripts
• Refactored z_string:to_lower/1 and to_upper/1 to use string:casefold/1 and string:uppercase/1 respectively, replacing custom character-by-character conversion logic
• Added customizable word mapping system via CSV file that uses persistent_term for efficient lookups, enabling language-specific transliterations (e.g., Ukrainian city names)

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/z_string_normalize.erl	New module implementing Unicode normalization with transliteration rules for Cyrillic, Ukrainian, Polish, Turkish, and Hebrew scripts, plus custom word mapping support
src/z_string.erl	Simplified to_lower/1 and to_upper/1 to delegate to built-in Erlang string functions; updated normalize/1 to call new z_string_normalize module
test/z_string_test.erl	Added test case to verify word mapping functionality (Cyrillic "Одесса" → "odesa")
priv/normalize-words-mapping.csv	CSV file containing custom word mappings for city names in multiple languages
src/z_mochinum.erl	Minor test improvement: added explicit positive sign to floating-point zero literal for clarity

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.