platform(actions): create automated action to generate emoji.json inline with unicode.org

[wax911]

AniTrend Issue Guidelines

Before opening a new issue, please take a moment to review our community
guidelines to make the
contribution process easy and effective for everyone involved.

You may find an answer in already closed issues:
https://github.com/AniTrend/android-emojify/issues?q=is%3Aissue+is%3Aclosed

Feature Information

https://home.unicode.org/ often provides new Unicode versions, see: https://unicode.org/Public/emoji/ where we can find all the Unicode versions released. Ideally, we’d love to generate a emoji.json file based on the latest Unicode standard and make an automated PR for it.

Solution Information

1. Fetch Unicode Emoji Test Data:

   • The script retrieves data from https://unicode.org/Public/emoji/latest/emoji-test.txt.
   • Extracts the latest Unicode Emoji version from the file's header.

2. Fetch Emojibase Data:

   • The script constructs the URL https://cdn.jsdelivr.net/npm/emojibase-data@<latest_version>/en/data.json.
   • Downloads and parses the Emojibase data to create a mapping of emoji characters to their shortcodes (aliases).

3. Process Emoji Data:

   • Extracts relevant details from the Unicode Emoji test file, such as description, status, version, and Unicode representations.
   • Maps aliases from the Emojibase data to each emoji. If no aliases are found, they are generated from the emoji’s description.
   • Filter out non fully-qualified emoji formats

4. Save to File:

   • Outputs the processed emoji data as a JSON file (emoji.json).

Proposed GitHub Workflow

To automate the process of generating and submitting a PR for emoji.json:

1. Schedule Workflow:

   • Configure a GitHub Actions workflow to run weekly or on-demand.

2. Fetch and Compare Emoji Versions:

   • Use the script to fetch the latest Unicode Emoji version.
   • Compare it with the currently processed version in the repository.

3. Generate Updated emoji.json:

   • If a new version is available, run the script to generate an updated emoji.json.

4. Automate PR Creation:

   • Commit the updated emoji.json.
   • Push changes to a new branch and open a PR with a description of the changes.

5. Notify Reviewers:

   • Tag maintainers or reviewers to ensure the PR is reviewed and merged.

Example GitHub Action

name: Update Emoji JSON

on:
  schedule:
    - cron: "0 0 * * 0"  # Run weekly
  workflow_dispatch:

jobs:
  update-emoji-json:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout Repository
      uses: actions/checkout@v3

    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.9"

    - name: Install Dependencies
      run: |
        python -m pip install --upgrade pip
        pip install requests

    - name: Run Emoji Update Script
      run: python app.py

    - name: Commit Changes
      run: |
        git config user.name "github-actions[bot]"
        git config user.email "github-actions[bot]@users.noreply.github.com"
        git checkout -b update-emoji-json
        git add emoji.json
        git commit -m "chore: update emoji.json to latest Unicode version"

    - name: Push Changes and Create PR
      uses: peter-evans/create-pull-request@v4
      with:
        branch: update-emoji-json
        title: "Update emoji.json to latest Unicode version"
        body: "This PR updates emoji.json to the latest Unicode standard."

Additional Context

TBA

[DanyloOliinykSSG]

As continuation of #293

Would you like to keep the same json structure as it is rn? So we need to create a script that will parse https://unicode.org/Public/emoji/latest/emoji-sequences.txt into a json, right?

Is this the json structure library is awaiting for?

{
  "emojiChar": "string",           // The emoji character
  "emoji": "string",                   // Probably we can remove this? As it duplicates emojiChar
  "description": "string",         // Description of the emoji
  "aliases": ["string", ...],        // Alternative names for the emoji
  "tags": ["string", ...],            // Associated tags for categorization
  "unicode": "string",              // Unicode representation of the emoji
  "htmlDec": "string",             // HTML decimal code
  "htmlHex": "string",             // HTML hexadecimal code
  "supports_fitzpatrick": boolean  // (Optional) Indicates Fitzpatrick scale support for skin tones
}

Here's a **draft** script for parsing data from unicode into a json (via ChatGPT, so it might be not accurate):

#!/usr/bin/env python3

import json
import re

def parse_emoji_sequences(file_path):
    emoji_data = []
    pattern = re.compile(r"^(?P<code_points>[0-9A-F\s.]+)\s+;\s+(?P<type_field>[A-Za-z_]+)\s+;\s+(?P<description>.+?)(\s+#\s+(?P<comments>.+))?$")

    def unicode_to_char(code_points):
        """Convert Unicode code points to a single character."""
        return "".join(chr(int(cp, 16)) for cp in code_points)

    def generate_html_codes(code_points):
        """Generate HTML decimal and hexadecimal codes."""
        html_dec = "".join(f"&#{int(cp, 16)};" for cp in code_points)
        html_hex = "".join(f"&#x{cp};" for cp in code_points)
        return html_dec, html_hex

    def expand_range(code_range):
        """Expand a range of Unicode code points (e.g., 1F411..1F412)."""
        if ".." in code_range:
            start, end = code_range.split("..")
            return [f"{i:04X}" for i in range(int(start, 16), int(end, 16) + 1)]
        return [code_range]

    def split_description(description, expanded_codes):
        """
        Split descriptions for emoji ranges.
        If the description contains "..", split it into parts.
        Otherwise, repeat the description for all expanded codes.
        """
        if ".." in description:
            split_desc = description.split("..")
            # Ensure the number of descriptions matches the number of expanded codes
            if len(split_desc) == len(expanded_codes):
                return split_desc
            else:
                # Pad with the last description if lengths don't match
                return split_desc + [split_desc[-1]] * (len(expanded_codes) - len(split_desc))
        return [description] * len(expanded_codes)  # Repeat description for each code point

    def check_fitzpatrick_support(code_points):
        """Check if the emoji supports Fitzpatrick skin tone modifiers."""
        # Simplified check: Fitzpatrick codes are in the range 1F3FB-1F3FF.
        return any(0x1F3FB <= int(cp, 16) <= 0x1F3FF for cp in code_points)

    with open(file_path, 'r', encoding='utf-8') as file:
        for line in file:
            line = line.strip()
            if not line or line.startswith("#"):
                continue  # Skip empty lines and comments
            match = pattern.match(line)
            if match:
                fields = match.groupdict()
                code_ranges = fields["code_points"].split()
                for code_range in code_ranges:
                    expanded_codes = expand_range(code_range)
                    descriptions = split_description(fields["description"], expanded_codes)
                    for i, code in enumerate(expanded_codes):
                        code_points = [code]
                        emoji_char = unicode_to_char(code_points)
                        html_dec, html_hex = generate_html_codes(code_points)
                        supports_fitzpatrick = check_fitzpatrick_support(code_points)

                        emoji_entry = {
                            "emojiChar": emoji_char,
                            "emoji": emoji_char,
                            "description": descriptions[i],
                            "aliases": [],  # No aliases provided in the file, keep empty
                            "tags": [],     # No tags provided in the file, keep empty
                            "unicode": " ".join(code_points),
                            "htmlDec": html_dec,
                            "htmlHex": html_hex,
                            "supports_fitzpatrick": supports_fitzpatrick
                        }
                        emoji_data.append(emoji_entry)

    return emoji_data

def save_to_json(data, output_path):
    with open(output_path, 'w', encoding='utf-8') as json_file:
        json.dump(data, json_file, indent=4, ensure_ascii=False)

# Input and output file paths
input_file_path = 'emoji-sequences.txt'
output_file_path = 'parsed_emoji_sequences.json'

# Parse and save
parsed_data = parse_emoji_sequences(input_file_path)
save_to_json(parsed_data, output_file_path)

print(f"Parsed emoji sequences saved to {output_file_path}")

Currently i've manually downloaded this txt: https://unicode.org/Public/emoji/latest/emoji-sequences.txt and ran script for it. We can update the script later so it automatically downloads that file, then parses.

I see an issue that unicode's emoji-sequences.txt doesn't contain aliases and tags. Any ideas where can we get those? (If it's important for the lib)

I've attached raw emoji-sequences.txt and parsed parsed_emoji_sequences.json as an example:
emoji-sequences.txt
parsed_emoji_sequences.json

[wax911]

Amazing, will circle back over the weekend. I don't mind changing the structure of the json file, we'll just release a major version which is a small price to pay if it means that we can keep up with the latest emoji standard

[wax911]

@DanyloOliinykSSG I was planning on using a secondary data source https://cdn.jsdelivr.net/npm/emojibase-data@latest/en/data.json to obtain additional meta-data, like tags/alias the only limitation with both approaches is that when we release this new version the tag/aliases may differ.

Some insights into supports_fitzpatrick which I can infer from the emoji-base package. The reason why this exists is to shrink the total size of the emoji.json file by removing emojis that need skin-tone modifiers as we can do that on-demand without the need of replication of those emojis.

An additional improvement is also adding a supports_gender which we can use for the exact same use-case as fitzpatrick, but this might be a nice to have.

[DanyloOliinykSSG]

Hi @wax911 ! So you want to merge data from https://unicode.org/Public/emoji/latest/emoji-sequences.txt and https://cdn.jsdelivr.net/npm/emojibase-data@latest/en/data.json?

Regarding the supports_fitzpatrick and supports_gender - yea, that would be a good optimisation 👍

Anything i can help with?

[wax911]

Hey @DanyloOliinykSSG would love a helping hand, I've created a branch feat/226-platform-actions-create-automated-action.. which has a script to generate a compliant .json file for emoji (15.1 for now), we'll replace with with an ENV variable that will be supplied by the github-action that will create an automated PR for each release of https://unicode.org, also noticed that emojibase is actually aligned with unicode.org so there was no need to merge data :D however I made some filtering decisions to reduce the size of the generated file.

The next few items we need to do is update the contract of IEmoji to match the new schema, I will release this change as a new major version update as the emoji data vastly differs.

So to wrap it up here's the next few things we need to do to complete this task:

• Create a new github-action to execute the python program
• Update the the kotlin projects' module to take accommodate the upcoming changes (Includes some tests)

[DanyloOliinykSSG]

Wow, that's a lot of work being done @wax911 ! 😯😎

also noticed that emojibase is actually aligned with unicode.org so there was no need to merge data

phew, that's cool 😌

we'll replace with with an ENV variable that will be supplied by the github-action that will create an automated PR

ENV variable with the latest release version?

I'm not sure how that "hook" should work.. i mean how to trigger automated PR for unicode emoji releases? Can it be reactive, or it's just like "check updates once in XXX days?" 🧐

Also regarding the Update the the kotlin projects' module to take accommodate the upcoming changes (Includes some tests) - what should be updated? You mean updating the BL to accept new model of Emoji?

And a sidenote/question: why do we need an interface for a model? Do you anticipate different implementation of it? AFAIK it's not common to have models under interfaces 🙄 But i'm a bit out of context of this repo, so sorry if that's a weird question 😄

[wax911]

ENV variable with the latest release version?

I'm not sure how that "hook" should work.. i mean how to trigger automated PR for unicode emoji releases? Can it be reactive, or it's just like "check updates once in XXX days?" 🧐

I was thinking cron job to run every Sunday e.g.

name: emoji-version-bump

on:
  workflow_dispatch:
  schedule:
    - cron: "@weekly"

jobs:
   ...

But to your comment perhaps we don't need to pass in the version as we could just use latest as the version tag and only create a PR if the current version is different than what we last ran against, need to think about how to go about this in more detail 😆

Also regarding the Update the the kotlin projects' module to take accommodate the upcoming changes (Includes some tests) - what should be updated? You mean updating the BL to accept new model of Emoji?

The current serializers will expect the current structure of emoji but we've dropped alias and some redundant keys like emojiChar e.t.c in addition to some new ones. In that case it'll be a case of "update this, see what needs to be refactored, tests, iterate and release an internal beta.."

And a sidenote/question: why do we need an interface for a model? Do you anticipate different implementation of it? AFAIK it's not common to have models under interfaces 🙄 But i'm a bit out of context of this repo, so sorry if that's a weird question 😄

Good question! Normally, models don’t require an interface, but in this case, it follows Dependency Inversion Principle (DIP) and is inspired by the Strategy Pattern. The library provides multiple serializer implementations (Moshi, Gson, Kotlinx Serialization, etc.), but the core module doesn’t care which one the consumer uses and by enforcing a contract (interface), all serializer modules can conform to a common structure without duplicating the data model. This ensures that if a consumer chooses Moshi, they don’t need to pull in Kotlinx Serialization or Gson as unnecessary dependencies.

The goal is to keep the library modular and lightweight, allowing consumers to pick only what they need.

The Strategy Pattern fits here since different serializer strategies (Gson, Moshi, Kotlinx) can be swapped without modifying the core logic, making it more flexible.

[wax911]

@DanyloOliinykSSG please review #321 when you have a moment, this takes care of the automation aspect of creating an update emoji-base alined .json file. The next step is to update the library itself and publish a new version of it 🧶

[DanyloOliinykSSG]

Yup, reviewed it. I guess it's fine - the only thing is maybe we can test isEmoji on suggested by android team emojies 🙄 (left emojies list and a link in the PR comment)