Regex: unicode property S doesn't match '🥰' emoji and others

[g-andrade]

Describe the bug

The regular expression "\\p{S}", matching codepoints categorized as symbols, doesn't match the string "🥰" - unlike for other emoji.

To Reproduce

% re:run("🥰", "\\p{S}", [unicode]).
nomatch

Expected behavior

What happens with other emoji:

% re:run("😎", "\\p{S}", [unicode]).
{match,[{0,4}]}

Character properties for 😎, as well as for 🥰, both list their general category as 'Other symbol'.

Affected versions

OTP 27.0.1 on GNU/Linux

[g-andrade]

Further investigation uncovered a total of 432 emojis, out of 1902, not matched by unicode property S:

# "full_emoji_list.txt" |> File.read!() |> String.split("\n", trim: true) |> Enum.filter(&(:re.run(&1, ~c'\\p{S}',  [:unicode]) === :nomatch)) |> length()
432

• full_emoji_list.txt, extracted from unicode.org.
• emojis_not_matching_regex_S.txt

At least some of them match the unicode property C instead (control characters).

[garazdawi]

I think that this is because the emojis in question are from a unicode standard from after the last update of the underlying pcre library. There is nothing we can do about this except migrate to a different re backend, which is something that we have planned to do for a long time, but have not gotten around to yet due to backwards compatability issues with other implementations.

[g-andrade]

I suspected pcre, that makes sense. In the process I noticed an up to date emoji-data.txt, used for unicode-related code generation, and wondered: is the info on grapheme (cluster) categories, if any, exposed somewhere in the stdlib? - as an alternative to regular expressions.

Essentially, a way to answer the question: according to the unicode standard shipped with a specific version of OTP, is X a symbol? (Or any other unicode category/subcategory)

[garazdawi]

You can do it through an undocumented function:

1> unicode_util:lookup($🥰).
#{category => {symbol,other},
  canon => [],ccc => 0,compat => []}

maybe @dgud knows of an official API that can be used?

[dgud]

No API is exposed today, unicode_util was made to support the module strings "new" API.

Some functionality could be exposed in the unicode module maybe?
Nothing we have intended to work with a PR would be interesting?

[garazdawi]

Closing this as there is not anything we can do about this except change re implementation and that is tracked elsewhere.