Braille: Variation Selectors break cursor positions

[aaclause]

Steps to reproduce:

1. Consider the following line:

⚠️ test

2. Move the cursor on the word (test) using routing cursors.

Actual behavior:

The cursor is moved on — the desired position + 1 —. Also ⚠️ takes 2 braille cells.

Expected behavior:

The cursor should be moved on the desired position. Also ⚠️ symbol should takes one cell only.
With wordpad and in form mode in Firefox, cursor is properly moved.

System configuration

NVDA installed/portable/running from source:

Installed and portable

NVDA version:

2019.3, 2020.1

Windows version:

10 Insider (64-bit) build 19592.1001

Name and version of other software in use when reproducing the issue:

notepad, Microsoft Word, Firefox (in browse mode), Run dialog (windows+r), etc..

Other information about your system:

Other questions

Does the issue still occur after restarting your computer?

Yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

No

If addons are disabled, is your problem still occuring?

Yes

Did you try to run the COM registry fixing tool in NVDA menu / tools?

No

CC @LeonarddeR

[LeonarddeR]

Could you test the behavior with this in NVDA 2019.2 please?

[aaclause]

More details (according to my modest research):
⚠️ is composed of 2 characters: \x26a0 (⚠) and \xfe0f.
The issue comes from \xfe0f (variation selector-16). This character belongs to variation selectors Unicode block. It seems that all characters in this block are invisible codepoint which specify that the preceding character should be displayed with another presentation/color. Also it seems that we shouldn't reach these characters with keyboard (arrow keys, etc.).
In the end, these are all characters in this range (FE00 to FE0F) that break cursor positions. Another example: ⚠︀ = \x26A0 + \xfe00.
With NVDA 2019.2, we can move cursor on variation selectors. Therefore cursor positions are not broken.

[LeonarddeR]

This is a difficult one. A quick solution would be: strip variation selectors from braille output. Not something I really like, as basically we're removing output that might be considered relevant for some people.

Thoughts @michaelDCurran @dkager @Adriani90 @lukaszgo1?

[aaclause]

It seems that there are other signs more complexes.
E.g.: 1️⃣ 2️⃣ 3️⃣ 4️⃣ 5️⃣ 6️⃣ 7️⃣ 8️⃣ 9️⃣
0️⃣ *️⃣ #️⃣
3 braille cells per signs including U+FE0F and U+20E3 (variation selector-16) and combining enclosing keycap).

[aaclause]

I've just made a first attempt in Braille Extender add-on, there's probably a better solution. See aaclause/BrailleExtender#63.
Doesn't work in wordpad and in Firefox (in form mode).

[LeonarddeR]

I'm pretty sure that in #16219, @mltony laid the groundwork to fix this. Braille cursor movement should start relying on code points rather than characters. As liblouis is using 32 bit internally, that will always match.

[LeonarddeR]

Unfortunately, my work had to be reverted due to an oversight in the code for UIA in Word. I'll try to fix this later this week.

[LeonarddeR]

The complexity here lies in the bullets and numbers part added in #8576

In this pr, @michaelDCurran changed bullet and number handling by adding the bullet/number to the line-prefix field on the format field before the text string. There are several problems with that approach:

1. It only works reliably when the line has text after the bullet. If not, the bullet is exposed as text anyway
2. It doesn't affect getting just the text of the textInfo, so when getting the text property on the TextInfo instance, the bullet is still exposed in text.

When the text in fields and the text without fields don't match, it is impossible to tell moveToUnicodePointOffset where to move to when routing braille.