Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Include Sentence Nav as part of NVDA #8518

Open
andrew-l-d opened this issue Jul 19, 2018 · 77 comments · May be fixed by #16384
Open

Feature request: Include Sentence Nav as part of NVDA #8518

andrew-l-d opened this issue Jul 19, 2018 · 77 comments · May be fixed by #16384
Labels
p5 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority triaged Has been triaged, issue is waiting for implementation.

Comments

@andrew-l-d
Copy link

Tony Malykh's Sentence Nav add-on is a most valuable resource. Not only does it resolve #5805, but it allows reading by sentence in a wide variety of applications. I humbly suggest that the facility to read by sentence should be an integral component of a screen reader. I have checked with Tony and he is happy for me to make this suggestion.
Andrew

@DrSooom
Copy link

DrSooom commented Jul 19, 2018

Please use NVDA+ALT-ArrowUp/ArrowDown instead of just ALT+ArrowUp/ArrowDown because otherwise it negatively influences the behaviour of ALT+ArrowDown in some programs like in MuseScore as well as comboboxes in Mozilla Firefox.

Quote from my "gestures.ini":

[globalPlugins.sentenceNav.GlobalPlugin]
previousSentence = kb:alt+nvda+uparrow
None = kb:alt+downarrow, kb:alt+uparrow
nextSentence = kb:alt+downarrow+nvda

@mltony
Copy link
Contributor

mltony commented Jul 19, 2018

This is Tony - the author of SentenceNav.

  1. My main reason to use Alt+Up/Down arrow is to make this shortcut consistent with Jaws. We probably don't want to make Jaws to NVDA transition harder by defining too many new keyboard shortcuts to learn.
  2. Alt+Up/Down arrows on the combo boxes should not be affected in any application. When either of these keystrokes is issued, I check for the currently focused element and if it happens to be a combo box, I issue gesture.send() to have this keystroke be processed by the operating system.
  3. Some other applications might indeed be affected by Alt+Up/Down shortcut being Overridden. But the same can be said about Control+UpDown arrows. In fact the same can be said about any NVDA shortcut that doesn't involve NVDA key. In this case I can argue that a potential workaround is to have the user to customize the keystroke in the configuration to avoid the conflict.

@fernando-jose-silva
Copy link

Outlook 2016 uses alt + arrow below to open the options of what to do with attachments.
For being such a common software would vote for creating a function so that the alt + arrow below ignore the phrases function in this list of attachments in outlook 2016.
For novice users pressing alt + down arrow in the list of attachments and seeing that it is not open can be frustrating and can lead to the user thinking that it is due to an error in nvda.
Remapping keys in nvda is also not very logical for beginners.

@DrSooom
Copy link

DrSooom commented Jul 19, 2018

At the SightCity 2018 a contributor told me that the screenreader user should never change the hotkey allocations. After he noticed that I'm an advanced user, he said to me that I can do this because I know exactly what I'm doing. (I don't want to name the company here.)

@mltony: Please do the following twice – once with your add-on enabled and once without it:

  1. Open the Windows Explorer and open C:\Users.
  2. Change the view to the detail view.
  3. Press the right arrow once and you should be in a protected pseudo text field.
  4. Press ALT+ArrowUp to go one folder higher (to C:).

So, please forget JAWS and use the NVDA+ALT-combos because it really isn't necessary in every application to have the ability to read by sentences. Only Alt+ArrowKeys cause to many navigations issues for all NVDA users. CTRL+ArrowKeys and CTRL+ALT+ArrowKeys haven't such negative effects yet (NVDA 2018.1).

@LeonarddeR
Copy link
Collaborator

Note that it is very unlikely and even undesirable to add sentence navigation to NVDA for cases other than the review cursor. Therefore, if sentence navigation is added, it will only be added to the review cursor (in which case kb(laptop):NVDA+ALT+upArrow and downArrow make much sense) and browse mode.

@feerrenrut and @michaelDCurran, do you have thought about this?

@DrSooom
Copy link

DrSooom commented Jul 20, 2018

Well, and the abbreviations has also to be translated into the different languages as well. There are hundreds in German and some of them are written differently in Austrian German (de-AT) like "zB" and "z.B." for "z. B.".

@Adriani90: See ÖNORM A 1080 (which seems to be completely stroked since May 1st, 2018) or this PDF file. Only the abbreviations ending with a dot are important here.

Sadly this PDF file isn't complete. The following are missing based on my quick look: Mo., Di., Mi., Do., Fr., Sa., So., Hr., Hrn., abzügl., bspw., bzgl., bzw., gem., Jhd., lit., o.ä., o. ä., od., o.g., o. g., u.U., u. U., u.a. (without the space), Z. and maybe even many more.

[Update: 01:40 CEST): Tsd., Mio., Mrd., tägl., wöchentl., monatl., jährl., inkl. and exkl.

@andrew-l-d
Copy link
Author

Re Leonard's comment: Note that it is very unlikely and even undesirable to add sentence navigation to NVDA for cases other than the review cursor. The Sentence Nav add-on is not restricted to review cursor and, in my experience, it is its general availability that makes it so valuable. Re the key combination, the only case so far where I have had to use the bypass is when saving attachments in Outlook. Having installed the add-on, it was immediately obvious that there would be a keyboard clash. It may not be as obvious to some users if the feature was part of NVDA upon installation (especially those who do not read manuals [smile]). But before getting bogged down in which key combination to use, a decision needs to be made as to whether the feature should be included as a core component of NVDA. While it is my view that people should not have to use an add-on to read by such a fundamental unit, that is not my call.
Andrew

@mltony
Copy link
Contributor

mltony commented Jul 21, 2018

Okay, I seem to be outgunned here, I can change the keystrokes to NVDA+Alt+Arrows.

  1. Do you want me to change it in my github repo? This is really a one-line change, so anyone can do it - e.g. whoever is going to be merging the code into NVDA.
  2. Do you want me to change Alt+Shift+Arrows to NVDA+Alt+Shift+Arrows for jump to next text paragraph as well? Or we are not talking about jump to text feature in this issue?
  3. NVDA has a built-in Alt+Up and Alt=Down keystrokes for MS Word and Wordpad - someone would have to change that to NVDA+Alt+Arrows to be consistent.

For the record, I still disagree that this is right decision. I agree to make the change, but I still need to address all your points here.

  1. @DrSooom, you cannot ask me to forget about Jaws. It's still a leader in this market and most of NVDA users come from Jaws, and I would think that NVDA ought to care how easy it is to switch. You can tell me that not creating keystroke conflicts is more important than that, and this is a judgement call, and I would respect your opinion.

  2. @fernando-jose-silva, @DrSooom, A quick search revealed that Control+arrows is used as a shortcut in Outlook and IE, I can send you the links if you want. Control+Alt+arrows is known to be a shortcut in Intel display drivers and remote console. So I would disagree that only Alt+arrows causes keystroke conflicts. It seems that NVDA users have been living for years with these conflicts. It's just that Control+arros seems to be a universal shortcut among all screenreaders and blind people never had a chance to use the original Control+arros keystrokes in Outlook. I would still agree that breaking Alt+arrows functionality in existing apps might be inappropriate, but I just want to point out that there are existing keystroke conflicts too. And the question again becomes whether we want to be consistent with Jaws or strive not to introduce any new keystroke conflicts? You have answered this question above, and I respect your answers.

  3. Having sentence navigation bound to Alt+arrows would be consistent in many ways. Besides being consistent with Jaws (which I understand you don't care so much about), it would be logically consistent with other NVDA shortcuts, such as Control+Up/Down for paragraph navigation. It would be easy to remember for new users that Control key means to navigate by paragraph and Alt means by sentence. Moreover, it would be consistent with current NVDA behavior in Microsoft Word and Wordpad - Alt+Up/Down arrows is already bound to sentence navigation commands there. But I hear you, you care more about not creating new keystroke conflicts.

  4. (Probably no one would care about this one, but I'll still say it here for completeness sake) I already use NVDA+Alt+Arrows shortcuts in my other add-on IndentNav. Sure, I can change it there too to something even longer, but we are running out of possible shortcuts involving the arrow keys. Soon someone might have to use some monstruous NVDA+Control+Shift+Alt+arrows for some new feature because all the other shortcuts will be taken by then.

  5. The ideal solution in my mind would be to have NVDA the ability to assign a keystroke selectively depending on the currently focused control - that is only assign Alt+arrows keystroke in those contexts where navigation by sentence makes sense, such as text editor controls and HTML controls inside browsers and other apps. But when I did my investigation, that was not possible due to NVDA API limitations - correct me if I'm wrong here. Back from when I wass working on SentenceNav I remember that I found a way to define Alt+arrows shortcut only for edit box and MS Word controls, but there was no way to define it for browse mode, at least in an add-on. If it is done within NVDA, it might be possible to add a new keystroke just to Browse Mode. I hope this way would not cause any conflicts, since Alt+Arrows is not used in these contexts, with the exception of combo box - which is already resolved in SentenceNav. Does anybody think this is a viable option? Someone would have to implement context dependent keystrokes in NVDA and I don't know how difficult that would be.

  6. @LeonarddeR, could you explain why it would be undesirable to have sentence navigation in system focus mode? I would argue that having sentence navigation in system focus mode is more important that in review mode. I don't know about you - everyone uses NVDA in their own way, but I only use review mode in the places where system cursor cannot go to, for example command line window. This is a place where I don't need sentence navigation whatsoever - typically in command line window nothing comes out formatted as sentences. The places where I do like sentence navigation are browsers and text editors. In browsers I would have to switch to review cursor to read sentence by sentence and then route system focus to review cursor to keep reading - just more hassle than reading by sentence right with system focus. So I'm afraid, restricting sentence navigation to review cursor might only make this feature more difficult to use. And by the way, Alt+arrows shortcuts work as sentence navigation in MS Word and Wordpad in system focus mode out of the box. So if you think that this behavior is undesirable, should you remove this feature first or modify it to work only in review cursor mode?

  7. @DrSooom, Surely enough other languages can be added. But I would like to point out that even without the dictionary of abbreviations SentenceNav can detect the vast majority of sentences correctly. You cannot detect all of them correctly. For example, "The president of U.S.A. is Trump.". In this sentence there is no way we can figure out that the dot after U.S.A. doesn't end the sentence. So even in the best case SentenceNav will only be 99% accurate. Now adding new abbreviations for say German might actually bring down accuracy for English, since there is no way to determine what language the text is written in. For example, if there is abbreviation "York." in german - I don't actually speak German, so this is just an example, then the dot in this sentence "I travelled to New York." will not work as a sentence delimiter in English text. So I just want to point out another trade-off here. I would say that adding abbreviations in other languages is Ok as long as it doesn't hurt accuracy in English.

@mltony
Copy link
Contributor

mltony commented Jul 21, 2018

Oh, and I forgot the most important one.

  1. Most likely any keystroke conflict can be resolved. I have already resolved the conflict of Alt+Down on a combo box in Sentence Nav. I wasn't aware of a conflict in Outlook, but it shouldn't take too long to detect this situation when the focus is in Outlook and make sure that the original action is triggered.

@Adriani90
Copy link
Collaborator

@mltony, I guess alt+arrow keys is still the best option for this. ctrl+alt+arrow keys can for example change the screen orientation in MS Excel or in other applications. NVDA+Alt+arrow keys is also used in indentnav. Right?

In my opinion, the best way would be to lock sentencenav in such a way that NVDA reads automatically by sentence when pressing up and down arrow key. That means, NVDA would not read one line but one sentence. The keystroke to lock sentencenav could be something like pressing ctrl, shift, ctrl in this order. To unlock it you could press shift, ctrl, shift.
Another possibility would be to integrate textnavigation in the settings ring and let the user decide what type of navigation should be applied for the nex arrow key presses (i.e. paragraph, sentence, line).

In any case this add-on should be part of NVDA's core as soon as the key conflicts are solved.

@Adriani90
Copy link
Collaborator

Guys, let's turn the discussion in the right way. We can discuss about NVDA and Jaws on the mail list if needed. We want to find the best way to work productively with NVDA and a comparison with Jaws is not the what we want to end up with.

@DrSooom
Copy link

DrSooom commented Jul 22, 2018

@mltony:

  1. I agree that it isn't really possible to correctly detect if "U.S.A." stands in the middle or at the end of a sentence. Example: I'm living in the U.S.A. And U.S.A. stands for United States of America. Microsoft Word 2010 navigates (because of the NVDA feature) without your add-on via ALT+ArrowUP/ArrowDown from "I" directly to the next sentence after "America.". Your add-on navigates here (now in Firefox, browse mode) correctly between this two sentences. So without an AI this won't be fully fixable.
  2. I would recommend to add the ability to choose the list of abbreviations by the end user via a combobox in the NVDA settings. Oh, no, wait. The translated list of abbreviations should be related to the current used TTS language. So it would be change automatically every time when the TTS changes its language (e.g. based on the lang-HTML-Attributes).
  3. I forgot "zzgl." in German and "e.g." in English.
  4. In my opinion whitelisting ALT+ArrowKeys for just a limited amount of software (which should be enabled and disabled by the user) is easier instead of fixing possible bugs with all the other software at the market. It's still too much.
  5. Only because I'm not using you add-on on my NUC, I was able to understand, that there wasn't a new bug in MuseScore regarding to ALT+ArrowDown and ALT+ArrowUp (which ´both are very, very important here!), this issue was caused to 100 % by your add-on. So if you add-on in version 1.0 is implemented in NVDA at this state, I guess that it would be quiet hard to checkout which application is responsible for this issue – especially for those who aren't familiar with PCs. We should never forget this group of users.
  6. Well, and then: Should a sentence also to be split after a colon like in this question here? Of course, in the end the user should have the option to enable or disable this.
  7. I'm just using your add-on in Firefox and Thunderbird. It doesn't work in HTML-E-Mails in Microsoft Outlook 2010 (browse and focus mode).
  8. NVDA+ALT+ArrowUP/ArrowDown based on your add-on will only move the Cursor in Microsoft Word 2010 between the sentences, but NVDA speaks nothing. But in browse mode in Firefox speech and braille output exists. Why is this different?
  9. And is it possible to disable the braille output (braille message) in browse and focus mode in Firefox, Thunderbird & Co., because hearing the braille cells moving irritates me a lot? That would be nice too. Thanks.
  10. And your add-on interrupts/stops SayAll, CTRL+ArrowUp/ArrowDown doesn't do this in browse mode in Firefox and Thunderbird.

@Adriani90: I see that including this feature into the settings ring could be useful for some users. But your recommendation about locking and unlocking with CTRL » Shift » CTRL is much too complex for non-advanced NVDA users – and inefficient as well because you have to press too many keys just for using a single function.

@mltony
Copy link
Contributor

mltony commented Jul 22, 2018

@DrSooom

  1. I like your idea to only enable Alt+arrows in a white list of apps as long as this list can be configured by the user. I think this is a reasonable compromise that would satisfy everybody. I would be OK with black list too.

  2. I admit that keystroke conflicts should be taken seriously. Unfortunately I haven't used MuseScore before and nobody brought it up to my attention until now, that there is a conflict in it. There are multiple ways to resolve this conflict, one of them being your white list idea, another being to only redefine Alt+arrows shortcuts in edit boxes and HTML controls - - as I mentioned that is not possible within add-on API, but it should be possible if implemented within NVDA itself.

  3. I just checked - it works in my Outlook 2016 in both focus and browse mode. I'll need to find a computer with Outlook 2010 to see what's wrong. Are you using the latest version 1.1?

  4. NVDA+Alt+arrows is currently not a shortcut in SentenceNav. Did you mean Alt+arrows?

  5. Sentence navigation in Microsoft Word comes with NVDA out of the box and is bound to Alt+arrows. That is you don't need SentenceNav for this to work.

  6. I'm not sure what did you mean about braille cells moving and Braille message. SentenceNav currently calls ui.message() to speak and to the best of my knowledge, this is a suggested way for an add-on to speak. If there is another function I should use rather than ui.message(), let me know. Same applies to say all - that must be how ui.message works.

@Adriani90,

  1. I don't like the idea of switching between line navigation and sentence navigation modes with a keystroke, because it adds some internal state that user would have to keep in mind. Also it prevents muscle memory from forming. This reminds me of rotor function in VoiceOver for IOS, where swipe up or down might mean jump by character, word or heading. In the end you never know which one is active and you always have to check the rotor setting. But in their case using rotor is justified because there are only so many swipe gestures available. Also navigation by line using Up and Down arrow keys is a system function, not an NVDA one, so if we have to catch these keystrokes and resend them to the system only when the mode is set to line navigation, this might cause some unexpected bugs.

  2. Yes, I am already using NVDA+Alt+arrows shortcuts in my other add-on IndentNav.

As for whether sentence navigation should be a part of NVDA core - you guys should decide. I am obviously biased, I missed this feature a lot when I switched to NVDA and it is extremely important to me. I find it especially useful when reading scientific papers or articles, where I need to repeat every sentence multiple times. In this case if I'm left with line navigation, my brain has to keep diverging some share of attention away from the topic of the article just to reconstruct the boundaries of the sentences.

@ehollig
Copy link
Collaborator

ehollig commented Jul 22, 2018

Two related issues, #5463 and #1873

@DrSooom
Copy link

DrSooom commented Jul 22, 2018

@mltony:

  1. As you could might remember I remapped the default hotkeys from ALT+ArrowKeys to NVDA+ALT+ArrowKeys. That's why your add-on in version 1.0 works differently in Microsoft Word 2010 – but not longer with version 1.1. (I updated it now.)
  2. The issue with Microsoft Outlook and HTML-E-Mails was fixed by version 1.1.
  3. Press NVDA+F12 for getting the current time. It is also shown for a few seconds on the braille display. And sadly it's exactly the same with your add-on in version 1.0 and 1.1 in browse mode in Firefox and Thunderbird as well as in (Windows) Notepad for sentences. In other words: Every sentence is shown with its first word at cell No. 1, even if the sentence beginning normally appears in the middle of the braille display. And after this 4 seconds (default Braille Message Timeout) the braille display will show the normal (browse mode) layout of the text. This behaviour doesn't exist with CTRL+ArrowUp/ArrowDown. Maybe there is an limitation for add-ons here. But I haven't the knowledge to claim this.

@LeonarddeR
Copy link
Collaborator

LeonarddeR commented Jul 23, 2018

Alt+upArrow and alt+downArrow both are shortcuts used in applications for sentence navigation. Note that in these cases (e.g. MS Word), the application itself is responsible for the sentence navigation. The only reason why NVDA reads by sentence in MS Word is because Word moves the cursor and NVDA reflects that behaviour with speech.

Update: it turns out that my assumption was wrong, see also #8541 3288

@netblue44
Copy link

netblue44 commented Jul 23, 2018 via email

@mltony
Copy link
Contributor

mltony commented Jul 23, 2018

@LeonarddeR:

  1. That's not true. Microsoft Word doesn't provide Alt+Arrows keystrokes out of the box. Try searching the list of MS Word shortcuts and you won't find Alt+Arrows bound to anything. Nor does MS Word provide any other keystroke to navigate between sentences. Alt+Up and Down arrows is an NVDA shortcut, that currently only works in MS Word. I am actually not quite sure how it works, but my best guess is that MS Word keeps track of sentences in the document and NVDA retrieves this information via some API call to Word. But I bet someone else knows more how exactly this happens. Also you can verify that Alt+arrows don't work in Narrator in Microsoft Word.

  2. Why do you think NVDA should not add read by sentence functionality with alt+arrows? If other developers have any arguments I am welcome to hear them.

  3. We are not stuck. You can view SentenceNav as a working prototype of this functionality.

@LeonarddeR
Copy link
Collaborator

Wow, my apologies there. I really was quite sure that this was Word functionality, but #3288 turns out that this is NVDA specific indeed.

@DrSooom
Copy link

DrSooom commented Jul 24, 2018

@mltony: Sadly I forgot to mention two more things regarding to date:

  1. German abbreviations for months: Jan., Jän., Feb., Mär., Mrz., Apr., Mai., Jun., Jul., Aug., Sep., Sept., Okt., Nov. and Dez.
  2. And of course the date format which is mostly used in German: d. mmmm yyyy or dd. mmmm yyyy. Currently your add-on in version 1.1 split a sentence within the mention date. So please try to implement a rule that "01. " to "09.", "1. " to "9. " and "10. " to "31. " won't be detected as the end of a sentence. And as the day is mentioned alone it also shouldn't split here.
    Example: Heute ist der 24. und morgen der 25. Juli 2018. (EN: Today is 24th and tomorrow July 25, 2018.)

@mltony
Copy link
Contributor

mltony commented Jul 24, 2018

@DrSooom:

  1. I propose to separate adding fine-tuned support for German and other languatges into a separate issue to be worked on perhaps later, assuming if we agree on this one first.

  2. Can a German sentence end with a number? Something like "Dieses Number ist 25." If this is a valid sentence, then we cannot claim that a dot right after a number is a sentence breaker. It might be still worth it to break the sentence here anyway, just because dates tend to appear way more often than sentences ending with numbers. In this case we should just add all the numerals 1-31 to German dictionary - that is when language-specific dictionaries are implemented - because probably it's a good idea not to introduce language-dependent code, especially when we can get around by language-dependent dictionary.

@DrSooom
Copy link

DrSooom commented Jul 24, 2018

Yep, of course this could be happen, but the date appears more often. In the end I suggest a checkbox for this, so the user can enable/disable sentence splitting here. By default splitting here should be disabled.

@netblue44
Copy link

netblue44 commented Jul 24, 2018 via email

@LeonarddeR
Copy link
Collaborator

This issue has been abandoned for some time now. Honestly, the more I think about this, the more I believe that Sentence Nav should stay as an add-on.

@andrew-l-d
Copy link
Author

I fundamentally disagree that one should have to get an addon to read by sentence. All other Windows-based screen readers I have used include this facility. As I mentioned previously, reading by sentence is (or should be) a basic reading mode. The discussion here became bogged down with debate about the best keystroke when the real issue is whether NVDA should provide people with the ability to read by sentence (incidentally, I changed to Windows-arrow keys). Can you give me one good reason why reading by sentence should not be provided, especially now that addons will cease to be available if their developers do not continue to revise them?

Andrew

@mltony
Copy link
Contributor

mltony commented Oct 3, 2019

@LeonarddeR
I'm not sure why did you reopen this discussion now, but a year ago you already called sentence navigation "undesirable" - see your own comment in this thread from on Jul 19, 2018. Has anything changed since then? And it would greatly help if you could provide some justification about your opinion - otherwise it sounds like a personal preference.

@Adriani90
Copy link
Collaborator

This feature is so often requested by users, honnestly I don't know why it is so complicated to implement it. There is already an addon which works. Why not taking its core function and implement it into the core? Obviously this feature is not an exception, it is a navigation mode that every screen reader user would expect to belong to the standard package.
cc: @feerrenrut, @michaelDCurran

@josephsl
Copy link
Collaborator

josephsl commented Oct 3, 2019 via email

@andrew-l-d

This comment was marked as resolved.

@bhavyashah
Copy link

A gem of a quote from #8518 (comment): "... primarily since only 10% of NVDA users know what is an add-on and how to install one, and that's probably an overly optimistic statement." Not sure of the accuracy of the statistic - my intuition is that it is - but I resonate with the sentiment. I was teaching NVDA to a beginner recently, and I realized how many add-ons I had to have him install in order to perform natural screen reading tasks he asked about. Anyhow, pinging @seanbudd in case there have been any updates since #8518 (comment). No worries if this fell through the cracks (happens to the best of us!), but please consider prioritizing sharing NV Access's position at the earliest given the age of this ticket and the level of interest.

@andrew-l-d
Copy link
Author

Due to other commitments, it took Tony some time to update SentenceNav to be compliant with NVDA 2023.1 and how frustrating it was for me. I intended to nag ... no, remind about this glaring omission in NVDA but did not get around to it. The above comment has coaxed me into action.

Andrew

@cary-rowen
Copy link
Contributor

We merged #13797. We've also added a Document Navigation category to the Settings panel. Sentence navigation is similar. What is NV Access's current stance on this feature request? Any updates?

@CyrilleB79
Copy link
Collaborator

@seanbudd, @gerald-hartig, given #16031 implements an algorithm to detect sentence separators, it would really be the opportunity to state on this issue and to triage it. Also, the more because @mltony (author of sentence nav add-on) is available these days (on contrary to usual) and is quite productive; it would be a pity to miss this opportunity.

NV Access has planned to look at this issue two years ago (see #8518 (comment)), but it seems that nothing has happened since then.

@michaelDCurran
Copy link
Member

We will take this feature. We already have moving by sentence in MS Word, using alt+up/downArrow. I have not looked at the add-on itself, but all I would ask is that it was abstract enough so that a particular TextInfo implementation can provide its own sentence unit if the underlying accessibility / platform API has a concept of sentences. Then in the cases where there is no native support, we will use the generalised algorithm.
There is still the concern of alt+up/downArrow in browse mode being used to manipulate open combo boxes. How would this conflict be addressed?

@michaelDCurran michaelDCurran added triaged Has been triaged, issue is waiting for implementation. p5 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority labels Jan 23, 2024
@CyrilleB79
Copy link
Collaborator

We will take this feature. We already have moving by sentence in MS Word, using alt+up/downArrow.

Thanks @michaelDCurran for the "triaged" label.

I have not looked at the add-on itself, but all I would ask is that it was abstract enough so that a particular TextInfo implementation can provide its own sentence unit if the underlying accessibility / platform API has a concept of sentences. Then in the cases where there is no native support, we will use the generalised algorithm.

@mltony are you still interested in implementing this in core with the design suggested by @michaelDCurran, i.e. a general algorithm in TextInfo to define sentence unit, that could be redefined by a specific subclass of Textinfo?

There is still the concern of alt+up/downArrow in browse mode being used to manipulate open combo boxes. How would this conflict be addressed?

Jaws also uses alt+up/downArrow in browse mode. It may be worth looking at how it behaves on webpages containing comboboxes.

Anyway, I am not concerned about this.
If we cannot fin a solution to handle this conflict, we still can add Sentence Nav in core with another non-conflicting gesture or with unassigned gestures, what would already be a plus with respect to what we have today (i.e. nothing except in Word).

@DrSooom
Copy link

DrSooom commented Jan 23, 2024

@michaelDCurran: I changed these hotkeys from ALT+ArrowUP/ArrowDown to NVDA+ALT+ArrowUP/ArrowDown.

@mltony
Copy link
Contributor

mltony commented Jan 23, 2024

Thanks @michaelDCurran , I will start working on that.
Re Alt+Down conflict: currently we handle it by checking the role of current element and in case it's combo box we toggle it instead of doing sentence navigation. AFAIK that'also how Jaws does it.
I see there's textInfos.UNIT_SENTENCE constant, I will try to rewrite SentenceNav so that it makes use of that framework.
Also one thing to mention is that SentenceNav tries to reconstruct sentences spanning across multiple paragraphs. I found it useful in two cases: when someone sends an email using GMail Basic HTML interface - in this case every line turns into its own paragraph, so sentences do span across multiple paragraphs most of the time; and the other use case is reading scientific papers, where PDF format has the same problem. SO the algorithm that reconstructs sentences across paragraphs is somewhat complex. I will add plenty of comments describing what is being done there, but I thought just wanted to call that out now.

@CyrilleB79
Copy link
Collaborator

Also one thing to mention is that SentenceNav tries to reconstruct sentences spanning across multiple paragraphs. I found it useful in two cases: when someone sends an email using GMail Basic HTML interface - in this case every line turns into its own paragraph, so sentences do span across multiple paragraphs most of the time; and the other use case is reading scientific papers, where PDF format has the same problem. SO the algorithm that reconstructs sentences across paragraphs is somewhat complex. I will add plenty of comments describing what is being done there, but I thought just wanted to call that out now.

I am unsure that the paragraph reconstruction of Sentence Nav add-on should be ported as is in NVDA core.
Indeed, I think that NVDA's TextInfo has its own notion of paragraph. If there are situations where paragraphs are not detected correctly, it should be fixed in NVDA globally, i.e. should also impact paragraph navigation commands and paragraph selection commands.

@cary-rowen
Copy link
Contributor

@DrSooom
I prefer Alt+Up/Down which is consistent with MS Word.
Also, I've been using sentence navigation for several years and haven't been bothered by the conflict between Alt+down to expand combo boxes and sentence navigation.

@cary-rowen
Copy link
Contributor

Also one thing to mention is that SentenceNav tries to reconstruct sentences spanning across multiple paragraphs. I found it useful in two cases: when someone sends an email using GMail Basic HTML interface - in this case every line turns into its own paragraph, so sentences do span across multiple paragraphs most of the time; and the other use case is reading scientific papers, where PDF format has the same problem. SO the algorithm that reconstructs sentences across paragraphs is somewhat complex. I will add plenty of comments describing what is being done there, but I thought just wanted to call that out now.

For me, this works really well, and my use of sentence-nav relies heavily on this feature.
I'm sure you'll want to review your code to make it more robust before bringing sentence-nav into NVDA.
There are currently several Open Issues for sentence-nav that you might want to take a look at.

@mltony
Copy link
Contributor

mltony commented Jan 24, 2024

@CyrilleB79 , I created a separaet issue #16089 to discuss whether it is feasible to fix paragraphing. TLDR, I don't see a good solution and I explain there why existing algorithm of SentenceNav won't work well for fixing paragraphing. But LMK if you have any other good ideas.

@mltony
Copy link
Contributor

mltony commented Jan 25, 2024

@CyrilleB79, I didn't hear back from you regarding fixing paragraph definition #16089 - I assume you don't have any good ideas there, so since I don't see a good way to fix paragraph definition myself either, I'll plan to preserve SentenceNav logic and only reconstruct sentences across paragraphs.

@CyrilleB79
Copy link
Collaborator

I have no better idea to detect paragraph accurately, and there will probably a need to do some tests.

Anyway, I think that the algorithm to reconstruct paragraphs for sentence nav feature should be available in the "Paragraph style" parameter of the "Document navigation" settings category. We may then decide if sentence nav feature should follow this parameter to decide of paragraph reconstruction or if the type of paragraph should specifically be forced to the better algorithm for this feature.

@mltony
Copy link
Contributor

mltony commented Feb 21, 2024

@seanbudd, @michaelDCurran
I am currently working on sentence navigation PR and I am running into following problem. I already have an algorithm that detects sentences in array of strings - where each string represents a paragraph. Let's call that array strings. Suppose a sentence starts at strings[0][5]. The problem I am struggling with is how to transform that offset 5 into TextInfo representing start of sentence. I have textInfo representing paragraph. What I figured out is that textInfo.move(UNIT_CHARACTER, 5) will not work because of different encodings in Python vs the app.
For browser textInfos I can use offsets and I can use textUtils.WideStringOffsetConverter to figure out the right offset. For Notepad++ I can I can write a similar utf-8 converter class and also compute offsets.
But what should I do for non-offset text infos? For example, notepad appears to be using UIA textInfo which is not offset based. I've been playing with UIA textInfos and I am slowly figuring out there are many different edge cases. For example in Microsoft Word (I am not targeting word in in my PR, but just want to illustrate quirks of UIA textInfos) - if we have a bulleted list and the cursor is at the beginning of the first item, then textInfos.move(UNIT_CHARACTER, 1) would actually move over 3 characters in textual representation: the bullet character, whitespace and actual first character of list item.
So far my best idea is to have a number of trial-and-error attempts. Something like:

def moveCharacters(paragraphInfo: TextInfo, offset: int):
    paragraphStartInfo = paragraphInfo.copy()
    paragraphStartInfo.collapse()
    info = paragraphStartInfo.copy()
    # first attempt!
    info.move(UNIT_CHARACTER, offset)
    # now compute textInfo span spanning from paragraphStart to info.
    s = span.text
    # now check if len(s) is greater or less than offset
    # and try to move info a few characters left or right, depending on whetehr we overshot or undershot.

I know this appears to be ugly and hacky, but so far I couldn't come up with any better solution. Does anyone have a better idea?

@mltony
Copy link
Contributor

mltony commented Feb 21, 2024

Also cc: @LeonarddeR - since you authored wide string converter - do you have any ideas regarding my question in previous comment?

@CyrilleB79
Copy link
Collaborator

For example in Microsoft Word (I am not targeting word in in my PR, but just want to illustrate quirks of UIA textInfos) - if we have a bulleted list and the cursor is at the beginning of the first item, then textInfos.move(UNIT_CHARACTER, 1) would actually move over 3 characters in textual representation: the bullet character, whitespace and actual first character of list item.

In Word, the caret cannot be placed before the bullet. In a line containing a bullet, the first position is after the bullet and the indentation, before the first text character. Thus the cursor actually moves over one character as expected.

@mltony
Copy link
Contributor

mltony commented Feb 21, 2024

I know. But the problem is the first character in bullet list item stands for 3 characters. E.g., Put your cursor in the beginning of said list item and do:

>>> t=focus.makeTextInfo('caret')
>>>t.move('character', 1, endPoint='end')
1
>>> t.text
'• F'

So moving by 1 in terms of textInfo characters corresponds to moving by 3 in term of Pythonic string indices. My question is this: is there a way to somehow obtain this mapping for UIA text infos? My guess is not, that's why I am planning to do some version of binary search, but would be happy to learn if there is a better way.

michaelDCurran pushed a commit that referenced this issue Apr 8, 2024
This function is needed for both #8518 and #16050.

Summary of the issue:
Suppose we have TextInfo that represents a paragraph of text:
```
> s = paragraphInfo.text
> s
'Hello, world!\r'
```

Suppose that we would like to put the cursor at the first letter of the word 'world'.
That means jumping to index 7:
```
> s[7:]
'world!\r'
```

The problem is that calling paragraphInfo.move(UNIT_CHARACTER, 7, "start") is not guaranteed to achieve desired effect. There are two main reasons for that:
1. In Wide character encoding, some 4-byte unicode characters are represented as two surrogate characters, whereas in pythonic string they would be represented by a single character.
2. In non-offset TextInfos (e.g. UIATextInfo) there is no guarantee on the fact that TextInfos.move(UNIT_CHARACTER, 1)would actually move by exactly 1 character. A good illustration of this is in Microsoft Word with UIA enabled always, the first character of a bullet list item would be represented by three pythonic characters: 
◦ Bullet character "•"
◦ Tab character \t
◦ And the first character of of list item per se.
3. The third problem of TextInfo.move(UNIT_CHARACTER) function is its performance in some implementations. In particular, moving by 10000 characters in Notepad++ takes over a second on a reasonably modern PC. I might not need 

to move by 10000 characters in my upcoming PRs, but I will need to move by a few thousands for sure since for sentence navigation I would need to move within a paragraph and some large paragraphs in typical texts can easily be few thousands characters. I need to find both beginning and end textInfos, and if each operation takes say 200ms, then we'd be wasting almost half a second on just moving by characters. Since there were previous concerns about sentence navigation being not fast enough, II would like to introduce this efficient implementation.
Here is how this can be done efficiently using this PR:

```
> info = paragraphInfo.moveToPythonicOffset(7)
> info.setEndPoint(paragraphInfo, "endToEnd")
> info.text
'world!\r'
```

Description of development approach
1. For general case, I implemented binary-search-like algorithm. I explained it in great detail in the code. Please see def moveToPythonicOffset function in textInfos\__init__.py.
2. I provided optimized implementations for OffsetsTextInfo and CompoundTextInfo.
3. I refactored textUtils.py making it conformant to OOP style. I implemented UTF8OffsetConverter and dummy IdentityOffsetConverter as well as their abstract base class and a function getOffsetConverter that selects correct converter based on encoding. I renamed a couple of methods of WideStringOffsetConverter in order to remove the word wide - as now I would like to use similar methods for UTF8 converter, and it has nothing to do with wide strings.
@mltony mltony linked a pull request Apr 11, 2024 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
p5 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority triaged Has been triaged, issue is waiting for implementation.
Projects
None yet
Development

Successfully merging a pull request may close this issue.