Add ICU sentence segmentation to OffsetsTextInfo

[LeonarddeR]

Link to issue number:

Part of #18901.
Follow up for #20602.

Summary of the issue:

Sentence navigation only works in Microsoft Word and Outlook, which have native sentence support. Browse mode in virtual buffers uses TextInfos based on OffsetsTextInfo, and its _getSentenceOffsets raised NotImplementedError. Working sentence navigation in those documents therefore needs a sentence backend first.

Description of user facing changes:

No direct changes. This PR is the necessary building block: programmatic sentence navigation is now possible in cursor managers and in TextInfos based on offsets. The next PR in the stack routes the browse mode sentence gestures to it.

Description of developer facing changes:

winBindings.icu gains UBRK.SENTENCE. textUtils.icu gains calculateSentenceOffsets. OffsetsTextInfo._getSentenceOffsets is implemented, so UNIT_SENTENCE becomes available on all TextInfos derived from it.

Description of development approach:

Sentence boundaries come from the Windows built-in ICU BreakIterator using UAX#29 default rules. Segmentation runs over the containing paragraph, so sentences can span lines. Results are rebased by the paragraph start offset. The root locale is used: UAX#29 sentence boundaries are driven by the language-neutral Sentence_Break property, and locale only adds abbreviation tailoring. A fixed locale also keeps boundaries a deterministic function of position, which the move/expand tiling invariant requires. When ICU is unavailable (Windows before version 1703), the method keeps raising NotImplementedError.

Testing strategy:

New unit tests in test_sentenceSegIcu.py cover the calculateSentenceOffsets primitive (English, Japanese, abbreviation splits, surrogate pairs, containment) and the OffsetsTextInfo integration, including forward and backward tiling across a block-paragraph fixture.

Known issues with pull request:

Abbreviations such as "Dr." end a sentence. Locale-specific tailoring could address this later.
documentNavigation.sentenceHelper still uses its own logic to detect sentence boundaries.

Code Review Checklist:

• Documentation:
  • Change log entry
  • User Documentation
  • Developer / Technical Documentation
  • Context sensitive help for GUI changes
• Testing:
  • Unit tests
  • System (end to end) tests
  • Manual testing
• UX of all users considered:
  • Speech
  • Braille
  • Low Vision
  • Different web browsers
  • Localization in other languages / culture than English
• API is compatible with existing add-ons.
• Security precautions taken.

[Copilot]

Pull request overview

This PR adds ICU (Windows built-in) sentence segmentation support for OffsetsTextInfo, enabling UNIT_SENTENCE navigation for all TextInfos derived from offsets-based implementations (e.g. virtual buffers), while preserving the existing NotImplementedError behavior when ICU is unavailable.

Changes:

• Add ICU sentence break support (UBRK.SENTENCE) and a new textUtils.icu.calculateSentenceOffsets primitive.
• Implement OffsetsTextInfo._getSentenceOffsets using ICU sentence segmentation over paragraph text and rebase to document offsets.
• Factor out shared ICU offset/encoding conversion via textUtils.icu.calculateOffsetsForEncoding, and update the ICU word segmentation strategy to use it.

Reviewed changes

Copilot reviewed 7 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/test_textUtils/test_wordSegIcu.py	Adds focused unit tests for IcuWordSegmentationStrategy selection and primitive invocation.
tests/unit/test_textUtils/test_wordSeg.py	Adds broader unit tests for word segmentation initialization and related utilities.
tests/unit/test_textUtils/test_sentenceSegIcu.py	Adds unit tests for ICU sentence segmentation primitive and OffsetsTextInfo tiling/iteration behavior.
tests/unit/test_textUtils/test_offsetConverters.py	Adds/extends unit tests validating offset converters (incl. surrogate-pair behavior).
tests/unit/test_textUtils/test_hyphenation.py	Adds unit tests for textUtils.hyphenation behavior for known/unknown languages.
tests/unit/test_textUtils/test_backendComparison.py	Switches ICU-availability skipping to shared skipIfNoICU.
tests/unit/test_textUtils/init.py	Introduces skipIfNoICU helper for ICU-dependent tests.
source/winBindings/icu.py	Adds UBRK.SENTENCE enum member for ICU sentence break iteration.
source/textUtils/icu.py	Adds sentence segmentation primitive and shared offset conversion helper; refactors word segmentation to reuse _containingSegment.
source/textUtils/_wordSeg/wordSegStrategy.py	Updates ICU word segmentation strategy to use the shared encoding conversion helper.
source/textInfos/offsets.py	Implements _getSentenceOffsets using ICU sentence segmentation over paragraph scope.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

[LeonarddeR]

Closing until #20602 is merged first

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.