Windows-1252 encoding for HTML numeric entities #191
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for that range, translate to Windows 1252 encoding; re-throw illegal argument exception with input if restrictions are violated add test which tests numeric entities that are improperly encoded in cp-1252 and including code points before and after range [128, 159]. algorithm for numeric entity applies in very restrictive conditions: it must be in the range where ISO 8859-1 and Windows-1252 decohere, it must be a non-hex numeric entity (this is to avoid tripping one of StringEscapeUtilsTests), it must also not be an invalid Windows-1252 point in that range.
invalid points shouldn't be modifiable
stopped creating a new decoder every time it is invoked; use static decoder instead. renamed to cp-1252 to save characters; fixed documentation in the constructor. also reformatted it to match what i think was intended by original author
|
There is a test failing... :-( |
kinow
reviewed
Dec 13, 2020
src/main/java/org/apache/commons/text/translate/NumericEntityUnescaper.java
Outdated
Show resolved
Hide resolved
capitalise the unicode escapes in the test because that seems to be the prevailing code style
|
What does the HTML spec say? That's what should be followed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, if given text like so:
StringEscapeUtilscurrently returns the corresponding Unicode characters for points 128, 133, 147, and 148, which are bunch of obscure basically-never-used control characters that display as spaces. Those code points are, however, used more often in Windows-1252 encoding, corresponding to characters like € and ™.I've changed
NumericEntityUnescaperto treat HTML numeric entities corresponding to valid CP-1252 code points between 128 and 159 (inclusive) as CP-1252 characters and decode them to the corresponding punctuation marks etc instead of the obscure Unicode control characters.