Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix various encoding problems with config 10895 #10900

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

cinex-ru
Copy link

@cinex-ru cinex-ru commented Oct 6, 2022

Issue: #10895

Description:

Fix problems with cruise-config encoding when saving from admin panel and from task and when migration performed.

@cinex-ru
Copy link
Author

cinex-ru commented Oct 6, 2022

Sorry there is a problem to run tests and no much time to figure out the problem.

@chadlwilson
Copy link
Member

chadlwilson commented Oct 8, 2022

The good news: all the (non-Windows) build regression tests pass with this change (annoyingly, it's a bit difficult right now to make these visible to contributors)

image

The bad news: While the intention was always to be UTF-8 only, as I mentioned at #10896 (comment) this is likely a breaking (backward-incompatible) change for those on Windows who might already have characters in their configs which have different representation in their system charset vs UTF-8. Will need to think about whether this is sensible or not, and take me a while to review and add-back appropriate tests to bring confidence there aren't further regressions here.

@cinex-ru
Copy link
Author

cinex-ru commented Oct 8, 2022

The bad news: While the intention was always to be UTF-8 only, as I mentioned at #10896 (comment) this is likely a breaking (backward-incompatible) change for those on Windows who might already have characters in their configs which have different representation in their system charset vs UTF-8. Will need to think about whether this is sensible or not, and take me a while to review and add-back appropriate tests to bring confidence there aren't further regressions here.

May be need enhanced migration? And may be bump version after which config will use utf only?
Btw I can't use non-ASCII at all with previous algorithm.

@chadlwilson
Copy link
Member

Btw I can't use non-ASCII at all with previous algorithm.

Can you be specific about what you mean and what happened? The behaviour may depend on your system's default encoding and which character you are trying to use. What is the default encoding that your JVM is using on Windows?

Also, did you try overriding the encoding per my comment at #10895 (comment) to confirm that works OK?

@stale
Copy link

stale bot commented Oct 22, 2022

This pull request has been automatically marked as stale because it has not had activity in the last 14 days. It will be closed in 7 days if no further activity occurs.
Thank you for your contributions!

@stale stale bot added the stale label Oct 22, 2022
@chadlwilson chadlwilson added no stalebot Don't mark this stale. and removed stale labels Oct 23, 2022
@chadlwilson
Copy link
Member

chadlwilson commented Oct 23, 2022

Hi @cinex-ru - I've done seem testing myself and the behaviour is indeed strange. For me with the system default as

"System properties": {
  "file.encoding": "Cp1252",
  ...

I can save non-ascii into a task command without an error being displayed and initially it gets persisted to config.xml garbled, but when I restart the server somehow it gets 'fixed' and repersisted as UTF-8 correctly.

I don't get an error when, say editing task commands, but do via the admin config editor, as you describe.

Can you tell me what the system file.encoding is for you so I can understand your specific case? It's included in the system properties list if you go to /go/api/support after logging in.

This will help me understand the various setups people might be in and whether changing this might might things worse for them, or whether I need to (and can) do some fix-up migration.

@chadlwilson chadlwilson linked an issue Oct 23, 2022 that may be closed by this pull request
@chadlwilson chadlwilson marked this pull request as draft October 18, 2023 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Various problems with cruise-config encoding on Windows
2 participants