Vim clone left "dirty" after make. src/po/ru.cp1251.po is modified.

[errael]

Steps to reproduce

I occasionally see this, and I think I've seen mention of this on the dev list; it might always be the same file; this morning I happened to catch it just after it happened. I also built yesterday morning. It doesn't happen too often (the revert file I see is from January), I haven't looked at makefiles, I don't know what dependency is triggering.

If anyone knows something specific to check...

After pull, make reconfig, the console shows the make ending as:

...
OLD_PO_FILE_INPUT=yes msgfmt -v -o vi.mo vi.po
1410 translated messages.
OLD_PO_FILE_INPUT=yes msgfmt -v -o zh_CN.UTF-8.mo zh_CN.UTF-8.po
2800 translated messages.
OLD_PO_FILE_INPUT=yes msgfmt -v -o zh_TW.UTF-8.mo zh_TW.UTF-8.po
1410 translated messages.
make[2]: Leaving directory '/ref/vim/src/po'
make[2]: Entering directory '/ref/vim/src/po'
OLD_PO_FILE_INPUT=yes msgfmt -v -o cs.cp1250.mo cs.cp1250.po
1272 translated messages.
...
OLD_PO_FILE_INPUT=yes msgfmt -v -o pl.cp1250.mo pl.cp1250.po
1858 translated messages.
rm -f ru.cp1251.po
iconv -f UTF-8 -t CP1251 ru.po | \
	LANG=C sed -e 's/charset=[uU][tT][fF]-8/charset=CP1251/' \
		-e 's/# Original translations/# Generated from ru.po, DO NOT EDIT/' \
		> ru.cp1251.po
OLD_PO_FILE_INPUT=yes msgfmt -v -o ru.cp1251.mo ru.cp1251.po
2979 translated messages.
...
make[2]: Leaving directory '/ref/vim/src/po'
make[1]: Leaving directory '/ref/vim/src'
err@harmony:/ref/vim/src
$ hg st
M src/po/ru.cp1251.po

The diff seems to be a lot of formatting. Here's an excerpt, as seen in vim.

-"W17: Äëÿ ðàáîòû ñ àðàáñêîé ïèñüìåííîñòüþ ââåäèòå êîìàíäó `:set "
-"encoding=utf-8`"
+"W17: Äëÿ ðàáîòû ñ àðàáñêîé ïèñüìåííîñòüþ ââåäèòå êîìàíäó `:set encoding=utf-8`"

It interesting that the wrap, as shown in vim above, is happening right around column 80. With all these formatting changes it's hard to see if there are any real changes.

Using gvim -d to look at the checked in file and the it looks like the only changes are around the linebreak and that the translation itself has no changes.

Expected behaviour

No modified checked-in file.

Version of Vim

9.1.0299

Environment

ubuntu/gtk

Logs and stack traces

No response

[RestorerZ]

Hmm, interesting! Do you have this happen sometimes, and always on this file?
I'm going to try building in MS Windows now - it's fine.
I'll check what's wrong with this file. If there is something wrong with it.

Can you please tell me what your gettext is? Version and so on...

[errael]

Hmm, interesting! Do you have this happen sometimes,

I see it every month or two

and always on this file?

Not sure, maybe.

For reference, take a look at https://groups.google.com/g/vim_dev/c/Gyqie6FQM4s/m/W2yZLzsuAwAJ from December, other people see this once and a while.

I'm going to try building in MS Windows now - it's fine. I'll check what's wrong with this file. If there is something wrong with it.

Can you please tell me what your gettext is? Version and so on...

I built with current source this morning.

[RestorerZ]

Okay, thank you!
I have a rough idea of what it could be related to.
From the dates you gave, it seems to happen when translations are updated periodically.
Okay, let's try to figure it out.

[RestorerZ]

Based on this line OLD_PO_FILE_INPUT=yes msgfmt -v -o vi.mo vi.po, you are using the "gettext" version before 0.22.

[julio-b]

I have also been bothered by this and i ended up just compiling with --disable-nls.
But here is what is happening and how to trigger this bug reliably.

I will take as an example commit 8fcc129 because it is the more recent one, but it could be any commit that updates the translation files.

To reproduce:

1. run this one-liner until you see that src/po/ru.po is newer than src/po/ru.cp1251.po
   $ git checkout 8fcc12977~ && ls -la --time-style=full-iso src/po/ru*po && git checkout master && ls -la --time-style=full-iso src/po/ru*po
   Example output on my fourth run:

HEAD is now at c9ec20d94 runtime(doc): Update documentation
-rw-r--r-- 1 ju ju 410K 2024-04-10 22:43:04.013415053 +0300 src/po/ru.cp1251.po
-rw-r--r-- 1 ju ju 539K 2024-04-10 22:43:04.016748378 +0300 src/po/ru.po
Previous HEAD position was c9ec20d94 runtime(doc): Update documentation
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
-rw-r--r-- 1 ju ju 413K 2024-04-10 22:43:04.050081632 +0300 src/po/ru.cp1251.po
-rw-r--r-- 1 ju ju 543K 2024-04-10 22:43:04.053414958 +0300 src/po/ru.po

notice how ru.po timestamp is newer by a few MS

2. run $ make

Explanation:

• You pull a commit that updates some translation files e.g src/po/ru.po and src/po/ru.cp1251.po
• when this commit is applied, the timestamp of ru.po could end up by a few milliseconds newer than ru.cp1251.po. This can happen randomly and it's unpredictable. It's more noticeable when the cpu is under heavy load.
• now make detects that ru.cp1251.po file is older and triggers the iconv rule here
• iconv is not really consistent between systems, so the results you get may be different. I think it's behavior depends on LC_ environmental variables and there are different implementations for windows.

This is possible for any translation file that uses iconv, listed in here.

[errael]

Based on this line OLD_PO_FILE_INPUT=yes msgfmt -v -o vi.mo vi.po, you are using the "gettext" version before 0.22.

I didn't know what you meant by gettext, now I see. And you are correct; I guess a lot of people are still using that version.

$ gettext --version
gettext (GNU gettext-runtime) 0.21

[RestorerZ]

• iconv is not really consistent between systems, so the results you get may be different.

Yeah, that's what I was thinking. But not "iconv", but on "msgmerge" and text formatting.

• the timestamp of ru.po could end up by a few milliseconds

In my opinion, this is too small a difference to handle the rule.

Anyway, according to the documentation, for "nmake.exe " the accuracy is a 2‐second interval.

Thank you for your explanations.

[errael]

BTW, I use popOS, I update regularly. So I guess their distribution hasn't incorporated the 0.22 version.

[RestorerZ]

I switched to version 0.22 and use it when preparing translations. Anyway, I'll try to figure out what's going on.

[benknoble]

1. run this one-liner until you see that src/po/ru.po is newer than src/po/ru.cp1251.po
   $ git checkout 8fcc12977~ && ls -la --time-style=full-iso src/po/ru*po && git checkout master && ls -la --time-style=full-iso src/po/ru*po

Probably it would have been simpler to

git checkout <REF>
touch src/po/ru.po

to force the “newer” timestamp (for reproduction; your method is nice to explain how Git might leave the timestamps in an unexpected state).

[julio-b]

Probably it would have been simpler to

git checkout <REF>
touch src/po/ru.po

to force the “newer” timestamp (for reproduction; your method is nice to explain how Git might leave the timestamps in an unexpected state).

Right, I wanted to show how this could be triggered just by using a simple git pull, without any special modification on the .po file itself. But for a fast reproduction a simple touch ru.po is all you need!

I debugged this a bit more and now i think know for sure what is happening:

• At some point, one of those faulty .po files was generated with msgconv (instead of iconv)
• msgconv, by default, will hard wrap lines at column 80
• So now every time that the iconv rule above is triggered (because of the different timestamp that i explained) the output is not wrapped at column 80

You can verify that by using $ msgconv -t CP1251 -o /tmp/ru.c1251.po ru.po and compare it with the current version of this file $ diff /tmp/ru.c1251.po src/po/ru.c1251.po. Almost identical, minus a single line that is changed by a simple sed script.

Right now msgconv is used only on windows, eg. by src/po/Make_mvc.mak.

Thus there are three fixes required:

1. Make the output of iconv and msconv equal. Probably by using msgconv --no-wrap
2. Fix the files that were generated with msgconv
3. And finally find a way for make to not trigger the iconv rule randomly. For gnu version of make this is done with .LOW_RESOLUTION_TIME, but i don't know if it is portable for other versions.

(3) is technically not required if (1,2) are fixed because the output will always be the same, but still worth fixing it so we don't waste time regenerating those files.

I hope this helps!

[k-takata]

If make ru is executed in src/po/ and cleanup.vim is applied, ru.po will be wrapped in 80 columns.
Then, make ru.cp1251.po is executed, ru.cp1251.po will be also wrapped in 80 columns.

All of (or most of) the other po files are wrapped in 80 columns. So, I think ru*.po should be also wrapped in 80 columns.

[chrisbra]

Hm, so we should add a check to check.vim.

Something like this:

" Check that all lines are no longer than 80 chars
let overlong = search('\%>80v', 'n')
if overlong > 0
  echomsg "Lines should be wrapped at 80 columns"
  if error == 0
    let error = overlong
  endif
endif

Interestingly, I checked this against de.po and got a match. So there is something I need to fix for German as well.