#26645: Errors when running i18n makemessages tests on Windows
--------------------------------------+------------------------------------
Reporter: ramiro | Owner: ramiro
Type: Bug | Status: assigned
Component: Internationalization | Version: master
Severity: Normal | Resolution:
Keywords: windows makemessages | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------
Comment (by ramiro):
AFAICS what happens on Windows is that by not passing `universal_newlines`
to `subprocess.Popen()` as we are doing since
fa08d27fb714534670b431fde0cd04a17d637585 the in -memory representation of
text content we capture from the standard output of `xgettext(1)`,
`msgmerge(1)`, etc. on this platform contains native line ending sequences
and not simply `\n`.
(All examples below are running the
`i18n.test_extraction.BasicExtractorTests.test_blocktrans_trimmed` test
case)
Blob of text that reaches
`django.core.management.commands.makemessages.write_pot_file()`:
{{{
(Pdb) msgs
u'# SOME DESCRIPTIVE TITLE.\r\n# Copyright (C) YEAR THE PACKAGE\'S
COPYRIGHT HOLDER\r\n# This file is distributed under the same license as
the PACKAGE package.\r\n# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.\r\n#\r\n#,
fuzzy\r\nmsgid ""\r\nmsgstr ""\r\n
}}}
Note the native `\r\n` line ending sequences.
Then, the temporary `.pot` file
(`test\i18n\test_extraction\commands\locale\django.pot`) is written. As we
open it in text mode (we can't open it in either binary or universal
newlines modes because we are passing the ''encoding'' parameter to
``io.open()``) the `\n` sequences are replaced with `\r\n` ones resulting
in lines separated by `\r\r\n`, e.g. (output of `hd(1)` on Linux,
`django.pot` transferred from the Windows system):
{{{
$ hd -n 240 django.pot
00000000 23 20 53 4f 4d 45 20 44 45 53 43 52 49 50 54 49 |# SOME
DESCRIPTI|
00000010 56 45 20 54 49 54 4c 45 2e 0d 0d 0a 23 20 43 6f |VE TITLE....#
Co|
00000020 70 79 72 69 67 68 74 20 28 43 29 20 59 45 41 52 |pyright (C)
YEAR|
00000030 20 54 48 45 20 50 41 43 4b 41 47 45 27 53 20 43 | THE
PACKAGE'S C|
00000040 4f 50 59 52 49 47 48 54 20 48 4f 4c 44 45 52 0d |OPYRIGHT
HOLDER.|
00000050 0d 0a 23 20 54 68 69 73 20 66 69 6c 65 20 69 73 |..# This file
is|
00000060 20 64 69 73 74 72 69 62 75 74 65 64 20 75 6e 64 | distributed
und|
00000070 65 72 20 74 68 65 20 73 61 6d 65 20 6c 69 63 65 |er the same
lice|
00000080 6e 73 65 20 61 73 20 74 68 65 20 50 41 43 4b 41 |nse as the
PACKA|
00000090 47 45 20 70 61 63 6b 61 67 65 2e 0d 0d 0a 23 20 |GE
package....# |
000000a0 46 49 52 53 54 20 41 55 54 48 4f 52 20 3c 45 4d |FIRST AUTHOR
<EM|
000000b0 41 49 4c 40 41 44 44 52 45 53 53 3e 2c 20 59 45 |AIL@ADDRESS>,
YE|
000000c0 41 52 2e 0d 0d 0a 23 0d 0d 0a 23 2c 20 66 75 7a |AR....#...#,
fuz|
000000d0 7a 79 0d 0d 0a 6d 73 67 69 64 20 22 22 0d 0d 0a |zy...msgid
""...|
000000e0 6d 73 67 73 74 72 20 22 22 0d 0d 0a 22 50 72 6f |msgstr
""..."Pro|
}}}
From this stage of the message extraction process onwards:
1. With each update of the on-disk ancillary temporary POT file additional
`\r` chars are accumulated. In the mentioned test case it ends with
`\r\r\r\r\n` line separators.
2. As further `popen_wrapper` calls are performed (i.e. to call
`msgmerge(1)` or `msguniq(1)`), problem 1 above gets carried to the final
`.po` file(s) and somehow results in multiple `\r\n` inserted:
This is the final
`test\i18n\test_extraction\commands\locale\de\LC_MESSAGES\django.po`:
{{{
$ hd -n 312 django.po
00000000 23 20 53 4f 4d 45 20 44 45 53 43 52 49 50 54 49 |# SOME
DESCRIPTI|
00000010 56 45 20 54 49 54 4c 45 2e 0d 0a 22 50 6c 75 72 |VE
TITLE..."Plur|
00000020 61 6c 2d 46 6f 72 6d 73 3a 20 6e 70 6c 75 72 61 |al-Forms:
nplura|
00000030 6c 73 3d 32 3b 20 70 6c 75 72 61 6c 3d 28 6e 20 |ls=2;
plural=(n |
00000040 21 3d 20 31 29 3b 5c 6e 22 0d 0a 0d 0a 0d 0a 0d |!=
1);\n".......|
00000050 0a 23 20 43 6f 70 79 72 69 67 68 74 20 28 43 29 |.# Copyright
(C)|
00000060 20 59 45 41 52 20 54 48 45 20 50 41 43 4b 41 47 | YEAR THE
PACKAG|
00000070 45 27 53 20 43 4f 50 59 52 49 47 48 54 20 48 4f |E'S COPYRIGHT
HO|
00000080 4c 44 45 52 0d 0a 0d 0a 0d 0a 0d 0a 23 20 54 68 |LDER........#
Th|
00000090 69 73 20 66 69 6c 65 20 69 73 20 64 69 73 74 72 |is file is
distr|
000000a0 69 62 75 74 65 64 20 75 6e 64 65 72 20 74 68 65 |ibuted under
the|
000000b0 20 73 61 6d 65 20 6c 69 63 65 6e 73 65 20 61 73 | same license
as|
000000c0 20 74 68 65 20 50 41 43 4b 41 47 45 20 70 61 63 | the PACKAGE
pac|
000000d0 6b 61 67 65 2e 0d 0a 0d 0a 0d 0a 0d 0a 23 20 46
|kage.........# F|
000000e0 49 52 53 54 20 41 55 54 48 4f 52 20 3c 45 4d 41 |IRST AUTHOR
<EMA|
000000f0 49 4c 40 41 44 44 52 45 53 53 3e 2c 20 59 45 41 |IL@ADDRESS>,
YEA|
00000100 52 2e 0d 0a 0d 0a 0d 0a 0d 0a 23 20 0d 0a 0d 0a |R.........#
....|
00000110 0d 0a 0d 0a 23 2c 20 66 75 7a 7a 79 0d 0a 0d 0a |....#,
fuzzy....|
00000120 6d 73 67 69 64 20 22 22 0d 0a 0d 0a 6d 73 67 73 |msgid
""....msgs|
00000130 74 72 20 22 22 0d 0a 0d |tr ""...|
}}}
3. The newly added newlines confuse `copy_plural_forms()` (see above) and
the code in charge setting of the `charset` value on the PO header to
`utf-8`. '''This is what causes the symptom reported by this ticket''':
`django.po:2:47: syntax error -- msgmerge: found 1 fatal error`
#25667 was about the inability to handle error output of `msgfmt(1)` which
is actually invoked from `compilemessages` buf the removal of the
`gettext_popen_wrapper` function (which was the one passing
`universal_newlines` to `subprocess.Popen()`) introduced this unintended
effect.
One possible path would be to re-introduce `universal_newlines` only for
the handling of the gettext tools used by `makemessages` which, unlike the
`compilemenssages`/`msgmft(1)`, do actually spew POT and PO content in-
band; but I'm afraid to do that given the difficulties we've experienced
on this front, see #20271, 6fb9dee470d57882e378247fd2706d5f9867b5f9 and
57202a112a966593857725071ecd652a87c157fb for some examples.
I'm working on an alternate approach. Will update this ticket with the
link to a PR.
--
Ticket URL: <https://code.djangoproject.com/ticket/26645#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/064.5e66df77b375a111735d815b37c2c809%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
