Dear Georg,
On 2015-09-09, Georg Baum wrote: > Guenter Milde wrote: ... >> I am pretty sure the unicodesymbols fix is correct, I am not sure whether >> the round-trip result is as it should be. > Both are correct. Since the patch in http://www.lyx.org/trac/ticket/9764 > does not revert the introduction of \b for 0x0331. and the tests are > currently failing, it is expected that they still fail after applying the > patch. > Since it is clear now that the change 268bd00 which made the tests fail was > correct, we should proceed in two steps: > 1) run "make updatetests" in a clean tree and submit the result with a log > referring to 268bd00. This makes it clear that the changed test cases are > solely caused by that change. > 2) submit the patch in ticket 9764. This would then proove that the patch > does not change anything in tex2lyx. I propose a different approach: Motivation: Commit 268bd00 introduced an ambiguity because the "before" state had a wrong mapping for the "MINUS BELOW". The patch from #9764 removes the ambiguity (\b as representation of two Unicode points). Both, commit 268bd00 and patch from #9764 do the right thing regarding "lib/unicodesymbols". Therefore, I would not change the files with expected test results in the intermediate state, but instead 1) Apply the patch in ticket #9764 (i.e. complete the change from commit 268bd00 and reach an unambiguous mapping of "combining line below characters"). 2) Test with a clean/unchanged test-dir. After that, we can investigate the test diff and decide on the correct way forward: change the test-expectations or change the LyX code: Actually, (as we solved the ambiguity) the result of a round-trip should remain unchanged: tex->lyx->tex \textsubbar -> 0x0320 -> \textsubbar \b -> 0x0331 -> \b \underbar -> 0x0332 -> \underbar lyx->tex->lyx 0x0320 -> \textsubbar -> 0x0320 # COMBINING MINUS SIGN BELOW 0x0331 -> \b -> 0x0331 # COMBINING MACRON BELOW 0x0332 -> \underbar -> 0x0332 # COMBINING LOW LINE Results: >> Interestingly, despite the "tests passed" message, git reports changed >> files: ... > It is quite easy: "make alltests" does never change the test references. ... > Since you have changed test references in your tree (as shown by git diff), > it means that you did either call "make updatetests", or you did run tex2lyx > manually. If you did neither of that then it would be a mystery in my eyes. It seems I must have a different setup: * Reverted the changes to "test-insets-basics.lyx.lyx" and "test-insets.lyx.lyx". * Run `make alltests` console output of the tex2lyx test: cd tex2lyx; make alltests-recursive make[2]: Entering directory '/usr/local/src/lyx/src/tex2lyx' Executing: ./tex2lyx -roundtrip -f ./test/test.ltx Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/test.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/algo2e.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/algo2e.lyx.lyx Ignoring options 'boxed' of package algorithm2e. Executing: ./tex2lyx -roundtrip -f ./test/box-color-size-space-align.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/box-color-size-space-align.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/CJK.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/CJK.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/CJKutf8.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/CJKutf8.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/test-insets.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/test-insets.lyx.lyx Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/DummyDocument.lyx.lyx Warning: Could not find included file 'Dummy Document.tex'. Warning: Could not find included file 'Dummy Document.tex'. Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/Dummy~Document.lyx.lyx Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/Dummy~Document.lyx.lyx Warning: Could not find included file 'Dummy"Document.tex'. Warning: Could not find included file 'foo'. Warning: Could not find included file 'foo'. output_latex.cpp (1190): Error in latexParagraphs: You should not mix title layouts with normal ones. Datei überschreiben? ---------------------------------------- Die Datei /usr/local/src/lyx/src/tex2lyx/test/xfigtest.pdftex_t existiert bereits. Möchten Sie diese Datei überschreiben? Assuming answer is &Nicht überschreiben &Überschreiben &Alle überschreiben Export &abbrechen Executing: ./tex2lyx -roundtrip -f ./test/test-insets-basic.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/test-insets-basic.lyx.lyx Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/DummyDocument.lyx.lyx Warning: Could not find included file 'Dummy Document.tex'. Warning: Could not find included file 'Dummy Document.tex'. Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/Dummy~Document.lyx.lyx Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/Dummy~Document.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/test-memoir.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/test-memoir.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/test-modules.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/test-modules.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/test-refstyle-theorems.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/test-refstyle-theorems.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/test-scr.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/test-scr.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/test-structure.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/test-structure.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/verbatim.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/verbatim.lyx.lyx Executing: ./tex2lyx -roundtrip -f ./test/XeTeX-polyglossia.tex Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/XeTeX-polyglossia.lyx.lyx ===================== tex2lyx tests passed. ===================== make[2]: Leaving directory '/usr/local/src/lyx/src/tex2lyx' make[1]: Leaving directory '/usr/local/src/lyx/src' I never run `make updatetests`, still there are many messages about overwriting files. Many of these may be generated from an earlier run of `make alltests`, however, there are also Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/test-insets.lyx.lyx Overwriting existing file /usr/local/src/lyx/src/tex2lyx/test/test-insets-basic.lyx.lyx and in fact, both files are changed and the git-diff shows that the MINUS SIGN BELOW is changed to a MACRON BELOW: - e̹ e̜ e̟ e̠ e̠ ë e̽ e̩ e̩ e̯ e˞e + e̹ e̜ e̟ e̱ e̱ ë e̽ e̩ e̩ e̯ e˞e I investigated furter and found: * in test-insets.tex, there is (added in b25016b6 by Uwe) \textsubplus{e} \=*{e} \b{e} \"{e} In addition to the IPA accent "minus below", there is also the standard TeX accent "macron below" (which is not an IPA character/command and actually does not belong there (testing the correct differentiation between them is good but maybe better done elsewhere)). * With the old (wrong) mappings, both commands translate to COMBINING MINUS SIGN BELOW and in fact, this is what is found in test-insets.lyx.lyx currently on Git. ! Ambiguity introduced by by the mapping of MINUS SIGN BELOW to \b in \unicodesymbols but now fixed in the test examples! * Opening the unchanged Git-version of test-insets.lyx.lyx with LyX, I see in the source panel, \textsubplus{e} \=*{e} \=*{e} ë Both instances are converted to the TIPA shortcut, which seems to take precedence over the LICR macro ("textcommand"). * A round-trip converts \=* to MACRON BELOW, which is wrong. The following patch should fix this: diff --git a/src/tex2lyx/text.cpp b/src/tex2lyx/text.cpp index ffeac41..3f86497 100644 --- a/src/tex2lyx/text.cpp +++ b/src/tex2lyx/text.cpp @@ -314,7 +314,7 @@ char const * const known_tipa_marks[] = {"textsubwedge", "textsubumlaut", "textovercross", "textsubarch", "textsuperimposetilde", "textraising", "textlowering", "textadvancing", "textretracting", "textdoublegrave", "texthighrise", "textlowrise", "textrisefall", "textsyllabic", -"textsubring", 0}; +"textsubring", "textsubbar", 0}; /// TIPA tones that need special handling char const * const known_tones[] = {"15", "51", "45", "12", "454", 0}; @@ -4787,7 +4787,7 @@ void parse_text(Parser & p, ostream & os, unsigned flags, bool outer, // fromLaTeXCommand() does not yet // recognize tipa short cuts p.get_token(); - name = "\\b"; + name = "\\textsubbar"; } else if (name == "\\textdoublevertline") { // FIXME: This is not correct, // \textvertline is higher than \textbardbl And indeed, with these changes `make alltests` writes a new "test-insets.lyx.lyx" file that has in the relevant TIPA inset the correct accents e̟ e̠ e̱ ë that map to: \textsubplus{e} \=*{e} \b{e} ë For improved clarity, I propose to change the source: --- a/src/tex2lyx/test/test-insets.tex +++ b/src/tex2lyx/test/test-insets.tex @@ -586,7 +586,7 @@ e\textsuperscript{h} \textsublhalfring{e} \textsubplus{e} \=*{e} -\b{e} +(don't mix up with \b{e}) \"{e} \textovercross{e} \s{e} --- a/src/tex2lyx/test/test-insets.tex +++ b/src/tex2lyx/test/test-insets.tex @@ -586,7 +586,7 @@ e\textsuperscript{h} \textsublhalfring{e} \textsubplus{e} \=*{e} -\b{e} +(don't mix up with \b{e}) \"{e} \textovercross{e} \s{e} I might commit these changes if I were allowed to (without them being posted to the csv list because I cannot subscribe). Otherwise I can add them to the patch at ticket #9764 for someone else to review and apply. Günter Günter
