finding macros in latex (was: improve latex-lyx roundtripping (not GSoC))

[Daniel Vainsencher]

Update: I have found that my current examples for "macros that are lost" are all explained (and fixed) by movement between the preamble and the document. That is, at some point they moved above the \begin{document} in the latex, I would guess I did that manually. So this is not strictly a round trip issue, and I think I'm dropping it for now. For now I closed the bug report I'd started putting together. Another issue is whether the location of a command definition before or after the \begin{document} should make a difference to it becoming a macro; I would guess not, but I don't know if anyone cares. As it is parse_text calls is_macro, and Preamble.cpp does not seem to, so that's not a mystery. [wrt defining and using a "math-macro" to annotate lyx-sourced defs in latex] This would be possible, but the main purpose of tex2lyx is not roundtrip, but parsing of real LaTeX documents. Therefore, the complicated parsing of all the different macro definitions would still be needed, even if LyX produced a nicer output. Therefore, this might make the TeX export of LyX look more nice, but it would not help much in tex2lyx. The real power of tex2lyx comes from the fact that it can even translate weirdly formatted LaTeX stuff into native LyX insets. Yes, I see, there's an inherent tradeoff: anything a latex->lyx converter assumes about the latex limits its usefulness, so a generic converter tries to deal with everything. And then the usefulness of conventions is smaller. Anyway, thanks for the help, Daniel On 06/05/2013 10:47 PM, Georg Baum wrote: Richard Heck wrote: On 06/05/2013 05:17 AM, Daniel Vainsencher wrote: Hi again Georg, everyone, Thank you for your answers, I'd missed this one. Since my requests did not recruit any GSOC love, I'm looking into it myself. I hope you can help me get started. Definitely. I may sometimes be slow to respond, but I am glad that you want to work on this, and can hopefully answer your questions. I'll try. I know very little about tex2lyx. Note, fyi, that we are moving toward the release of 2.1, and we are mentoring three GSOC students, so people's attention may be hard to get. You might add this sort of project to our GSOC list. We hope to do this again next year. - Math-macros, if turned into \global\long\def, should be turned back so LyX's beautiful visual editing is restored (can keep the math-macro def in a comment in the latex version and restore it). BTW, when does a math-macro become a \newcommand vs \global\long\def? In old versions, all math formulas in LyX were stored in LaTeX syntax, so that the LyX and LaTeX formats were 100% identical. This changed some time ago for math macros, and some pseudo-LaTeX syntax was invented for LyX (see MathMacroTemplate::write() for details). This was a mistake IMHO, and the best fix for this problem would be to get rid of the pseudo-syntax and restore the 100% LaTeX compatibility. There are also comments in tex2lyx about situations were the conversion to the LyX pseudo-syntax does not work. I don't actually see this any more. What do you have in mind, Georg? I don't find it anymore either. Either my memory was wrong, or it has been removed. Anyway, you can see all the differences between the LyX and LaTeX code in MathMacroTemplate::write(). Either tex2lyx needs to convert the LaTeX version to the LyX version whenever it ouputs a FormulaMacro (which it currently does not do), or the LyX version needs to be ditched, and MathMacroTemplate::write() needs to write the LaTeX version also to the LyX file. As I wrote, I strongly prefer the latter. I'm trying to understand why some latex files (all of which began as exports from lyx with similar macros), when converted back to lyx have math-macros, and some do not. If you could reduce them to some small test cases it would be nice to file a bug report for this (even if you intend to fix it yourself). With a look at tex2lyx I didn't find where this conversion happens. text.cpp and Preamble.cpp seem to use add_known_command and do relevant things, but I might be missing some of the context. Pointers to design documents or hints? There is no design in tex2lyx (ok, this is not 100% true, but a lot of code is simply ad hoc). I take it that macros are supposed to be recognized by the is_macro() routine and then parsed by the parse_macro() routine. The renewcommandx stuff does not get recognized there, but elsewhere. It might be a good start to try to make is_macro() and parse_macro() handle both cases. I note that, both in parse_macro() and in the other place (search text.cpp for "FormulaMacro") there is a test that is done, that decides whether to use ERT or a macro. It is presumably the failue of this test that sometimes messes up re-import. But I don't really understand what this test is doing. Georg wrote it, so he probably does. The test tests whether the examined code is something which can be output by a math macro. I find it somewhat surprising that math-macros are recovered at all from the currently output latex. It seems just defining a \math-macro command and using it would make later identification easier, without being offensive to latex users at all. This would be possible, but the main purpose of tex2lyx is not roundtrip, but parsing of real LaTeX documents. Therefore, the complicated parsing of all the different macro definitions would still be needed, even if LyX produced a nicer output. Therefore, this might make the TeX export of LyX look more nice, but it would not help much in tex2lyx. The real power of tex2lyx comes from the fact that it can even translate weirdly formatted LaTeX stuff into native LyX insets. While we're at it, if we're using newcommandx for optional-argument cases, why not use it always for simplicity? I admit that I only know the most important differences between all the different forms of macro definitions (e.g. \newcommand vs. \def vs. \long\def etc). Stefan fixed a number of bugs with the odl macro code, and I am pretty sure that there are reasons to prefer one or the other form in different cases. In short, I would imagine a solution like the following would be reasonably simple, but really don't know the code well enough. What do you think? 1. Never use \global\long\def, replacing that with just the \newcommandx currently used with optional arguments. 2. In MathMacroTemplate::write() convert every math-macro to a call to a latex command called math-macro that acts like newcommandx, except for carry the bit of information: "this is only used in math, and should be converted to a math-macro". Add an approriate definition of math-macro in every exported latex file. 3. Change tex2lyx to recognize math-macro and convert it accordingly (now should be trivial). It would be nice if we could somehow get Stefan Schimanski's input on this, as he was respnsible for the total rewrite of the macro stuff back in 2008. Yes. Georg