Hi!First of all, I'm glad to see this patch you're working on; unfortunately I haven't had a chance to look at it, maybe over the weekend I'll get to it.
BTW, you are aware of bug 1820, right?Yes, but I did not try to fix that. This patch addresses only the encoding issues, bug 1820 is a language problem. After looking at LyXFont::latexWriteStartChanges I was not surprised that this bug exists. It is most probably a consequence of using a "paragraph language" as opposed to the language of the character in question. IMHO this "paragraph language" is not useful in general, and we should rather use the languages of the single characters we are dealing with.
Regarding 1820 --- I started looking at it a while ago, and I think that the problem stems from this: in TeXOnePar, previous_language is set to the language of the previous paragraph, or to the document language if this is the first paragraph. Then, if the new paragraph's language is different from the previous language, the language command is inserted, otherwise it isn't.
The problem is that inside an inset (such as a footnote), the paragraphs are numbered from 0 again. So the first paragraph in the footnote is identified as the "first", and therefore the previous_language is set to the document language, namely English (say it's an English document).
However, from latex's point of view, the current language in this case is not the document language, but the language which was set when entering the inset. So in our example, if the paragraph in which the footnote appears is in Hebrew, then as far as latex is concerned, the current language of the first paragraph inside the footnote is still Hebrew. However, LyX decided that the current language is English. Then when the first word is in English, LyX will decide that that's the current language, and therefore no language command will be inserted. Latex, however, knows that the current language is Hebrew, and will output the English word as if it were Hebrew.
I'm attaching a file which demonstrates the above: it contains all four combinations of a Hebrew/English footnote inside a Hebrew/English paragraph. Just switching the document's language will affect which footnotes appear correctly, and which do not: if the document language is English, the problem is for English in Hebrew; if the document language is Hebrew, the problem is for Hebrew in English. (For those of you who don't read Hebrew --- the footnotes which are in Hebrew should have the same word which appears in the Hebrew paragraphs --- which are always displayed correctly).
Two notes: (1) The attached file is actually from LyX 1.3, because I've played around with this a bit in my svn of 1.5, and may have done some damage of my own. (2) In some situations when the wrong language is chosen, encoding problems may result, in which case output can't even be generated; in this case, changing the encoding of the document between default/auto/cp1255 should help (although right now I seem to be having some trouble regardless of the encoding in r17355 --- but again, this is with some changes of my own, I'm not sure if they're the cause or not; in LyX 1.3, changing the encoding solves the problem).
Finally, regarding possible solutions:*) first of all, the patches you've been submitting in this thread, which I haven't looked at yet, certainly may have some impact on this; I haven't tested it at all yet. *) Secondly, it seems to me that something that's missing in the current latex output mechanism, is some sort of stack, which would keep track of the current language, and perhaps other things like that, while going deeper and deeper into insets, paragraphs, etc. Such a stack could either be stored explicitly, or perhaps implicitly by using the call stack itself (i.e., TeXOnePar->...->InsetFoot->...->TeXOnePar again, ...) --- though I think you've been going in the opposite direction, of getting rid of parameters while calling these functions, and I certainly see the sense in that. However, specifically for these kinds of issues, perhaps such parameters should be passed around. The problem with this is that they would need to be passed around even to functions which would not use them at all --- such as latexParagraphs, which is in the middle there between the two TeXOnePars --- beyond just passing them deeper. *) I tried to naively solve this by just printing the language commands all the time. this actually does solve the problem, but: (a) that's precisely what you're trying to fix in this thread, unneeded language/encoding commands, and (b) this caused a problem specifically inside tables, where the language commands, which get printed in every cell (which is, I guess, an inset?), cause a newline to be printed. *) So I next thought of perhaps printing the language commands always only if I'm inside a footnote. However, inside TeXOnePar (which is where the language command would get printed), I don't know how I can know what kind of inset I'm in. or indeed if I'm inside an inset at all? This is also the kind of information which I think maybe should be passed around on the "stack" I mentioned above.
Hope this helps! Dov
footnotes.lyx
Description: application/lyx
