Re: new procedure with GitLab CI

[Masamichi Hosoda]

> I can't tell you for sure either. It could be that the fork at 
> https://gitlab.com/trueroad/lilypond has Pipelines enabled, but only
> visible to project members. Hosoda-san, could you check this in your
> project? The drop-down is at Settings > General > Visibility ... >
> Pipelines. I'm guessing here, could be something else...

I've changed the Pipelines settings
from `Only Project Members' to `Everyone With Access'.

Re: Huge PDF doc files

[Masamichi Hosoda]

> The very problem seems to be that the PDF snippets in 'lybook-db' are
> no longer created with `-dgs-never-embed-fonts=#t`.

It seems `-dgs-never-embed-fonts=#t` does not work in current LilyPond.
I've created merge request !218 that fixes it.

Re: NR pdf is larger with current git by 15%

[Masamichi Hosoda]

> your fix for gs works fine; the resulting PDFs are small again.
> Thanks a lot!
> 
> BUT: They are not as small as previously.  For example, with commit
> 21a20de3, the NR has 10 pages more and is now 8.3MByte (I've tested
> compilation with both gs 9.21 and 9.52) instead of 7.1MByte (with gs
> 9.21, produced April 20th using commit f86f7705).  As far as I can
> see, my used xetex binary is the same, as is `texinfo.tex` (version
> 2019-02-16.14), and the included fonts.
> 
> Do you have any explanation for this?  Is this expected, probably a
> side effect of your latest changes?  I uncompressed the PDFs using
> `pdftk` and did a comparison; unfortunately I couldn't see big
> differences in the diff file, which probably hints at small but many
> changes that accumulate to the 15% difference.

If I understand correctly,
it is due to the difference in the method of invoking Ghostscript.
The new method was introduced from commit 017927b4.

For example, here is my experiment with commit 21a20de3 and gs-9.52.

```
$ cat foo.ly
{ c d e f g a b }

$ lilypond -dgs-never-embed-fonts --ps --pdf foo.ly
...snip...

$ echo "mark /OutputFile (foo.new-method.pdf) /PageSize [595.28 841.89] 
(pdfwrite) finddevice putdeviceprops setdevice newpath fill (foo.ps) run" > 
gs-command.ps

$ gs -dNODISPLAY -dNOSAFER -dNOPAUSE -dBATCH -dAutoRotatePages=/None 
-dPrinted=false gs-command.ps
...snip...

$ gs -dSAFER -dNOPAUSE -dBATCH -dAutoRotatePages=/None -dPrinted=false 
-sDEVICE=pdfwrite -dDEVICEHEIGHTPOINTS=841.89 -dDEVICEWIDTHPOINTS=595.28 
-sOutputFile=foo.old-method.pdf -f foo.ps
...snip...

$ ls -l *.pdf
-rw-r--r--+ 1 trueroad none 7081 Jul 11 14:41 foo.new-method.pdf
-rw-r--r--+ 1 trueroad none 6423 Jul 11 14:41 foo.old-method.pdf
-rw-r--r--+ 1 trueroad none 7084 Jul 11 14:41 foo.pdf

$
```

>From the same PostScript file, the new method generates a PDF of 7081 bytes
and the old method generates a PDF of 6423 bytes.
The new method is more than 10 % larger than the old method.

Next, I tried to find out what the increase was.

```
$ qpdf --qdf foo.pdf foo.qdf

$ qpdf --qdf foo.new-method.pdf foo.new-method.qdf

$ qpdf --qdf foo.old-method.pdf foo.old-method.qdf

$ diff -u foo.qdf foo.new-method.qdf > direct-to-new.diff

$ diff -u foo.old-method.qdf foo.new-method.qdf > old-to-new.diff

$
```

>From `old-to-new.diff`, it seems that the amount of drawing operators
in the /Contents has increased significantly.

Re: NR pdf is larger with current git by 15%

[Masamichi Hosoda]

>> From the same PostScript file, the new method generates a PDF of
>> 7081 bytes and the old method generates a PDF of 6423 bytes.  The
>> new method is more than 10 % larger than the old method.
> 
> Do you consider this a bug in gs or a feature?
> 
>> From `old-to-new.diff`, it seems that the amount of drawing
>> operators in the /Contents has increased significantly.
> 
> OK.  What do you think about opening a ghostscript issue, asking the
> developers for an explanation?  Maybe our new method can be slightly
> adapted to avoid the size increase.

I tried another method that adds `-sDEVICE=pdfwrite -sOutputFile=/dev/null`
and removes `-dNODISPLAY` from the new method.

```
$ gs -dNOSAFER -dNOPAUSE -dBATCH -dAutoRotatePages=/None -dPrinted=false 
-sDEVICE=pdfwrite -sOutputFile=/dev/null gs-command.ps
...snip...

$ qpdf --qdf foo.new-method.pdf foo.new-method.qdf

$ diff -u foo.old-method.qdf foo.new-method.qdf > old-to-new.diff

$
```

As a result, the drawing operators in PDF are the same as the old method.
Ghostscript seems to need `-sDEVICE=pdfwrite` to produce the PDF we expect.

Is the device specifying way on the new method (without `-sDEVICE=pdfwrite`)
documented by Ghostscript documents?
If not, it is not a bug in my humble opinion.

Re: NR pdf is larger with current git by 15%

[Masamichi Hosoda]

> Ghostscript seems to need `-sDEVICE=pdfwrite` to produce the PDF we expect.
> 
> Is the device specifying way on the new method (without `-sDEVICE=pdfwrite`)
> documented by Ghostscript documents?
> If not, it is not a bug in my humble opinion.

I've found the documented way to specify Ghostscript devices
without `-sDEVICE=pdfwrite`.
It is using the operator `selectdevice`.

https://www.ghostscript.com/doc/9.52/Use.htm#Output_device
https://www.ghostscript.com/doc/9.26/Use.htm#Output_device

Here is my experiment.

```
$ echo "mark /OutputFile (foo.new-documented.pdf) /PageSize [595.28 841.89] 
(pdfwrite) finddevice putdeviceprops setdevice (pdfwrite) selectdevice newpath 
fill (foo.ps) run" > gs-command-documented.ps

$ gs -dNODISPLAY -dNOSAFER -dNOPAUSE -dBATCH -dAutoRotatePages=/None 
-dPrinted=false gs-command-documented.ps
...snip...

$ qpdf --qdf foo.new-documented.pdf foo.new-documented.qdf

$ diff -u foo.old-method.qdf foo.new-documented.qdf > old-to-documented.diff

$
```

As a result, the drawing operators in PDF are the same as the old method.

Re: NR pdf is larger with current git by 15%

[Masamichi Hosoda]

>>> I've found the documented way to specify Ghostscript devices
>>> without `-sDEVICE=pdfwrite`.  It is using the operator
>>> `selectdevice`.
> 
> Thanks for investigating.  The main question remains why gs produces
> suboptimal results without `selectdevice`.

`selectdevice` is defined in gs_init.ps as follows.

```
/selectdevice
{ finddevice setdevice .setdefaultscreen } bind def
```

`selectdevice` does `setdevice` and then `.setdefaultscreen`.
If we use only `setdevice`, then `.setdefaultscreen` is not done.

I've found that `.setdefaultscreen` is described here.
https://www.ghostscript.com/doc/current/History3.htm#V3.61_Interpreter

Here's a quote.

-
Defines a .setdefaultscreen procedure that sets the default halftone screen,
transfer function, and stroke adjustment for the current device.  Changes
selectdevice and setpagedevice so that they call .setdefaultscreen;
setpagedevice calls .setdefaultscreen before calling the Install procedure,
so that Install can set different parameters if it wishes.  Note that
setdevice does not do this.  (gs_init.ps, gs_setpd.ps)
-

It seems to set some defaults for the current device.

This means that if we use only `setdevice`, those defaults are not set.
In that case, many drawing operators need to be written,
including the defaults that can be omitted if I understand correctly.

Re: NR pdf is larger with current git by 15%

[Masamichi Hosoda]

>> `selectdevice` does `setdevice` and then `.setdefaultscreen`.
>> If we use only `setdevice`, then `.setdefaultscreen` is not done.
> 
> Aha. So should we just call .setdefaultscreen ourselves? (calling
> "[...] setdevice (pdfwrite) finddevice setdevice" is duplicate) Looks
> like it was introduced long ago, so we can safely rely on it.

It seems that `.setdefaultscreen` is not described
at current Ghostscript documents.
Ghostscript sometimes removes old or undocumented features,
so it's better not to use them in my humble opinion.
e.g. https://ghostscript.com/pipermail/gs-devel/2020-May/010295.html

To avoid duplicate `setdevice`, we can do the following.

```
mark [...snip...] (pdfwrite) finddevice putdeviceprops pop (pdfwrite) 
selectdevice
```

I've created merge request !242.

Re: 2,21,4 released

[Masamichi Hosoda]

> Testing with Frescobaldi would be appreciated, and also with special
> characters in file names. I tried to model this after the code
> Masamichi-san wrote in March, but I'm not sure I understand what it's
> really needed for?

`stat ()` in msvcrt.dll has a problem with some Unicode file names.
So we use `_wstat ()` instead of `stat ()` if msvcrt.dll is used.
To use `_wstat ()`, we need a wide string,
so we use `MultiByteToWideChar ()` to convert it.

`stat ()` in newer UCRTs than msvcrt.dll does not have the problem.
Therefore, if `_UCRT` is defined,
we do not use `_wstat ()` and `MultiByteToWideChar ()`.

If I understand correctly,
`MoveFileExA ()` and `CopyFileA ()` do not have the problem.
So we do not necessarily need to use `MoveFileExW ()` and `CopyFileW ()`.

Re: 2,21,4 released

[Masamichi Hosoda]

>> `stat ()` in msvcrt.dll has a problem with some Unicode file names.
>> So we use `_wstat ()` instead of `stat ()` if msvcrt.dll is used.
>> To use `_wstat ()`, we need a wide string,
>> so we use `MultiByteToWideChar ()` to convert it.
> 
> Do you have an example of a file name that should not work? I now have
> three versions from GUB (one with MoveFileExW; one without but with
> wstat; and one without wstat) and all work correctly on a recent
> Windows 10. Does that mean the issue is gone with a recent update?

The file name that should not work depends on the language of Windows.
If a short file name exceeds 8 bytes + 3 bytes when written in UTF-8,
`stat ()` in msvcrt.dll cause an error.

In Windows, most files have a short file name
in addition to the normal file name.
The short file names are up to 8 bytes + 3 bytes in length
and are stored in a Windows language-dependent encoding.

For example, on Japanese Windows,
the short file name `インスト.LY` is stored in CP932 encoding
with 8 bytes + 2 bytes.
If the name is written in UTF-8 encoding, it is 12 bytes + 2 bytes.
So the short file name `インスト.LY` in Japanese Windows should not work.
However, in other languages Windows, it may work.

If the normal file name, such as `☃.ly`,
cannot be written in CP932, the problem does not occur
because the short file name is given in US-ASCII only.

Re: 2,21,4 released

[Masamichi Hosoda]

> Thanks for all the explanation, I think I'm slowly starting to
> understand the problem. I've tried to get short names with special
> characters, but failed so far (maybe you need a setting for this?).
> I'm giving up on this for now and will keep the code path with
> MoveFileExW unless you feel strongly about removing it.

Do you use German Windows?
Is its ANSI encodings CP1252?

If so, maybe `.ly` should not work with `stat ()` in msvcrt.dll.
If I understand correctly, `Ä` is 1 byte in CP1252 and is 2 bytes in UTF-8.
`.LY` is 8 bytes + 2 bytes in CP1252 and 16 bytes + 2 bytes in UTF-8.

Re: 2,21,4 released

[Masamichi Hosoda]

>> Do you have an example of a file name that should not work? I now have
>> three versions from GUB (one with MoveFileExW; one without but with
>> wstat; and one without wstat) and all work correctly on a recent
>> Windows 10. Does that mean the issue is gone with a recent update?
> 
> Might it be dependent on which VC++ package you have installed (or
> not) that determines which definition is used by Windows?
> 
> i.e. (from Hosoda-san)
> 
>> `stat ()` in newer UCRTs than msvcrt.dll does not have the problem. 
> 
> I won't pretend to know the internals here but my work laptop has a
> number of different iterations of the older VC++ components installed,
> some of them go back to 2013 (for old internally made tools we still
> use now and again) and they seem to co-exist but I don't know what it
> is LP calls in this case. Some were installed manually and some by the
> various software installers I have run.

If I understand correctly,
lilypond.exe generated by GUB uses msvcrt.dll instead of UCRT.

The recent Visual C++ generated executables use UCRT.
The executables generated by MinGW normally use msvcrt.dll.
The recent MinGW can generate executables that use UCRT
by a special configuration.
However, GUB contains older MinGW that can not be configured as such.

Re: 2,21,4 released

[Masamichi Hosoda]

> That was also my understanding, but "dir /x" already shows a short name
> of "D15A~1.LY" for "ä.ly". Maybe I'm doing something wrong? (not used
> Windows at that technical level for a few years now...)

Sorry, I have no idea because I'm not familiar with non-Japanese Windows.
At least in Japanese Windows, it is easy to create the file that `stat ()`
in msvcrt.dll raises the error.

Re: font problems

[Masamichi Hosoda]

> * Documentation/snippets/utf-8.ly:
> 
> [for Japanese]  →  Source Han Serif

For Japanese fonts, I suggest HaranoAji.
It is the default Japanese font from TeX Live 2020.

https://www.ctan.org/pkg/haranoaji

Re: font problems

[Masamichi Hosoda]

>>> For Japanese fonts, I suggest HaranoAji.
>>> It is the default Japanese font from TeX Live 2020.
>>> 
>>> https://www.ctan.org/pkg/haranoaji
>> 
>> OK, thanks for the suggestion.
> 
> After some thinking I'm not sure whether HaranoAji is the best
> solution.  Given that both LilyPond and ghostscript need the fonts, a
> TeXLive package doesn't help much – usually, FontConfig doesn't scan
> TeXLive font directories.  Additionally, older GNU/Linux distributions
> don't provide a separate package for this font.
> 
> I'm not sure how to tackle both the building of the Japanese texinfo
> PDFs and the Japanese demo stuff in the LilyPond example files at the
> same time.
> 
> Suggestions?  Ideas?

If I understand correctly, LilyPond needs to find the HaranoAji font,
but Ghostscript does not need to find it.

If LilyPond uses the HaranoAji font,
the font will be embedded in the generated Postscript file.
Even if you use `-dgs-never-embed-fonts`,
you can still pass the font to Ghostscript with `-dfont-ps-resdir`.

In Texinfo PDF building, you don't need Japanese fonts
if only the included figure PDFs use the Japanese fonts.
If you want to use Japanese characters in the text instead of PDF figures
of Texinfo, you need the Japanese font.

Of course, it is required that Fontconfig can find the HaranoAji font.

Most environments do not have Japanese fonts.
How about using generic font names (e.g serif and sans-serif etc.)
for Japanese font fall back?

This selects the generic serif font
if the environment that doesn't have the HaranoAji font.

```
\override #'(font-name . "HaranoAjiMincho-Regular,serif")
```

Re: Problem with OTF support

[Masamichi Hosoda]

> [lilypond dcd531b0f, gs 9.52]
> 
> Processing the following sample code
> 
>   \paper {
> #(define fonts
>   (set-global-fonts
>#:roman "Source Han Serif TW"
>#:sans "sans-serif"
>#:typewriter "monospace"
>  ))
>   }
> 
>   \markup { "ろ" }   % This is character U+308D.
> 
> causes the attached image, which is obviously wrong (inspite of the
> name, the 'TW' variant of the 'Source Han Serif' family contains
> Hiragana syllables).  Running `pdffonts` on the output PDF
> demonstrates that the right font gets used:
> 
>   UOIEOY+SourceHanSerifTW-Regular  CID Type 0C  Identity-H  yes yes no   9  0
> 
> Looking into the `--eps` output of LilyPond I see the following:
> 
>   /SourceHanSerifTW-Regular /CIDFont findresource [...]
>   2.1852 0. 0. 1277
>   1 print_glyphs
> 
> The main question is where the value 1277 comes from.
> 
> In the (OTF) font's `cmap` table, U+308D maps to glyph 'cid01536',
> which has index 1277 in the font.
> 
> However, the glyph shown in the image is U+2609, glyph 'cid01277'.
> index 1021.
> 
> If I extract the (raw) CFF file from the EPS file, glyph index 1277 is
> U+2609 (so CID numbers are one-to-one mapped to glyph indices, as
> indicated by the 'Identity-H' entry above).  The correct value should
> be – as can be guessed now – 1536.
> 
> In other words, CID values (this is, glyph indices in the raw CFF
> font) and OpenType glyph indices are mixed up.  Masamichi-san, do you
> have a quick fix for this?

If I understand correctly,
"Source Han Serif TW" is one of "Region-specific Subset OTFs".
They are subset fonts, so CID != GID.

"Language-specific OTFs" (e.g. "Source Han Serif TC") are not subset fonts,
so CID==GID.
I think these non-subset fonts would be fine.

Of course, if you use Hiragana,
it's better to use "Source Han Serif" (for Japanese)
instead of "Source Han Serif TC" (for Traditional Chinese).

Re: NotoSansCJKjp-Regular cannot be loaded

[Masamichi Hosoda]

>> ```
>> Layout output to `./1a/lily-d0aeef0b.eps'...
>> fatal error: Font NotoSansCJKjp-Regular cannot be loaded via Ghostscript 
>> because it is an OpenType/CFF Collection (OTC) font.
>> ```

> /usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc: Noto Sans CJK 
> JP:style=Regular

This file contains NotoSansCJKjp-Regular
but it is an OpenType/CFF Collection (OTF) font.

LilyPond can handle OTC fonts but Ghostscript cannot.
https://bugs.ghostscript.com/show_bug.cgi?id=696808

If you use LilyPond without `--dgs-load-fonts`,
OTC fonts don't raise the error because LilyPond processes them.
Instead, with `--dgs-load-fonts`,
LilyPond does not process font files
and passes through the file name to Ghostscript.
But Ghostscript cannot handle the OTC fonts.

Another solution is to not use OTC fonts.

Non-OTC version of NotoSansCJKjp-Regular is contained
by `NotoSansCJKjp-Regular.otf`.

Note: NotoSansJP-Regular and NotoSansCJKjp-Regular are similar
but different fonts.

Unfortunately, Debian/Ubuntu does not seem to have a package
that contains `NotoSansCJKjp-Regular.otf`.
https://packages.debian.org/search?searchon=contents&keywords=NotoSansCJK&mode=filename&suite=stable&arch=any

Re: NotoSansCJKjp-Regular cannot be loaded

[Masamichi Hosoda]

> Note: NotoSansJP-Regular and NotoSansCJKjp-Regular are similar
> but different fonts.

I've noticed that utf-8.ly confused the font names.

It requires the font `Noto Serif JP`.
https://gitlab.com/lilypond/lilypond/-/blob/d6b62def753b268fc5adfbe8443aa477d38395ef/input/regression/utf-8.ly#L30

And, it requires fonts-noto-cjk package for Debian/Ubuntu.
https://gitlab.com/lilypond/lilypond/-/blob/d6b62def753b268fc5adfbe8443aa477d38395ef/input/regression/utf-8.ly#L23

However, fonts-noto-cjk package contains `Noto Serif CJK JP`
instead of `Noto Serif JP`.
https://packages.debian.org/buster/fonts-noto-cjk

`Noto Serif CJK JP` and `Noto Serif JP` are similar but different fonts.
In other words, Debian/Ubuntu with fonts-noto-cjk package
does not have `Noto Serif JP` that utf-8.ly requires.

Then, I think fontconfig found NotoSansCJKjp-Regular
by searching for other fonts that can be used in Japanese
because the specified font is not available.

On the other hand, for Fedora,
utf-8.ly requires google-noto-serif-jp-fonts package.
https://gitlab.com/lilypond/lilypond/-/blob/d6b62def753b268fc5adfbe8443aa477d38395ef/input/regression/utf-8.ly#L18

It contains non-OTC version of exactly `Noto Serif JP`.
https://fedora.pkgs.org/33/fedora-x86_64/google-noto-serif-jp-fonts-20190416-7.fc33.noarch.rpm.html
Therefore, there is no problem.



There are many variations of the Noto font that can be used for Japanese.

* Region-specific Subset OpenType/CFF (Subset OTF)
+ Font name: Noto{Sans|Serif}JP-*
+ File name: Noto{Sans|Serif}JP-*.otf
* Language-specific OpenType/CFF (OTF)
+ Font name: Noto{Sans|Serif}CJKjp-*
+ File name: Noto{Sans|Serif}CJKjp-*.otf
* OpenType/CFF Collection (OTC)
+ Font name: Noto{Sans|Serif}CJKjp-* and non-Japanese font names
+ File name: Noto{Sans|Serif}CJK-*.ttc
* Super OpenType/CFF Collection (Super OTC)
+ Font name: Noto{Sans|Serif}CJKjp-* and non-Japanese font names
+ File name: Noto{Sans|Serif}CJK.ttc

Noto's OTF, OTC, and SuperOTC have the same font name, the same font content,
and only the file format is different.
Noto's Subset OTF is similar but different font names
and font content from the other fonts.

Re: NotoSansCJKjp-Regular cannot be loaded

[Masamichi Hosoda]

> I've noticed that utf-8.ly confused the font names.

I've created a merge request.
https://gitlab.com/lilypond/lilypond/-/merge_requests/595

It solves the confusion of Japanese font names in utf-8.ly
but does not solve the `--dgs-load-fonts` warning.

Re: Cairo plans

[Masamichi Hosoda]

> * when could/would we drop the PS backend altogether?

In my humble opinion,
the PS backend is still needed for CJK character handling.
When you copy and paste text from a PDF generated by the cairo backend,
some CJK characters are garbled.
By the PS backend, no such garbling occurs.

Here's a sample.

```
\version "2.22.0"
\markup {
  \override #'(font-name . "HaranoAjiMincho")
  { 初見はハ長調で高音の白玉があった } }
\markup {
  \override #'(font-name . "HaranoAjiMincho")
  \override #'(font-features . ("jp90"))
  { 辻井 逗子 飴玉 } }
```

Re: Cairo plans

[Masamichi Hosoda]

> I installed your Harano fonts in ~/.fonts. Fontconfig sees them, but I
> still get DroidSans when I compile the snippet. How do I reproduce the
> problem?

Would you show me the results of the follwing commands?

```
find ~/.fonts -name "HaranoAjiMincho-*.otf"
fc-list "HaranoAjiMincho"
lilypond -dshow-available-fonts 2>&1 | grep "Harano Aji Mincho"
```

Re: Cairo plans

[Masamichi Hosoda]

> I analyzed more closely in the issue
> https://gitlab.com/lilypond/lilypond/-/issues/6172
> 
> Would you know how the mapping of character index => codepoint is
> supposed to work for this font with character 3056 (辻) ?

Most PDF readers have a mapping
from Adobe-Japan1 CID to Unicode code points as follows.
https://github.com/adobe-type-tools/mapping-resources-pdf/blob/master/pdf2unicode/Adobe-Japan1-UCS2

In the case of Acrobat Reader on Windows, it is stored in the following path.
C:\Program Files (x86)\Adobe\Acrobat Reader DC\Resource\CMap\Adobe-Japan1-UCS2

The number 3056 in the Adobe-Japan1 CID is 0xbf0 in hexadecimal.
The Adobe-Japan1-UCS2 file contains the following line,
which shows that CID 0xbf0 is U+8FBB.
https://github.com/adobe-type-tools/mapping-resources-pdf/blob/20190401/pdf2unicode/Adobe-Japan1-UCS2#L2329

```
<0bf0> <8fbb>
```

In the case of poppler, the file format is different,
but the following file has a similar mapping.
https://gitlab.freedesktop.org/poppler/poppler-data/-/blob/master/cidToUnicode/Adobe-Japan1

There is a Unicode code point at line 3056 + 1 (i.e. line 3057),
which shows U+8FBB.
https://gitlab.freedesktop.org/poppler/poppler-data/-/blob/POPPLER_DATA_0_4_10/cidToUnicode/Adobe-Japan1#L3057

Re: Cairo plans

[Masamichi Hosoda]

>> Most PDF readers have a mapping
>> from Adobe-Japan1 CID to Unicode code points as follows.
>> https://github.com/adobe-type-tools/mapping-resources-pdf/blob/master/pdf2unicode/Adobe-Japan1-UCS2
> 
> Is it impossible to discover this mapping from the OTF file alone?

If I understand correctly, yes.

> Where does the OTF file say it is using encoding Adobe-Japan1-UCS

It is described in ROS in the CFF table.
https://docs.microsoft.com/en-us/typography/opentype/spec/cff
http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/font/pdfs/5176.CFF.pdf

ROS means `Registry-Ordering-Supplement`.
For the current HaranoAjiMincho-Regular.otf,
Registry is `Adobe`, Ordering is `Japan1`, and Supplement is `7`.
So it is an `Adobe-Japan1-7` font.

PDF readers use Adobe-Japan1-UCS2 for `Adobe-Japan1-*` fonts.

Re: Cairo plans

[Masamichi Hosoda]

 Most PDF readers have a mapping from Adobe-Japan1 CID to Unicode
 code points as follows.
 https://github.com/adobe-type-tools/mapping-resources-pdf/blob/master/pdf2unicode/Adobe-Japan1-UCS2
>>>
>>> Is it impossible to discover this mapping from the OTF file alone?
>> 
>> If I understand correctly, yes.
> 
> I guess Masamichi means that the mapping is not part of the font
> itself.

Yes. Thanks.

>>> Where does the OTF file say it is using encoding Adobe-Japan1-UCS
>> 
>> It is described in ROS in the CFF table.
>> https://docs.microsoft.com/en-us/typography/opentype/spec/cff
>> http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/font/pdfs/5176.CFF.pdf
>> 
>> ROS means `Registry-Ordering-Supplement`.
>> For the current HaranoAjiMincho-Regular.otf,
>> Registry is `Adobe`, Ordering is `Japan1`, and Supplement is `7`.
>> So it is an `Adobe-Japan1-7` font.
>> 
>> PDF readers use Adobe-Japan1-UCS2 for `Adobe-Japan1-*` fonts.
> 
> Hopefully there is a solution that makes Cairo do the mapping work for
> us...

If I understand correctly, it is PDF readers that use Adobe-Japan1-UCS,
not PDF generators such as cairo.
If you use the PS backend and Ghostscript outputs the PDF,
it does not use Adobe-Japan1-UCS, but it works fine.

Re: Cairo plans

[Masamichi Hosoda]

> I analyzed more closely in the issue
> https://gitlab.com/lilypond/lilypond/-/issues/6172

The garbling sample I have shown has two different issues.
#6172 is one of them.

I created #6173 for the other one.
https://gitlab.com/lilypond/lilypond/-/issues/6173

Re: Releasing 2.20

[Masamichi Hosoda]

> Hans Aikema  writes:
> 
>> David,
>>
>> Make sure that your main focus will be getting back into good shape.
>>
>> If schedule and energy levels allow for starting of a 2.20 branch I
>> think it would be good to update ghostscript to a version that has the
>> PNG-transparency glitches fixed that surfaced in 2.19.51.
>> Ghostscript has made a release with the fix for the issue that was
>> created by Masamichi Hosoda as a result of the discovery of the
>> PNG-transparency bug in the 2.19.51 Lilypond build)
> 
> That's more a matter of GUB than of LilyPond.  GhostScript is more of a
> runtime dependency so it's not really something we would refuse a build
> for.

I've sent a pull request that updates GUB's Ghostscript to 9.21.
https://github.com/gperciva/gub/pull/39

However, on FreeBSD 32 bit, both Ghostscript 9.20 and 9.21 crash.
So only for FreeBSD 32 bit, Ghostscript 9.15 is used.

Linux 64 bit, Linux 32 bit, Linux PPC, FreeBSD 64 bit,
and Windows are no problem.

In my humble opinion, after or before releasing 2.20,
it is better to discontinue some platforms' binary release.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB Ghostscript 9.21

[Masamichi Hosoda]

>> Phil Holmes:
>> ...
>>> /usr/bin/ld: cannot find -lfreetype
>> ...
>> ld cannot find libfreetype.so
>> Do you have it installed ?
>> Try this:
>> $ locate libfreetype.so
>> ...
>> /usr/lib/i386-linux-gnu/libfreetype.so
>> /usr/lib/i386-linux-gnu/libfreetype.so.6
>> /usr/lib/i386-linux-gnu/libfreetype.so.6.8.1
>> ...
>> 
> 
> I get:
> 
> /home/gub/NewGub/gub/target/freebsd-64/root/usr/lib/libfreetype.so
> /home/gub/NewGub/gub/target/freebsd-64/root/usr/lib/libfreetype.so.18
> /home/gub/NewGub/gub/target/freebsd-x86/root/usr/lib/libfreetype.so
> /home/gub/NewGub/gub/target/freebsd-x86/root/usr/lib/libfreetype.so.18
> /home/gub/NewGub/gub/target/linux-64/root/usr/lib/libfreetype.so
> /home/gub/NewGub/gub/target/linux-64/root/usr/lib/libfreetype.so.6
> /home/gub/NewGub/gub/target/linux-64/root/usr/lib/libfreetype.so.6.12.5
> /home/gub/NewGub/gub/target/linux-ppc/root/usr/lib/libfreetype.so
> /home/gub/NewGub/gub/target/linux-ppc/root/usr/lib/libfreetype.so.6
> /home/gub/NewGub/gub/target/linux-ppc/root/usr/lib/libfreetype.so.6.12.5
> /home/gub/NewGub/gub/target/linux-x86/root/usr/lib/libfreetype.so
> /home/gub/NewGub/gub/target/linux-x86/root/usr/lib/libfreetype.so.6
> /home/gub/NewGub/gub/target/linux-x86/root/usr/lib/libfreetype.so.6.12.5
> /home/gub/NewGub/gub/target/tools/root/usr/lib/libfreetype.so
> /home/gub/NewGub/gub/target/tools/root/usr/lib/libfreetype.so.6
> /home/gub/NewGub/gub/target/tools/root/usr/lib/libfreetype.so.6.12.5
> /usr/lib/i386-linux-gnu/libfreetype.so.6
> /usr/lib/i386-linux-gnu/libfreetype.so.6.11.1

If I understand correctly,
your system is Ubuntu 14.04 LTS 32 bit (i386-linux-gnu).
It seems that your system missing this file:
/usr/lib/i386-linux-gnu/libfreetype.so

So `ld` cannot find `-lfreetype`.

My system (Ubuntu 14.04 LTS 64 bit, x86_64-linux-gnu) has this file:
/usr/lib/x86_64-linux-gnu/libfreetype.so

So `ld` can find `-lfreetype`.
There is no problem.

I cannot reproduce the issue. However, I've created a patch.
Would you try it?
https://github.com/gperciva/gub/pull/40

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

translation branch and stable/2.20 branch

[Masamichi Hosoda]

2.19.65 has been released,
it seems that it does not contain some recent translations.

e.g.
eb38c33a95 Doc-ja: fix cindex in NR
eccad9a9f8 Doc-ja: update Usage
1c1a932358 Doc-es: update one texidoc.
e7385f6e58 Web-es: update Introduction.
2cfed6cc93 Doc-es: update Notation/Simultaneous.
9f53d74eaf Web: Add a new document to what We Wrote list.
ce1c9d0ea4 Doc-ca: Catalan translation of the notation manual: expressive.itely 
and text.itely
etc.

Is the translation branch not contained in the release?
Or will it be contained in the next release?

On the other hand,
the current "translation" branch does not seem to contain
recent document updates.

e.g.
0712559601 Issue 5164: Doc: Add usage of OpenType font feature
97956152eb Issue 5166/2: Doc: Delete note for svg backend which become 
unnecessary
f40aa7dc74 Issue 5166/1: Doc: Update default font description
0e23f12caa Issue 5163: Update URW++ OTF files
68dbc841e3 Doc: CG, explain other git prompt variables
54a35cdc57 Web: Authors.itexi update
2dcca6dacd Doc: add comment about automatically generated documentation
etc.

Which branch should the translators translate?
e.g. translate the English document in the master branch,
commit to the translation branch, etc.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

ghostscript 9.22 will remove PDFDontUseFontObjectNum option

[Masamichi Hosoda]

I've noticed that ghostscript 9.22
will remove `PDFDontUseFontObjectNum` option.

http://git.ghostscript.com/?p=ghostpdl.git;a=commit;h=ca1ec9b486ddba3f921355fd1d775f27f4871356

So `--bigpdfs` will not work with gs-9.22.
It already did not work with gs-9.22rc1.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: ghostscript 9.22 will remove PDFDontUseFontObjectNum option

[Masamichi Hosoda]

>> I've noticed that ghostscript 9.22
>> will remove `PDFDontUseFontObjectNum` option.
>>
>> http://git.ghostscript.com/?p=ghostpdl.git;a=commit;h=ca1ec9b486ddba3f921355fd1d775f27f4871356
>>
>> So `--bigpdfs` will not work with gs-9.22.
>> It already did not work with gs-9.22rc1.
> 
> Did they give a rationale?  What kind of PostScript code would have a
> chance to work here instead?

I have no idea.
I've send a request that
I'd like to use `PDFDontUseFontObjectNum` option to gs-devel.
https://ghostscript.com/pipermail/gs-devel/2017-September/009987.html

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: [gs-devel] Ghostscript/GhostPDL 9.22 Release Candidate 1

[Masamichi Hosoda]

>> > Please give them a try on your system if you're interested in helping
>> > test the release-in-progress. Your feedback is appreciated.
>>
>>It seems that `-dPDFDontUseFontObjectNum` option does not work.
> 
> It has been removed, as documented in History9.htm:
> 
> 2017-08-02 13:41:59 +0100
> Ken Sharp 
> ca1ec9b486ddba3f921355fd1d775f27f4871356
> 
> PDF interpreter - remove the PDFDontUseObjectNum switch
> 
> This was implemented to allow us to restore the default behaviour if
> it caused problems. No real problems reported, so lets get rid of
> (yet another) of our many, many command line switches.
> 
> 
>>`-dPDFDontUseFontObjectNum` is useful to unify duplicate fonts.
>>So I would like to use `-dPDFDontUseFontObjectNum`.
> 
> The fonts can't realistically be described as duplicates, if they have
> different object numbers. Or if they are then the original PDF file
> (this only works with PDF input) was poorly constructed. Using this
> option isn't really sensible, it was only ever present in case the
> Font numbering usage went awry. Consider creating the original PDF
> files more efficiently instead.
> 
> I'll discuss it with the other developers but I am not inclined to
> restore this. Obviously if you feel its important you can revert the
> change locally.

When you create a PDF document using something like a TeX system
you may include many small PDF files in the main PDF file.
It is common for each of the small PDF files to use the same fonts.

If the small PDF files contain embedded full font sets,
the TeX system includes all of them in the main PDF.
The main PDF contains duplicates of the same full sets of fonts.
Therefore, `PDFDontUseFontObjectNum` can remove the duplicates.
This may considerably reduce the main PDF-file's size.

There is a tool for using this method of removing duplicate fonts.
https://www.ctan.org/pkg/extractpdfmark
https://packages.debian.org/stretch/extractpdfmark
http://packages.ubuntu.com/zesty/extractpdfmark

LilyPond has option `--bigpdfs` for unifying duplicate fonts in this method.
http://lilypond.org/doc/v2.19/Documentation/usage/command_002dline-usage#basic-command-line-options-for-lilypond

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Ghostscript/GhostPDL 9.22 Release Candidate 1

[Masamichi Hosoda]

> So the question then becomes 'why are the fonts subset ?' That's a
> really good question, and the answer is that I don't know. Its
> possible that there is a genuine pdfwrite bug here. The piece of
> information I'm missing is the step used to create the PDF files from
> the EPS files, I don't know how you are doing that.

We use the following command to convert from EPS to PDF.

$ gs -dSAFER -dEPSCrop -dCompatibilityLevel=1.4 -dNOPAUSE -dBATCH -r1200 
-dSubsetFonts=false -sDEVICE=pdfwrite -dAutoRotatePages=/None 
-sOutputFile=filename.pdf -c.setpdfwrite -ffilename.eps

We believed that Ghostscript generates full font embedded (non-subset) PDF
when `-dSubsetFonts=false` is specified.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Ghostscript/GhostPDL 9.22 Release Candidate 1

[Masamichi Hosoda]

>>We use the following command to convert from EPS to PDF.
>>
>>$ gs -dSAFER -dEPSCrop -dCompatibilityLevel=1.4 -dNOPAUSE -dBATCH
>>-r1200 -dSubsetFonts=false -sDEVICE=pdfwrite -dAutoRotatePages=/None
>>-sOutputFile=filename.pdf -c.setpdfwrite -ffilename.eps
>>
>>We believed that Ghostscript generates full font embedded (non-subset)
>>PDF
>>when `-dSubsetFonts=false` is specified.
> 
> It should. In this case, I believe it does not. No clue why at the
> moment though.

Thank you for your answer.

If there is a full font embedded (non-subset) PDF,
does the bigpdfs trick work effectively?
Or, even so, should we take other methods (e.g. using non-embedded PDFs)?

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Ghostscript/GhostPDL 9.22 Release Candidate 1

[Masamichi Hosoda]

>>If there is a full font embedded (non-subset) PDF,
>>does the bigpdfs trick work effectively?
> 
> Its still, in my opinion, a risky thing to do. However, if the font
> were fully embedded, you wouldn't need to use Ghostscript and the
> PDFDontUseFontObjectNum bug approach (which is the risky part).
> 
> Because the fonts would be genuinely identical, MuPDF would be able to
> spot the font streams at least as being the same and would be able to
> reliably remove the duplicates.
> 
> The Font and FontDescriptor dictionaries might not be possible to
> remove, so the effect wouldn't be quite as good as the current
> approach, but these dictionaries run to a few tens or at most a few
> hundred bytes. The FontFile streams are where most of the space is
> going, and those would be possible to remove, if they were truly
> duplicates.

I understand that the bigpdfs trick should to be fixed
to use MuPDF instead of PDFDontUseFontObjectNum bug.

>>Or, even so, should we take other methods (e.g. using non-embedded
>>PDFs)?
> 
> I'm not sure what other methods there would be. Using EPS inclusions
> would have the same effect as PDF. Rendering to bitmaps would be (I'd
> guess) as large as the PDF files, and would suffer from
> non-scalability.
> 
> Converting to TeX format would probably work, but apparently there
> were problems with that.
> 
> Is there some other approach available ?

There is a method of using font non-embedded PDF.
In my experiment, it seems to work fine except TrueType fonts.

It uses like
`-c ".setpdfwrite << /NeverEmbed [ /Emmentaler-20 /TeXGyreSchola-Regular ] >> 
setdistillerparams"`
instead of
`-sOutputFile=filename.pdf`.

I've made sample files `20170922_lilypond_eps_pdf_examples.tar.xz`.
https://drive.google.com/file/d/0ByGBX3PDrqjsSFhVdXJfbjFjRlk/view?usp=sharing

In the tarball:

`fonts?-bigpdfs.eps`: EPS that is generated by LilyPond.
`fonts?-bigpdfs-noembed.pdf`: No font embedded PDF
  that is generated by Ghostscript
  with `/NeverEmbed` parameter like above.
`include-bigpdfs-noembed.pdf`: TeX outputted PDF that lacks LilyPond fonts.
`include-bigpdfs-noembed-gse.pdf`: Finally PDF that is font embedded
   by Ghostscript.

In `include-bigpdfs-noembed-gse.pdf`,
OTF seems to be fine. However, some glyphs of TTF are broken.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: [gs-devel] Ghostscript/GhostPDL 9.22 Release Candidate 1

[Masamichi Hosoda]

> In my impression, if
> 
> $ gs -dSAFER -dEPSCrop -dCompatibilityLevel=1.4 -dNOPAUSE -dBATCH
> -r1200
> -dSubsetFonts=false -sDEVICE=pdfwrite -dAutoRotatePages=/None
> -sOutputFile=filename.pdf -c.setpdfwrite -ffilename.eps
> 
> does not embed the completed font into PDF, Ghostscript developers
> would
> regard it as a bug and they might be willing to fix it (at least, try
> to
> fix it).
> 
> Could you provide some sample EPS files to reproduce the issue,
> and the resulted PDF in your environment?

Here is sample files `20170922_lilypond_eps_pdf_examples.tar.xz`
https://drive.google.com/file/d/0ByGBX3PDrqjsSFhVdXJfbjFjRlk/view?usp=sharing

It contains `README.md`.
Would you read the file?

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: [gs-devel] Ghostscript/GhostPDL 9.22 Release Candidate 1

[Masamichi Hosoda]

This discussion concerns two mailing lists,
some mails exist in both archives,
but other mail seems to exist only in one archive.
For convenience of people subscribing to only one mailing list,
both mailing list archive URLs are as follows.

http://lists.gnu.org/archive/html/lilypond-devel/2017-09/index.html
https://ghostscript.com/pipermail/gs-devel/2017-September/date.html

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Ghostscript/GhostPDL 9.22 Release Candidate 1

[Masamichi Hosoda]

> Note that I didn't attempt this on 9.21, just the 9.22 release
> candidate. While the fonts are properly dropped from the individual
> PDF files, none of the text was visible in the final PDF file, when
> rebuilding with Emmentaler supplied as an external font in fontmap.GS.
> 
> Its interesting that the OTF worked for you because I would expect it
> to; OTF fonts with CFF outlines are handled simply be extracting the
> CFF, which is usable directly in PostSceript. TrueType fonts are not
> directly supported by PostScript and that's where the heuristics come
> back again, we have to guess at some aspects of the encodings.

In my experiment, I used gs-9.22rc1 but I didn't use fontmap.GS.

Instead, I created a PostScript file
that contains only font resource in the same format as EPSs.
That is, for the  Emmentaler-20 font,
I created a file `fonts/Emmentaler-20.font.ps` with the following contents.

```
%%BeginFont: Emmentaler-20
%%BeginResource: font Emmentaler-20
%!PS-Adobe-3.0 Resource-FontSet
%%DocumentNeededResources: ProcSet (FontSetInit)
%%Title: (FontSet/Emmentaler-20)
%%Version: 0
%%EndComments
%%IncludeResource: ProcSet (FontSetInit)
%%BeginResource: FontSet (Emmentaler-20)
/FontSetInit /ProcSet findresource begin
%%BeginData: 67355 Binary Bytes
[...snip...]
%%EndData
%%EndResource
%%EndResource
%%EndFont
```

Then, I invoked Ghostscipt with the following command.

$ gs -dBATCH -dNOPAUSE -sDEVICE=pdfwrite \
 -sOutputFile=include-bigpdfs-noembed-gse.pdf \
 -c ".setpdfwrite" \
 -f fonts/*.font.ps include-bigpdfs-noembed.pdf

`fonts/Emmentaler-20.font.ps` and other font files
`fonts/TeXGyreSchola-Regular.font.ps` etc. are in my sample tarball.

In this method, for TrueType fonts,
I think that the encoding is confusing and broken as you mentioned.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: speed of fontconfig

[Masamichi Hosoda]

 [... replacing] 2.12.6 [... with] 2.12.93 indeed provides ~100x
 improvement and builds the cache on my system in 600ms.
>>>
> 
>>> This was a report of using fontconfig on Windows, IIRC.  So my
>>> question: Does the Windows tarball of lilypond contain a recent
>>> fontconfig version?
>> 
>> According to
>> https://github.com/gperciva/gub/blob/master/gub/specs/fontconfig.py
>> it's 2.12.1.
> 
> Should we upgrade?  Masamichi-san?

For LilyPond 2.21, we should upgrade fontconfig, in my humble opinion.
But, for LilyPond 2.20, I'm wondering whether we should upgrade or not.

If I understand correctly, the new fontconfig has significant changes.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB

[Masamichi Hosoda]

>> Ubuntu 14 for both.  I would expect Ubuntu 16 to work but have not tried it.
> 
> Not sure about that.  We had several incompatibilities in the LilyPond
> code base since then (2.18.2 does not compile on current GCC versions)
> and GCC is written in C++ these days.  Though the bootstrap process
> might actually go through C-written GCC to get to its C++ compiler: not
> sure about that.

I tried to use GUB in Ubuntu 16.04 LTE 64 bit.
But It failed with some errors.

So I use GUB in Ubuntu 14.04 LTE 64 bit.
I have not tried Ubuntu 18.04 LTE yet.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB

[Masamichi Hosoda]

>> GUB worked for a long time, but we have a) an unsolved problem
>> building the pdfs of the english documentation of stable/2.20,
> 
> That sounds like more of a race condition to me, so it's likely
> unrelated to GUB but may be related to building in a separate directory
> or to cross-compilation.

I found a possibility that the wrong PDF is output
if the build directory is not clean.
This issue is unrelated to GUB.
With GUB, without GUB, it might happen in both.

You can reproduce by the following procedure without GUB.

```
$ git checkout release/2.19.82-1
$ ./autogen.sh --noconf
$ rm -fr build
$ mkdir build
$ mkdir -p out-www/offline-root/Documentation/
$ echo "wrong file" > out-www/offline-root/Documentation/notation.pdf
$ mkdir -p out-www/online-root/Documentation/
$ echo "wrong file" > out-www/online-root/Documentation/notation.pdf
$ ../configure
$ make -j 8
$ make -j 8 CPU_COUNT=8 WEB_TARGETS='offline online' LANGS='' doc
```

The result is as follows.

```
$ find . -name 'notation.pdf' -exec ls -lah {} \;
-rw-r--r-- 1 trueroad none 6.5M Jul 13 22:37 
./Documentation/out-www/notation.pdf
-rw-r--r-- 1 trueroad none 11 Jul 13 21:39 
./out-www/offline-root/Documentation/notation.pdf
-rw-r--r-- 1 trueroad none 11 Jul 13 21:39 
./out-www/online-root/Documentation/notation.pdf

$ cat out-www/offline-root/Documentation/notation.pdf
wrong file

$ cat out-www/online-root/Documentation/notation.pdf
wrong file

$
```

I'll create a patch which fixes this issue.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB

[Masamichi Hosoda]

> Masamichi Hosoda  writes:
> 
>>>> GUB worked for a long time, but we have a) an unsolved problem
>>>> building the pdfs of the english documentation of stable/2.20,
>>> 
>>> That sounds like more of a race condition to me, so it's likely
>>> unrelated to GUB but may be related to building in a separate directory
>>> or to cross-compilation.
>>
>> I found a possibility that the wrong PDF is output
>> if the build directory is not clean.
> 
> Like when a build aborted, maybe.
> 
>> I'll create a patch which fixes this issue.
> 
> Let's hope that this will avoid this problem in future.

I've created the patch.
https://sourceforge.net/p/testlilyissues/issues/5380/
https://codereview.appspot.com/355750043

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB

[Masamichi Hosoda]

 I found a possibility that the wrong PDF is output
 if the build directory is not clean.
>>
>> I've created the patch.
>> https://sourceforge.net/p/testlilyissues/issues/5380/
>> https://codereview.appspot.com/355750043
> 
> Thanks a lot!

If I understand correctly,
there is a possibility that intermediate PDFs are output as final PDFs
since the intermediate filename is the same as the final filename.

So I've created a patch that changes the intermediate PDF filename.
https://sourceforge.net/p/testlilyissues/issues/5381/
https://codereview.appspot.com/357760043/

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Ghostscript 9.26 cannot embed CID fonts

[Masamichi Hosoda]

I've noticed that gs-9.26 cannot embed CID fonts
for the extractpdfmark method.

In the method, we defined fonts in PostScript, and use it in PDF.
gs-9.26 cannot define the fonts for PDF in PostScript.
https://bugs.ghostscript.com/show_bug.cgi?id=700367#c4

To define fonts for ghostscript,
root privileges seems to be required on most systems.
https://bugs.ghostscript.com/show_bug.cgi?id=700367#c6

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: gub's `make bootstrap' is useless

[Masamichi Hosoda]

> So it seems that you can save some hours of compilation time if you
> don't build the `bootstrap' target.  I now wonder whether this is true
> on my openSuSE GNU/Linux box only ...

Sorry, I have not worked on GUB because I have no time for the time being.
Thank you for your work on GUB.

In my experience, it seems that `make lilypond' succeeded
even if I forgot `make bootstrap'.
My GUB environment is Ubuntu 14.04 64 bit.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: gub: I can now completely build lilypond

[Masamichi Hosoda]

> Hi Werner!
>>> But building of lilypond-test fails.
>>>
>>> ./target/linux-64/build/lilypond-git.sv.gnu.org--lilypond.git-master/input/regression/lilypond-book/suffix-tely.texi2pdf.log:
>>>
>>> /home/gub/NewGub/gub/target/tools/root/usr/bin/texi2dvi: texinfo.tex
>>> appears to be broken.
>>> This may be due to the environment variable TEX set to something
>>> other than (plain) tex, a corrupt texinfo.tex file, or
>>> to tex itself simply not working.
>>> etex:
>>> /home/gub/NewGub/gub/target/linux-64/root/usr/lib/libstdc++.so.6:
>>> version `CXXABI_1.3.9' not found (required by etex)
>> Obviously another linking path issue: While executing the external TeX
>> binaries, DLL stuff from `target/...' must not be used.
> 
> The problem here is that our texi2* stuff needs LD_LIBRARY_PATH to
> find our DLLs in target/..., but the system's etex binary, executed by
> texi2*, fails on openSuSE Tumbleweed because it is incompatible to
> those libs in target/
> 
> I think the easiest solution is a simple etex wrapper script in
> tools/root that clears LD_LIBRARY_PATH and then executes the system's
> etex.

texi2dvi / texi2pdf can be specified external program name.
Previously, there was a similar error in XeTeX with texi2pdf.

For XeTeX, we use wrapper scripts.
http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=blob;f=scripts/build/xetex-with-options.sh
http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=blob;f=scripts/build/xelatex-with-options.sh

In the LilyPond's makefiles, it is specified as follows.

```
PDFTEX=$(PDFTEX) PDFLATEX=$(PDFLATEX)  ... $(LILYPOND_BOOK)
```

The valiables, PDFTEX and PDFLATEX are set to be
xetex-with-options.sh and xelatex-with-options.sh at configure script.

To avoid the error,
I've added a line to clear LD_LIBRARY_PATH to the scripts in GUB.
https://github.com/gperciva/gub/commit/57d66d1262551c062a3edc85e27ac69748f2f136#diff-94f19c1786c0853ef47371aa55ba2b21

If I understand correctly,
it may be resolved simply by changing LilyPond's makefiles as follows.

```
TEX=$(PDFTEX) PDFTEX=$(PDFTEX) PDFLATEX=$(PDFLATEX)  ... $(LILYPOND_BOOK)
```

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: gub: I can now completely build lilypond

[Masamichi Hosoda]

 But building of lilypond-test fails.

 ./target/linux-64/build/lilypond-git.sv.gnu.org--lilypond.git-master/input/regression/lilypond-book/suffix-tely.texi2pdf.log:

 /home/gub/NewGub/gub/target/tools/root/usr/bin/texi2dvi: texinfo.tex
 appears to be broken.
 This may be due to the environment variable TEX set to something
 other than (plain) tex, a corrupt texinfo.tex file, or
 to tex itself simply not working.
 etex:
 /home/gub/NewGub/gub/target/linux-64/root/usr/lib/libstdc++.so.6:
 version `CXXABI_1.3.9' not found (required by etex)
>>> Obviously another linking path issue: While executing the external TeX
>>> binaries, DLL stuff from `target/...' must not be used.
>> 
>> The problem here is that our texi2* stuff needs LD_LIBRARY_PATH to
>> find our DLLs in target/..., but the system's etex binary, executed by
>> texi2*, fails on openSuSE Tumbleweed because it is incompatible to
>> those libs in target/
>> 
>> I think the easiest solution is a simple etex wrapper script in
>> tools/root that clears LD_LIBRARY_PATH and then executes the system's
>> etex.
> 
> texi2dvi / texi2pdf can be specified external program name.
> Previously, there was a similar error in XeTeX with texi2pdf.
> 
> For XeTeX, we use wrapper scripts.
> http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=blob;f=scripts/build/xetex-with-options.sh
> http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=blob;f=scripts/build/xelatex-with-options.sh
> 
> In the LilyPond's makefiles, it is specified as follows.
> 
> ```
> PDFTEX=$(PDFTEX) PDFLATEX=$(PDFLATEX)  ... $(LILYPOND_BOOK)
> ```
> 
> The valiables, PDFTEX and PDFLATEX are set to be
> xetex-with-options.sh and xelatex-with-options.sh at configure script.
> 
> To avoid the error,
> I've added a line to clear LD_LIBRARY_PATH to the scripts in GUB.
> https://github.com/gperciva/gub/commit/57d66d1262551c062a3edc85e27ac69748f2f136#diff-94f19c1786c0853ef47371aa55ba2b21
> 
> If I understand correctly,
> it may be resolved simply by changing LilyPond's makefiles as follows.
> 
> ```
> TEX=$(PDFTEX) PDFTEX=$(PDFTEX) PDFLATEX=$(PDFLATEX)  ... $(LILYPOND_BOOK)
> ```

I noticed I was wrong...
texi2dvi / texi2pdf should not invoke etex in GUB.
xetex should be invoked.

Does your openSuSE system have both xetex and xelatex?
LilyPond's configure script sets PDFTEX=/path/to/xetex-with-option.sh
if your openSuSE system has both xetex and xelatex.
Then, texi2pdf invokes xetex-with-option.sh instead of etex.
In this case, etex does not be invoked.
The test succeeds because GUB makes
xetex-with-option.sh clears LD_LIBRARY_PATH.

On the other hand,
LilyPond's configure script sets PDFTEX=etex
if your openSuSE system does not have xelatex, pdfetex, and pdftex.
In this case, texi2pdf invokes etex. LD_LIBRARY_PATH is not cleared.
GUB has older libc and older libstdc++.
LD_LIBRARY_PATH makes etex uses the old libc and the old libstdc++.
But, openSuSE system's etex requires newer libc and newer libstdc++.
This is the cause of the error.

Again, if I understand correctly,
etex is not invoked if your openSuSE system have both xetex and xelatex.
If your openSuSE system has both xetex and xelatex,
GUB clears LD_LIBRARY_PATH and the test will succeed.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: I cannot run make check since Issue 5450: relocate.cc: Introduce new command `set?'

[Masamichi Hosoda]

> This is a gs 9.26 issue and I cannot see how this might be related to what we 
> have hit here, so maybe Hosoda-san will be abel to figure why make check is 
> breaking without extractpdfmark installed.

If I understand correctly, this patch solves the error.

```
--- a/scripts/build/output-distance.py
+++ b/scripts/build/output-distance.py
@@ -652,7 +652,7 @@ class SignatureFileLink (FileLink):
 data_option = ''
 if options.local_data_dir:
 data_option = 
('-slilypond-datadir=%s/share/lilypond/current '
-   % dir)
+   % (cur_dir + '/' + dir))

 cmd = ('gs'
' -sDEVICE=png16m'
```

Without extractpdfmark,
each eps file has relative font path from `lilypond-datadir`.
e.g.

```
%%BeginProlog
/lilypond-datadir where {pop} {userdict /lilypond-datadir 
(/path/to/lilypond/build-without-extractpdfmark/out/share/lilypond/current) put 
} ifelse
%%BeginFont: Emmentaler-20
lilypond-datadir (/fonts/otf/emmentaler-20.otf) concatstrings (r) file .loadfont
%%EndFont
```

Then, `output-distance.py` invoked gs with commandline option
`-slilypond-datadir=input/regression/out-test-baseline/share/lilypond/current`.

It specified `lilypond-datadir` as relative path
from `/path/to/lilypond/build-without-extractpdfmark`.
However, current directory was
`/path/to/lilypond/build-without-extractpdfmark/input/regression/out-test-baseline/share/lilypond/current`
because 
http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=blob;f=scripts/build/output-distance.py;h=ba787931a5edb6f1f2f46e6c53e851241ba4b9d1;hb=HEAD#l648
changes current directory.
Since current directory was different from the origin of the relative path,
gs could not find the font file.

My patch shown above fixes the wrong relative path specification.

On the other hand, with extractpdfmark,
each eps file has absolute font path.
e.g.

```
%%BeginProlog
/lilypond-datadir where {pop} {userdict /lilypond-datadir 
(/path/to/lilypond/build-with-extractpdfmark/out/share/lilypond/current) put } 
ifelse
%%BeginFont: Emmentaler-20
(/path/to/lilypond/build-with-extractpdfmark/out/share/lilypond/current/fonts/otf/emmentaler-20.otf)
 (r) file .loadfont
%%EndFont
```

In this case,
even if `lilypond-datadir` is specified as such a wrong relative directory,
gs can find the font file.
However, I think absolute font path in eps is undesirable behavior.
So, I've created Issue 5467.
https://sourceforge.net/p/testlilyissues/issues/5467/
https://codereview.appspot.com/349110043/

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: I cannot run make check since Issue 5450: relocate.cc: Introduce new command `set?'

[Masamichi Hosoda]

>>> This is a gs 9.26 issue and I cannot see how this might be related to what 
>>> we have hit here, so maybe Hosoda-san will be abel to figure why make check 
>>> is breaking without extractpdfmark installed.
>>
>> If I understand correctly, this patch solves the error.
>>
>> ```
>> --- a/scripts/build/output-distance.py
>> +++ b/scripts/build/output-distance.py
>> @@ -652,7 +652,7 @@ class SignatureFileLink (FileLink):
>>  data_option = ''
>>  if options.local_data_dir:
>>  data_option = 
>> ('-slilypond-datadir=%s/share/lilypond/current '
>> -   % dir)
>> +   % (cur_dir + '/' + dir))
>>
> 
> Is dir guaranteed to be a relative path?

It seems that there is no guarantee.
But, `make check` invokes `output-distance` with relative paths.
http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=blob;f=GNUmakefile.in;h=da1f6f64e088756e07d20f8e1603ccc3a102053e;hb=HEAD#l340

Therefore, `dir` is also a relative path.
There seems to be no place invoking `output-distance` other than that.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: I cannot run make check since Issue 5450: relocate.cc: Introduce new command `set?'

[Masamichi Hosoda]

>>> Is dir guaranteed to be a relative path?
>>
>> It seems that there is no guarantee.
>> But, `make check` invokes `output-distance` with relative paths.
>> http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=blob;f=GNUmakefile.in;h=da1f6f64e088756e07d20f8e1603ccc3a102053e;hb=HEAD#l340
>>
>> Therefore, `dir` is also a relative path.
>> There seems to be no place invoking `output-distance` other than that.
> 
> Python has os.path.join that should allow to combine cur_dir with either
> relative or absolute paths.

Thank you for your advice.
But, Werner's Issue 5466 uses `abs_dir` by `os.path.abspath ()`.
So I've uploaded Patch Set 2 using `abs_dir`.
It requires Issue 5466.

https://sourceforge.net/p/testlilyissues/issues/5467/
https://codereview.appspot.com/349110043

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Please test new lilypond installers

[Masamichi Hosoda]

> $ uname -a
> FreeBSD freebsd-hyperv 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC  
> amd64

> ld-elf.so.1: Shared object "libm.so.4" not found, required by "lilypond"
> 
> Are there any further efforts anyone would like to see made here?

Maybe, installing compat6x package is required.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: build problem

[Masamichi Hosoda]

> Hello,
> 
> Since I can't get a successful doc-build on my Fedora 30 box, I'm
> unable to check doc-changes and merge translation into stable.
> 
> It's the same with extractpdfmark 1.0.2 1.0.3 and 1.1.0
> 
> log spits:
> 
> extractpdfmark -o ./out-www/collated-files.pdfmark
> ./out-www/collated-files.tmp.pdf
> extractpdfmark: error while loading shared libraries:
> libpoppler.so.78: cannot open shared object file: No such file or
> directory
> 
> And I won't try to package poppler-0.78. There are too many
> dependencies for me (Fedora is stuck to 0.73).

In Fedora 30, I suggest to use extractpdfmark from package,
rather than self-built extractpdfmark.
https://apps.fedoraproject.org/packages/extractpdfmark

If I understand correctly,
Fedora 30 has popler-0.73.0 package that has libpoppler.so.84.
https://koji.fedoraproject.org/koji/rpminfo?rpmID=17656985
Fedora 29 has popler-0.67.0 package that has libpoppler.so.78.
https://koji.fedoraproject.org/koji/rpminfo?rpmID=17657086

Did you build extractpdfmark on Fedora 29?
If so, it depends on libpoppler.so.78.
However, Fedora 30 does not have libpoppler.so.78.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: 2.91.84 - windows-only bugs

[Masamichi Hosoda]

> I ask because, in the german forum Arnold found a method to cure some
> windows-only bugs., about mis-predicted force and probably several
> assertion-failures:
> https://lilypondforum.de/index.php/topic,609.msg3463.html#msg3463

The patch is very interesting.

I've tried x87mant53.dll showed in the german forum.
It solved not only Issue 4943 but also Issue 4975.

I guess that these issues do not only occur on Windows,
but on all x86 32 bit platforms except Linux.
Therefore, I think that `defined (__MINGW32__)` is not necessary.

Re: Inline assembler fallback for _FPU_SETCW() missing in MINGW libraries (issue 577450043 by thomasmorle...@gmail.com)

[Masamichi Hosoda]

>>   -fexcess-precision=style
>>   This option allows further control over excess precision on
>>   machines where floating-point operations occur in a format
>>   with more precision or range than the IEEE standard and
>>   interchange floating-point types.  By default,
>>   -fexcess-precision=fast is in effect; this means that
>>   operations may be carried out in a wider precision than the
>>   types specified in the source if that would result in faster
>>   code, and it is unpredictable when rounding to the types
>>   specified in the source code takes place.
> 
> This sounds promising.

If I understand correctly, such compiler options are not promising.

Even if LilyPond C++ source codes are compiled with such option,
libguile without the option may calculate in a wider precision.
In this case, the floating point calculation
inside GUILE causes these issues.

Re: Inline assembler fallback for _FPU_SETCW() missing in MINGW libraries (issue 577450043 by thomasmorle...@gmail.com)

[Masamichi Hosoda]

>> This sounds promising.
>>
>>>   -fexcess-precision=standard is not implemented for languages
>>>   other than C.
>>
>> Never mind.
> 
> Hm.  Maybe
> 
> -mfpmath=sse
> 
> instead?  The problem with switching the whole FPU to reduced precision
> is that some library functions might not expect that, so it is making me
> queasy.  On the other hand, using SSE might have a negative performance?
> I just don't have a good idea what we are dealing with here.

In my experiment, `-fexcess-precision=standard` cannot be used for C++.

```
$ g++ -fexcess-precision=standard test.cc
cc1plus: sorry, unimplemented: -fexcess-precision=standard for C++

$ g++ --version
g++ (GCC) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


$
```

On the other hand, `-msse -mfpmath=sse` uses SSE,
which can only perform single-precision floating-point calculation.
SSE is too low precision.

`-msse2 -mfpmath=sse` uses SSE2,
which can perform double-precision floating-point calculation.
Precision is sufficient, but older 32-bit x86 CPUs do not have SSE2.

Re: Inline assembler fallback for _FPU_SETCW() missing in MINGW libraries (issue 577450043 by thomasmorle...@gmail.com)

[Masamichi Hosoda]

>>> You provided a lot of good information in your post, but the
>>> conclusion was not entirely clear.
>>> Are you suggesting requiring SSE2 at this time?
>> 
>> Yes.  It appears to get used anyway for 64bit executables, and it seems
>> safe enough to demand it for 32bit executables.
> 
> +1
> far better than inline asm poking control registers

The cause of the issues is that
the unexpected precision conversions raises small difference
in floating-point calculation.

If libguile uses x87 FPU, high-precision calculation
and low-precision calculation coexist inside GUILE,
and precision conversion occurs between them.

Even if C++ uses SSE2,
libguile and other shared libraries still may use x87 FPU.

I wonder this may cause problems.

Re: Inline assembler fallback for _FPU_SETCW() missing in MINGW libraries (issue 577450043 by thomasmorle...@gmail.com)

[Masamichi Hosoda]

> Nothing?  I am currently proposing that we compile with
> 
> -mfpmath=sse -msse2
> 
> as options which should shift arithmetic off to the SSE2 instruction set
> which doesn't work with 80-bit arithmetic.  That would be a default way
> of compilation.

I've created a patch for master.

https://sourceforge.net/p/testlilyissues/issues/5725/
https://codereview.appspot.com/565600045/

Re: Inline assembler fallback for _FPU_SETCW() missing in MINGW libraries (issue 577450043 by thomasmorle...@gmail.com)

[Masamichi Hosoda]

> We currently have the problem that the compiler used in GUB for
> compiling 32bit binaries gets an internal compiler fault for those
> options.
> 
> We'll need to figure out whether a newer compiler does the trick, and if
> it does, update GUB.  Or find a different way of proceeding.
> 
> I'll see whether I can convince my current compilers to generate 32bit
> code and see whether those are currently up to using those options.  If
> current compilers don't want them, we'll need to revert to a different
> plan.

It seems that static cast from `unsigned long long` to `double`
by x86 SSE2 raises the internal compile error.
However, static cast from `signed long long` to `double`
does not raise the errir.
I think it can be a workaround for rational.cc.

i.e.
First, static cast from `unsigned long long` to `signed long long`
Then, static cast from `singed long long` to `double`

Re: Inline assembler fallback for _FPU_SETCW() missing in MINGW libraries (issue 577450043 by thomasmorle...@gmail.com)

[Masamichi Hosoda]

>>> We currently have the problem that the compiler used in GUB for
>>> compiling 32bit binaries gets an internal compiler fault for those
>>> options.
>>> 
>>> We'll need to figure out whether a newer compiler does the trick, and if
>>> it does, update GUB.  Or find a different way of proceeding.
>>> 
>>> I'll see whether I can convince my current compilers to generate 32bit
>>> code and see whether those are currently up to using those options.  If
>>> current compilers don't want them, we'll need to revert to a different
>>> plan.
>>
>> It seems that static cast from `unsigned long long` to `double`
>> by x86 SSE2 raises the internal compile error.
>> However, static cast from `signed long long` to `double`
>> does not raise the errir.
>> I think it can be a workaround for rational.cc.
>>
>> i.e.
>> First, static cast from `unsigned long long` to `signed long long`
>> Then, static cast from `singed long long` to `double`
> 
> Oh wow.  I would never have thought one could identify something as
> specific.  I think I have convinced my system to build a 32bit Guile,
> but have problems convincing LilyPond to do the same.  So I cannot help
> with debugging this situation yet or even finding out whether it
> persists into newer compiler versions.

I've attached the workaround patch for stable/2.20 branch.
The results of my experiment is here.

Current stable/2.20 bcad34e31d7866eb126e73abc89eeeb01faf081f without the patch:

```
$ ~gub/gub/target/darwin-x86/root/usr/cross/bin/i686-apple-darwin8-g++ -c 
-msse2 -mfpmath=sse -I include -I .. rational.cc
rational.cc: In member function 'double Rational::to_double() const':
rational.cc:48:1: internal compiler error: in gen_reg_rtx, at emit-rtl.c:838
 }
 ^
0x773934 gen_reg_rtx(machine_mode)
/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/emit-rtl.c:838
0xc53c53 gen_split_4130(rtx_def*, rtx_def**)

/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/config/i386/sse.md:884
0x7772a7 try_split(rtx_def*, rtx_def*, int)
/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/emit-rtl.c:3444
0x8feb71 split_insn
/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/recog.c:2897
0x9034e4 split_all_insns()
/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/recog.c:2987
0x903578 rest_of_handle_split_after_reload
/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/recog.c:3938
0x903578 execute
/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/recog.c:3967
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
$ ~gub/gub/target/linux-x86/root/usr/cross/bin/i686-linux-g++ -c -msse2 
-mfpmath=sse -I include -I .. rational.cc
$ ~gub/gub/target/freebsd-x86/root/usr/cross/bin/i686-freebsd6-g++ -c -msse2 
-mfpmath=sse -I include -I .. rational.cc
$
```

Current stable/2.20 bcad34e31d7866eb126e73abc89eeeb01faf081f with the patch:

```
$ ~gub/gub/target/darwin-x86/root/usr/cross/bin/i686-apple-darwin8-g++ -c 
-msse2 -mfpmath=sse -I include -I .. rational.cc
rational.cc:52:5: warning: floating constant exceeds range of 'double' 
[-Woverflow]
 return -HUGE_VAL;
 ^
rational.cc:54:5: warning: floating constant exceeds range of 'double' 
[-Woverflow]
 return HUGE_VAL;
 ^
$ ~gub/gub/target/linux-x86/root/usr/cross/bin/i686-linux-g++ -c -msse2 
-mfpmath=sse -I include -I .. rational.cc
$ ~gub/gub/target/freebsd-x86/root/usr/cross/bin/i686-freebsd6-g++ -c -msse2 
-mfpmath=sse -I include -I .. rational.cc
$
```
>From 2a3816e49067e026c7180dc6a3b2d5614d9c20d6 Mon Sep 17 00:00:00 2001
From: Masamichi Hosoda 
Date: Tue, 4 Feb 2020 23:30:29 +0900
Subject: [PATCH] Add workaround for avoiding GUB darwin-x86 error

In GUB, g++ 4.9.4 for darwin-x86 (macOS x86),
it seems that static cast from  `unsigned long long` to `double`
by x86 SSE2 raises an internal compile error.
However, static cast from `signed long long` to `double`
does not raise the error.
So we use it for a workaround.

i.e.
First, static cast from `unsigned long long` to `signed long long`.
Then, static cast from `singed long long` to `double`.
---
 flower/rational.cc | 16 
 1 file changed, 16 insertions(+)

diff --git a/flower/rational.cc b/flower/rational.cc
index 559e1646a0..9435edbb8f 100644
--- a/flower/rational.cc
+++ b/flower/rational.cc
@@ -31,7 +31,23 @@ double
 Rational::to_double () const
 {
   if (sign_ == -1 || sign_ == 1 || sign_ == 0)
+// FIXME: workaround: In GUB, g++ 4.9.4 for darwin-x86,
+// it seems that static cast from `unsigned long long` to `double`
+// by x86 SSE2 raises an internal compile error.
+// However, static cast from `si

Re: Inline assembler fallback for _FPU_SETCW() missing in MINGW libraries (issue 577450043 by thomasmorle...@gmail.com)

[Masamichi Hosoda]

>>>>> We currently have the problem that the compiler used in GUB for
>>>>> compiling 32bit binaries gets an internal compiler fault for those
>>>>> options.
>>>>> 
>>>>> We'll need to figure out whether a newer compiler does the trick, and if
>>>>> it does, update GUB.  Or find a different way of proceeding.
>>>>> 
>>>>> I'll see whether I can convince my current compilers to generate 32bit
>>>>> code and see whether those are currently up to using those options.  If
>>>>> current compilers don't want them, we'll need to revert to a different
>>>>> plan.
>>>>
>>>> It seems that static cast from `unsigned long long` to `double`
>>>> by x86 SSE2 raises the internal compile error.
>>>> However, static cast from `signed long long` to `double`
>>>> does not raise the errir.
>>>> I think it can be a workaround for rational.cc.
>>>>
>>>> i.e.
>>>> First, static cast from `unsigned long long` to `signed long long`
>>>> Then, static cast from `singed long long` to `double`
>>> 
>>> Oh wow.  I would never have thought one could identify something as
>>> specific.  I think I have convinced my system to build a 32bit Guile,
>>> but have problems convincing LilyPond to do the same.  So I cannot help
>>> with debugging this situation yet or even finding out whether it
>>> persists into newer compiler versions.
>>
>> I've attached the workaround patch for stable/2.20 branch.
>> The results of my experiment is here.
> 
> 
>>>From 2a3816e49067e026c7180dc6a3b2d5614d9c20d6 Mon Sep 17 00:00:00 2001
>> From: Masamichi Hosoda 
>> Date: Tue, 4 Feb 2020 23:30:29 +0900
>> Subject: [PATCH] Add workaround for avoiding GUB darwin-x86 error
>>
>> In GUB, g++ 4.9.4 for darwin-x86 (macOS x86),
>> it seems that static cast from  `unsigned long long` to `double`
>> by x86 SSE2 raises an internal compile error.
>> However, static cast from `signed long long` to `double`
>> does not raise the error.
>> So we use it for a workaround.
>>
>> i.e.
>> First, static cast from `unsigned long long` to `signed long long`.
>> Then, static cast from `singed long long` to `double`.
>> ---
>>  flower/rational.cc | 16 
>>  1 file changed, 16 insertions(+)
>>
>> diff --git a/flower/rational.cc b/flower/rational.cc
>> index 559e1646a0..9435edbb8f 100644
>> --- a/flower/rational.cc
>> +++ b/flower/rational.cc
>> @@ -31,7 +31,23 @@ double
>>  Rational::to_double () const
>>  {
>>if (sign_ == -1 || sign_ == 1 || sign_ == 0)
>> +// FIXME: workaround: In GUB, g++ 4.9.4 for darwin-x86,
>> +// it seems that static cast from `unsigned long long` to `double`
>> +// by x86 SSE2 raises an internal compile error.
>> +// However, static cast from `signed long long` to `double`
>> +// does not raise the error.
>> +// So we use it for a workaround.
>> +#if defined (__i386__) && defined (__APPLE__) && \
>> +  defined (__SSE2_MATH__) && __GNUC__ < 5
> 
> Wouldn't the same problem occur on Windows?  We have 32bit executables
> there as well.  Or is the compiler version we use there not afflicted?

Sorry, I forgot MinGW.
The internal compiler error seems to raise only in darwin-x86.
I don't have newer g++ for darwin-x86, so I'm not sure
if it was fixed in the newer g++.

without the patch:

```
$ ~gub/gub/target/darwin-x86/root/usr/cross/bin/i686-apple-darwin8-g++ -c 
-msse2 -mfpmath=sse -I include -I .. rational.cc
rational.cc:36:5: warning: floating constant exceeds range of 'double' 
[-Woverflow]
 return -HUGE_VAL;
 ^
rational.cc:38:5: warning: floating constant exceeds range of 'double' 
[-Woverflow]
 return HUGE_VAL;
 ^
rational.cc: In member function 'double Rational::to_double() const':
rational.cc:43:1: internal compiler error: in gen_reg_rtx, at emit-rtl.c:838
 }
 ^
0x773934 gen_reg_rtx(machine_mode)
/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/emit-rtl.c:838
0xc53c53 gen_split_4130(rtx_def*, rtx_def**)

/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/config/i386/sse.md:884
0x7772a7 try_split(rtx_def*, rtx_def*, int)
/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/emit-rtl.c:3444
0x8feb71 split_insn
/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/recog.c:2897
0x9034e4 split_all_insns()
/home/gub/gub/target/darwin-x86/src/cross/gcc-4.9.4/gcc/recog.c:2987
0x903578 rest_of_handle_split_af

Re: Inline assembler fallback for _FPU_SETCW() missing in MINGW libraries (issue 577450043 by thomasmorle...@gmail.com)

[Masamichi Hosoda]

>> $ ~gub/gub/target/linux-x86/root/usr/cross/bin/i686-linux-g++ -c -msse2 
>> -mfpmath=sse -I include -I .. rational.cc
>> $ ~gub/gub/target/freebsd-x86/root/usr/cross/bin/i686-freebsd6-g++ -c -msse2 
>> -mfpmath=sse -I include -I .. rational.cc
> 
> So this only affects darwin-x86 which confirms my observation that
> linux-x86 still works. AFAICS darwin-x86 never had a fpu_control.h, so
> I'd propose to drop -msse2 -mfpmath=sse for i686-apple-darwin8.
> This unblocks the release and the situation on darwin-x86 remains the
> same as before, but we have the fix for linux-x86 and mingw.
> 
> My fear is that not only rational.cc is affected, but also many other
> files. Did you test if a full compile works? If no, I'm against adding
> workarounds for all files that suffer from the compiler error.

I didn't test full compiler works.
However, If I understand correctly,
it seems that static cast from `unsigned long long` to `double`
is only in rational.cc.

Re: NR: no working links in included PDFs

[Masamichi Hosoda]

Sorry, I'm too late.

As with the Java part of `pax`,
it may be possible to extract annotations from PDFs with a C/C++ program.
But, it looks like `latex-pax` is adding the extracted annotations
to the PDF using `pax.sty`.
The `pax.sty` is for LaTeX, and it looks like it is for pdfLaTeX only.
It would be quite difficult to get it to work with Texinfo on XeTeX/pdfTeX.

Alternatively, you could ask the xdvipdfmx developer
to add an option to keep the links and the annotations.

> Look at the documention of `\with-url` in the PDF version of the NR:
> The link in the included snippet (to 'lilypond.org') doesn't work.
> This is a limitation of pdfTeX (or XeTeX), which drops all links and
> annotations for included PDF files.
> 
> Masamichi-san, is it possible to extend `extractpdfmark` to cover
> this, too?  Looking around it seems that the tool 'pax'
> ('PDFAnnotExtractor') in TeXLive can do that.  However, it is based on
> Java...
> 
>   https://github.com/bastien-roucaries/latex-pax
> 
> The reason why I ask is the planned inclusion of the Visual Grob Index
> in the NR – all grobs shown have links to the corresponding
> documentation.  If these links are lost the value of the index is
> reduced enormously, and it would be probably better to not include it
> at all but to have a stand-alone, simple, two-page document instead.

Re: NR: no working links in included PDFs

[Masamichi Hosoda]

>> As with the Java part of `pax`, it may be possible to extract
>> annotations from PDFs with a C/C++ program.  But, it looks like
>> `latex-pax` is adding the extracted annotations to the PDF using
>> `pax.sty`.  The `pax.sty` is for LaTeX, and it looks like it is for
>> pdfLaTeX only.  It would be quite difficult to get it to work with
>> Texinfo on XeTeX/pdfTeX.
> 
> Oh, I don't mean that we should exactly follow the `pax` algorithm.  I
> only mentioned it because it provides a solution.
> 
> Isn't it possible to add links with the `pdfmark` operator in the same
> way as is already done for annotations by `extractpdfmark`?  In
> Adobe's pdfmark reference manual (`5150.Pdfmark.pdf`) I see a section
> 2.1.2 called 'Links', which appears to be exactly what's needed.

The PDF output by pdfTeX/XeTeX has no information about
the file name or position (page number, x-y coordinates, size, etc.)
of the PDFs it includes.
It means that even if we extract the link information from the included PDFs,
we don't know where to apply it in the positions
of the PDF output by pdfTeX/XeTeX.

`pax.sty` seems to recognize the positions of the PDFs included
by the LaTeX layer and reproduce the links.
If something similar to this can be done with Texinfo,
link reproduction can be realized.
However, I have no idea how to do this in Texinfo.

Re: gs 10.04 will break doc builds again

[Masamichi Hosoda]

> `gs` compiled from the current 'master' branch of the 'ghostpdl'
> repository makes a doc build fail if 'extractpdfmark' gets used: In
> the created `notation.pdf`, Emmentaler fonts are missing in all
> included snippets.
> 
[...snip...]
> 
> Sigh.  Masamichi-san, please have a look!

It seems that the Ghostscript generated PDF is broken
regardless of LilyPond or extractpdfmark.

I tested Ghostscript GIT PRERELEASE at
commit 99b48f1992844e30ab4b05dbb2764ba99f7b66f4
Date:   Mon Feb 19 08:33:23 2024 +
.
Here's results.

```
$ cat test.ps
/NimbusSans-Regular findfont 14 scalefont setfont
50 100 moveto
(Hello World) show

$ ./gs -q -dBATCH -dNOPAUSE -sDEVICE=pdfwrite -sOutputFile=test.pdf test.ps

$ qpdf --check test.pdf
checking test.pdf
PDF Version: 1.7
File is not encrypted
WARNING: test.pdf: object 6/0: error reading object: integer out of range 
converting 889058230272 from a 8-byte signed type to a 4-byte signed type
WARNING: test.pdf (object 9 0, offset 100): stream dictionary lacks /Length key
WARNING: test.pdf (object 9 0, offset 146): attempting to recover stream length
WARNING: test.pdf (object 9 0, offset 146): recovered stream length: 82
File is not linearized
qpdf: operation succeeded with warnings

$
```

lilypad.exe Japanese resource (menu, dialog, etc.)

[Masamichi Hosoda]

Hi.

Japanese resource (menu, dialog, etc.) of
official pre-built lilypad.exe is broken.

I installed lilypond-2.19.13-1.mingw.exe
(download from 
http://download.linuxaudio.org/lilypond/binaries/mingw/lilypond-2.19.13-1.mi
ngw.exe)
and started lilypad.exe.
Then, all the Japanese character strings are garbled.

So, I tried to build lilypad.exe from source
(https://github.com/gperciva/lilypad).
As a result, my built lilypad.exe is not garbled.

Which windres (resource compiler) version do you use
to built official pre-built lilypad.exe?
Since old windres (less than ver 2.18) is not supporting codepage,
it may cause garbled characters.



___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: lilypad.exe Japanese resource (menu, dialog, etc.)

[Masamichi HOSODA]

Your attached zipped lilypad.exe works fine.
Thanks.

From: "Phil Holmes" 
Subject Re: lilypad.exe Japanese resource (menu, dialog, etc.)
Date Fri, 5 Sep 2014 13:52:07 +0100
Message-ID: <40A388970C6540E08618DFC1C4932F1E@Advent>

> Now with the attachment :-(
> 
> --
> Phil Holmes
> 
> 
> - Original Message ----- 
> From: "Phil Holmes" 
> To: ; "Masamichi Hosoda"
> 
> Sent: Friday, September 05, 2014 1:19 PM
> Subject: Re: lilypad.exe Japanese resource (menu, dialog, etc.)
> 
> 
>> - Original Message - 
>> From: "Masamichi Hosoda" 
>> To: 
>> Sent: Friday, September 05, 2014 12:18 PM
>> Subject: lilypad.exe Japanese resource (menu, dialog, etc.)
>>
>>
>>> Hi.
>>>
>>> Japanese resource (menu, dialog, etc.) of
>>> official pre-built lilypad.exe is broken.
>>>
>>> I installed lilypond-2.19.13-1.mingw.exe
>>> (download from
>>> http://download.linuxaudio.org/lilypond/binaries/mingw/lilypond-2.19.13-1.mi
>>> ngw.exe)
>>> and started lilypad.exe.
>>> Then, all the Japanese character strings are garbled.
>>>
>>> So, I tried to build lilypad.exe from source
>>> (https://github.com/gperciva/lilypad).
>>> As a result, my built lilypad.exe is not garbled.
>>>
>>> Which windres (resource compiler) version do you use
>>> to built official pre-built lilypad.exe?
>>> Since old windres (less than ver 2.18) is not supporting codepage,
>>> it may cause garbled characters.
>>
>>
>> I'll need to check on my other machine.  Could you tell me whether the
>> attached zipped lilypad.exe is OK?  This was compiled with gcc on my
>> Windows machine.
>>
>> --
>> Phil Holmes
>>
>> ___
>> lilypond-devel mailing list
>> lilypond-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/lilypond-devel
>> 

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: lilypad.exe Japanese resource (menu, dialog, etc.)

[Masamichi HOSODA]

I noticed another problem in your attached zipped lilypad.exe.
Your lilypad.exe is not UNICODE version but ANSI version.
The file name of ANSI version lilypad should be "lilypad-ascii.exe".

The ANSI version lilypad cannot display Japanese characters
in default font (Courier New).
The UNICODE version lilypad can do it.

It is better to run "make clean", before building "lilypad.exe".
Otherwise, ANSI version objects may be mixed to UNICODE version "lilypad.exe".

From: "Phil Holmes" 
Subject Re: lilypad.exe Japanese resource (menu, dialog, etc.)
Date Fri, 5 Sep 2014 13:52:07 +0100
Message-ID: <40A388970C6540E08618DFC1C4932F1E@Advent>

> Now with the attachment :-(
> 
> --
> Phil Holmes
> 
> 
> - Original Message - 
> From: "Phil Holmes" 
> To: ; "Masamichi Hosoda"
> 
> Sent: Friday, September 05, 2014 1:19 PM
> Subject: Re: lilypad.exe Japanese resource (menu, dialog, etc.)
> 
> 
>> - Original Message - 
>> From: "Masamichi Hosoda" 
>> To: 
>> Sent: Friday, September 05, 2014 12:18 PM
>> Subject: lilypad.exe Japanese resource (menu, dialog, etc.)
>>
>>
>>> Hi.
>>>
>>> Japanese resource (menu, dialog, etc.) of
>>> official pre-built lilypad.exe is broken.
>>>
>>> I installed lilypond-2.19.13-1.mingw.exe
>>> (download from
>>> http://download.linuxaudio.org/lilypond/binaries/mingw/lilypond-2.19.13-1.mi
>>> ngw.exe)
>>> and started lilypad.exe.
>>> Then, all the Japanese character strings are garbled.
>>>
>>> So, I tried to build lilypad.exe from source
>>> (https://github.com/gperciva/lilypad).
>>> As a result, my built lilypad.exe is not garbled.
>>>
>>> Which windres (resource compiler) version do you use
>>> to built official pre-built lilypad.exe?
>>> Since old windres (less than ver 2.18) is not supporting codepage,
>>> it may cause garbled characters.
>>
>>
>> I'll need to check on my other machine.  Could you tell me whether the
>> attached zipped lilypad.exe is OK?  This was compiled with gcc on my
>> Windows machine.
>>
>> --
>> Phil Holmes
>>
>> ___
>> lilypond-devel mailing list
>> lilypond-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/lilypond-devel

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: lilypad.exe Japanese resource (menu, dialog, etc.)

[Masamichi HOSODA]

I noticed another problem in your lilypad.exe.
Your lilypad.exe is not UNICODE version but ANSI version.
The file name of ANSI version lilypad should be "lilypad-ascii.exe".

The ANSI version lilypad cannot display Japanese characters
in default font (Courier New).
The UNICODE version lilypad can do it.

It is better to run "make clean", before building "lilypad.exe".
Otherwise, ANSI version objects may be mixed to UNICODE version "lilypad.exe".

From: Masamichi HOSODA 
Subject Re: lilypad.exe Japanese resource (menu, dialog, etc.)
Date Fri, 05 Sep 2014 22:45:42 +0900 (JST)
Message-ID: <20140905.224542.1368425677750117260.truer...@trueroad.jp>

> Your attached zipped lilypad.exe works fine.
> Thanks.
> 
> From: "Phil Holmes" 
> Subject Re: lilypad.exe Japanese resource (menu, dialog, etc.)
> Date Fri, 5 Sep 2014 13:52:07 +0100
> Message-ID: <40A388970C6540E08618DFC1C4932F1E@Advent>
> 
>> Now with the attachment :-(
>> 
>> --
>> Phil Holmes
>> 
>> 
>> - Original Message - 
>> From: "Phil Holmes" 
>> To: ; "Masamichi Hosoda"
>> 
>> Sent: Friday, September 05, 2014 1:19 PM
>> Subject: Re: lilypad.exe Japanese resource (menu, dialog, etc.)
>> 
>> 
>>> - Original Message - 
>>> From: "Masamichi Hosoda" 
>>> To: 
>>> Sent: Friday, September 05, 2014 12:18 PM
>>> Subject: lilypad.exe Japanese resource (menu, dialog, etc.)
>>>
>>>
>>>> Hi.
>>>>
>>>> Japanese resource (menu, dialog, etc.) of
>>>> official pre-built lilypad.exe is broken.
>>>>
>>>> I installed lilypond-2.19.13-1.mingw.exe
>>>> (download from
>>>> http://download.linuxaudio.org/lilypond/binaries/mingw/lilypond-2.19.13-1.mi
>>>> ngw.exe)
>>>> and started lilypad.exe.
>>>> Then, all the Japanese character strings are garbled.
>>>>
>>>> So, I tried to build lilypad.exe from source
>>>> (https://github.com/gperciva/lilypad).
>>>> As a result, my built lilypad.exe is not garbled.
>>>>
>>>> Which windres (resource compiler) version do you use
>>>> to built official pre-built lilypad.exe?
>>>> Since old windres (less than ver 2.18) is not supporting codepage,
>>>> it may cause garbled characters.
>>>
>>>
>>> I'll need to check on my other machine.  Could you tell me whether the
>>> attached zipped lilypad.exe is OK?  This was compiled with gcc on my
>>> Windows machine.
>>>
>>> --
>>> Phil Holmes
>>>
>>> ___
>>> lilypond-devel mailing list
>>> lilypond-devel@gnu.org
>>> https://lists.gnu.org/mailman/listinfo/lilypond-devel
>>> 
> 
> ___
> lilypond-devel mailing list
> lilypond-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/lilypond-devel

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: LilyPad development

[Masamichi HOSODA]

I am interested in lilypad (Windows) development. 
Thank you for merging my pull requests.

> I've updated the CG with what I believe are the correct steps for 
> developments to LilyPad.  The new text is in master, but not yet on the 
> website.  I'd be interested in comments from anyone with experience of 
> making changes to LilyPad: what Ive written is based on my memory and 
> forensics of the GUB code and file time changes.  Also: anyone interested 
> in a convoluted development mechanism might be interested.
> 
> If you can't find/see the changes, please shout and I'll post the text 
> directly here.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

> I'm trying to update GCC on GUB and have a new virtual machine with 
> updated versions.  I was having problems getting MPFR to build, but it 
> looks like I'ev fixed that with the new VM.  However, it looks to me like 
> GCC 4.8.2 has a new dependency on MPC that older versions did not: there 
> appears no mention of MPC as a package in the current version of GUB.  My 
> most recent failure says this:
> 
> checking for the correct version of gmp.h... buggy but acceptable
> checking for the correct version of mpfr.h... yes
> checking for the correct version of mpc.h... no
> configure: error: Building GCC requires GMP 4.2+, MPFR 2.4.0+ and MPC 
> 0.8.0+.
> 
> So it looks like we need to add a dependency on MPC.  Problem is, I don't 
> have a clue how to do this.  I could try some stuff, but can anyone here 
> help with how this dependency should be added to GUB?
> 
> TIA.
> 
> 
> ___
> lilypond-devel mailing list
> lilypond-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/lilypond-devel

Did the error occur, while building cross/gcc?
If that is right, GMP/MPFR/MPC may be required for not GUB but host OS.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

> - Original Message - 
> From: "Masamichi HOSODA" 
> To: 
> Sent: Thursday, October 23, 2014 12:07 PM
> Subject: Re: GUB and mpfr/mpc
> 
> 
>>> I'm trying to update GCC on GUB and have a new virtual machine with
>>> updated versions.  I was having problems getting MPFR to build, but it
>>> looks like I'ev fixed that with the new VM.  However, it looks to me
>>> like
>>> GCC 4.8.2 has a new dependency on MPC that older versions did not:
>>> there
>>> appears no mention of MPC as a package in the current version of GUB.
>>> My
>>> most recent failure says this:
>>>
>>> checking for the correct version of gmp.h... buggy but acceptable
>>> checking for the correct version of mpfr.h... yes
>>> checking for the correct version of mpc.h... no
>>> configure: error: Building GCC requires GMP 4.2+, MPFR 2.4.0+ and MPC
>>> 0.8.0+.
>>>
>>> So it looks like we need to add a dependency on MPC.  Problem is, I
>>> don't
>>> have a clue how to do this.  I could try some stuff, but can anyone
>>> here
>>> help with how this dependency should be added to GUB?
>>>
>>> TIA.
>>
>> Did the error occur, while building cross/gcc?
>> If that is right, GMP/MPFR/MPC may be required for not GUB but host
>> OS.
> 
> 
> The error is in the linux-86/log/cross/gcc-core.log file, so I assume
> the answer is yes.  However, my understanding is that the only package
> that should need separately installing on plain Ubuntu/linux is git:
> the rest should be downloaded and built by make bootstrap.
> 
> --
> Phil Holmes 
> 

cross/gcc-core is built by host OS native gcc.
Would you try the following commands on your Ubuntu?

sudo apt-get update
sudo apt-get install libgmp-dev
sudo apt-get install libmpfr-dev
sudo apt-get install libmpc-dev

In my environment, I have succeeded in building of linux-x86::cross/gcc-core.
However, I have still failed in building of linux-x86::glibc-core. 

https://github.com/trueroad/gub/tree/binutils-2.24

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

> - Original Message - 
> From: "Masamichi HOSODA" 
> To: 
> Cc: ; 
> Sent: Saturday, October 25, 2014 2:58 PM
> Subject: Re: GUB and mpfr/mpc
> 
> 
>>> - Original Message - 
>>> From: "Masamichi HOSODA" 
>>> To: 
>>> Sent: Thursday, October 23, 2014 12:07 PM
>>> Subject: Re: GUB and mpfr/mpc
>>>
>>>
>>>>> I'm trying to update GCC on GUB and have a new virtual machine with
>>>>> updated versions.  I was having problems getting MPFR to build, but it
>>>>> looks like I'ev fixed that with the new VM.  However, it looks to me
>>>>> like
>>>>> GCC 4.8.2 has a new dependency on MPC that older versions did not:
>>>>> there
>>>>> appears no mention of MPC as a package in the current version of GUB.
>>>>> My
>>>>> most recent failure says this:
>>>>>
>>>>> checking for the correct version of gmp.h... buggy but acceptable
>>>>> checking for the correct version of mpfr.h... yes
>>>>> checking for the correct version of mpc.h... no
>>>>> configure: error: Building GCC requires GMP 4.2+, MPFR 2.4.0+ and MPC
>>>>> 0.8.0+.
>>>>>
>>>>> So it looks like we need to add a dependency on MPC.  Problem is, I
>>>>> don't
>>>>> have a clue how to do this.  I could try some stuff, but can anyone
>>>>> here
>>>>> help with how this dependency should be added to GUB?
>>>>>
>>>>> TIA.
>>>>
>>>> Did the error occur, while building cross/gcc?
>>>> If that is right, GMP/MPFR/MPC may be required for not GUB but host
>>>> OS.
>>>
>>>
>>> The error is in the linux-86/log/cross/gcc-core.log file, so I assume
>>> the answer is yes.  However, my understanding is that the only package
>>> that should need separately installing on plain Ubuntu/linux is git:
>>> the rest should be downloaded and built by make bootstrap.
>>>
>>> --
>>> Phil Holmes
>>>
>>
>> cross/gcc-core is built by host OS native gcc.
>> Would you try the following commands on your Ubuntu?
>>
>> sudo apt-get update
>> sudo apt-get install libgmp-dev
>> sudo apt-get install libmpfr-dev
>> sudo apt-get install libmpc-dev
>>
>> In my environment, I have succeeded in building of
>> linux-x86::cross/gcc-core.
>> However, I have still failed in building of linux-x86::glibc-core.
>>
>> https://github.com/trueroad/gub/tree/binutils-2.24
>>
> 
> apt-get update ran fine.  However, I get this for libgmp:
> 
> gub@gub-V2:~/gub$ sudo apt-get install libgmp-dev
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> E: Couldn't find package libgmp-dev
> 
> --
> Phil Holmes 
> 

libgmp-dev package exists after Ubuntu 12.04.
http://packages.ubuntu.com/search?keywords=libgmp-dev&searchon=names&suite=all§ion=all

My environment is Ubuntu 14.04LTS amd64.
Are you using Ubuntu 10.04?

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

How about this patch?

--- librestrict/xstatconv.c.org 2014-10-19 09:52:52.951262300 +0900
+++ librestrict/xstatconv.c 2014-11-01 20:26:35.734854500 +0900
@@ -48,20 +48,16 @@
   {
struct stat *buf = ubuf;
 
+   /* zero clear */
+   memset(buf, 0, sizeof(*buf));
/* Convert to current kernel version of `struct stat'.  */
buf->st_dev = kbuf->st_dev;
-#ifdef _HAVE_STAT___PAD1
-   buf->__pad1 = 0;
-#endif
buf->st_ino = kbuf->st_ino;
buf->st_mode = kbuf->st_mode;
buf->st_nlink = kbuf->st_nlink;
buf->st_uid = kbuf->st_uid;
buf->st_gid = kbuf->st_gid;
buf->st_rdev = kbuf->st_rdev;
-#ifdef _HAVE_STAT___PAD2
-   buf->__pad2 = 0;
-#endif
buf->st_size = kbuf->st_size;
buf->st_blksize = kbuf->st_blksize;
buf->st_blocks = kbuf->st_blocks;
@@ -77,21 +73,6 @@
buf->st_mtime = kbuf->st_mtime;
buf->st_ctime = kbuf->st_ctime;
 #endif
-#ifdef _HAVE_STAT___UNUSED1
-   buf->__unused1 = 0;
-#endif
-#ifdef _HAVE_STAT___UNUSED2
-   buf->__unused2 = 0;
-#endif
-#ifdef _HAVE_STAT___UNUSED3
-   buf->__unused3 = 0;
-#endif
-#ifdef _HAVE_STAT___UNUSED4
-   buf->__unused4 = 0;
-#endif
-#ifdef _HAVE_STAT___UNUSED5
-   buf->__unused5 = 0;
-#endif
   }
   break;
 
@@ -116,11 +97,10 @@
   {
struct stat64 *buf = ubuf;
 
+   /* zero clear */
+   memset(buf, 0, sizeof(*buf));
/* Convert to current kernel version of `struct stat64'.  */
buf->st_dev = kbuf->st_dev;
-#ifdef _HAVE_STAT64___PAD1
-   buf->__pad1 = 0;
-#endif
buf->st_ino = kbuf->st_ino;
 #ifdef _HAVE_STAT64___ST_INO
buf->__st_ino = kbuf->st_ino;
@@ -130,9 +110,6 @@
buf->st_uid = kbuf->st_uid;
buf->st_gid = kbuf->st_gid;
buf->st_rdev = kbuf->st_rdev;
-#ifdef _HAVE_STAT64___PAD2
-   buf->__pad2 = 0;
-#endif
buf->st_size = kbuf->st_size;
buf->st_blksize = kbuf->st_blksize;
buf->st_blocks = kbuf->st_blocks;
@@ -148,21 +125,6 @@
buf->st_mtime = kbuf->st_mtime;
buf->st_ctime = kbuf->st_ctime;
 #endif
-#ifdef _HAVE_STAT64___UNUSED1
-   buf->__unused1 = 0;
-#endif
-#ifdef _HAVE_STAT64___UNUSED2
-   buf->__unused2 = 0;
-#endif
-#ifdef _HAVE_STAT64___UNUSED3
-   buf->__unused3 = 0;
-#endif
-#ifdef _HAVE_STAT64___UNUSED4
-   buf->__unused4 = 0;
-#endif
-#ifdef _HAVE_STAT64___UNUSED5
-   buf->__unused5 = 0;
-#endif
   }
   break;
 
@@ -185,12 +147,11 @@
 {
 case _STAT_VER_LINUX:
   {
+   /* zero clear */
+   memset(buf, 0, sizeof(*buf));
/* Convert current kernel version of `struct stat64' to
`struct stat'.  */
buf->st_dev = kbuf->st_dev;
-#ifdef _HAVE_STAT___PAD1
-   buf->__pad1 = 0;
-#endif
 #ifdef _HAVE_STAT64___ST_INO
 # if __ASSUME_ST_INO_64_BIT == 0
if (kbuf->st_ino == 0)
@@ -220,9 +181,6 @@
buf->st_uid = kbuf->st_uid;
buf->st_gid = kbuf->st_gid;
buf->st_rdev = kbuf->st_rdev;
-#ifdef _HAVE_STAT___PAD2
-   buf->__pad2 = 0;
-#endif
buf->st_size = kbuf->st_size;
/* Check for overflow.  */
if (sizeof (buf->st_size) != sizeof (kbuf->st_size)
@@ -253,21 +211,6 @@
buf->st_ctime = kbuf->st_ctime;
 #endif
 
-#ifdef _HAVE_STAT___UNUSED1
-   buf->__unused1 = 0;
-#endif
-#ifdef _HAVE_STAT___UNUSED2
-   buf->__unused2 = 0;
-#endif
-#ifdef _HAVE_STAT___UNUSED3
-   buf->__unused3 = 0;
-#endif
-#ifdef _HAVE_STAT___UNUSED4
-   buf->__unused4 = 0;
-#endif
-#ifdef _HAVE_STAT___UNUSED5
-   buf->__unused5 = 0;
-#endif
   }
   break;
 


> I get:
> 
> ~/gub/target/tools/build/librestrict-1.9.a$ gcc -W -Wall
> -fno-stack-protector -I. -fPIC -shared -o librestrict-stat.so
> restrict-stat.c
> In file included from restrict-stat.c:106:0:
> ./xstatconv.c: In function ‘__xstat_conv’:
> ./xstatconv.c:90:5: error: ‘struct stat’ has no member named
> ‘__unused4’
> buf->__unused4 = 0;
> ^
> ./xstatconv.c:93:5: error: ‘struct stat’ has no member named
> ‘__unused5’
> buf->__unused5 = 0;
> ^
> ./xstatconv.c: In function ‘__xstat32_conv’:
> ./xstatconv.c:266:5: error: ‘struct stat’ has no member named
> ‘__unused4’
> buf->__unused4 = 0;
> ^
> ./xstatconv.c:269:5: error: ‘struct stat’ has no member named
> ‘__unused5’
> buf->__unused5 = 0;
> ^
> 
> 
> --
> Phil Holmes
> 
> 
> - Original Message - 
> From: "Jan Nieuwenhuizen" 
> To: "Phil Holmes" 
> Cc: "David Kastrup" ; "Federico Bruni"
> ; 
> Sent: Tuesday, October 28, 2014 12:58 PM
> Subject: Re: GUB and mpfr/mpc
> 
> 
> Phil Holmes writes:
> 
> If there's nothing in the log file, you'll have to investigate by
> hand.
> Go to the build directory and see
> 
>cd /home/gub/gub/target/tools/build/librestrict-1.9.a
>gcc -W -Wall -fno-stack-protector -I. -fPIC -shared -o
>librestrict-stat.so restrict-stat.c
> 
> (eg, doing gcc -vvv or cpp -E or somesuch) why it fails to compile.
> It's a simple c file wit

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

> Masamichi HOSODA  writes:
> 
>> How about this patch?
>>
>> --- librestrict/xstatconv.c.org  2014-10-19 09:52:52.951262300 +0900
>> +++ librestrict/xstatconv.c  2014-11-01 20:26:35.734854500 +0900
>> @@ -48,20 +48,16 @@
>>{
>>  struct stat *buf = ubuf;
>>  
>> +/* zero clear */
>> +memset(buf, 0, sizeof(*buf));
>>  /* Convert to current kernel version of `struct stat'.  */
>>  buf->st_dev = kbuf->st_dev;
> 
> On the surface of it, this looks very reasonable.  However, since
> everything is guarded with #ifdef and similar and the definitions are
> probably generated by autoconf, it would appear that different versions
> of stat.h are employed in the compilation and the resulting mismatch of
> definitions and structures (possibly between host and target
> definitions) could lead to crashes and memory corruption.
> 
> So it would seem like a good idea to first figure out what goes wrong
> here before fixing it in this manner.  It might be a nice fix (and the
> code certainly looks nicer for it) but I have no idea if it is correct.
> The "if it were that easy, it would have been written like that in the
> first place" mantra may be wrong surprisingly often, but it would still
> be nice to make sure.
> 
> -- 
> David Kastrup

librestrict does not have a configure script. 
Therefore, it does not use autoconf.
Furthermore, in my understanding, librestrict is not cross compiled. 
That is, the host and the target is always the same as host OS.

If this is right, I think that my patch does not have a problem. 

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

I've succeeded to build lilypond for the following targets by gcc-4.8.2.

  linux-x86
  linux-64
  freebsd-x86
  freebsd-64
  mingw

(I don't build lilypond for darwin environments because I don't have them.)

My building environment is Ubuntu 14.04 LTS 64 bit.
I use the following gub repository.

  https://github.com/trueroad/gub/tree/binutils-2.24

linux-x86 and linux-64, freebsd-64 lilypond which I built
look like working fine.

In mingw, lilypond crashes as follows. 
Does someone know this reason? 

```
C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>type test.ly
{ c d e f g a b }

C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>lilypond test.ly
GNU LilyPond 2.19.16
Processing `test.ly'
Parsing...
test.ly:1: warning: no \version statement found, please add

\version "2.19.16"

for future compatibility
Interpreting music...
Preprocessing graphical objects...terminate called after throwing an instance of
 'std::bad_alloc'
  what():  std::bad_alloc

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>
```

In freebsd-x86, gs fails as follows.
(I tested this on 64bit FreeBSD. Perhaps it may succeed on 32bit FreeBSD.)

```
$ cat test.ly
{ c d e f g a b }

$ ./lilypond test.ly
GNU LilyPond 2.19.16
Processing `test.ly'
Parsing...
test.ly:1: warning: no \version statement found, please add

\version "2.19.16"

for future compatibility
Interpreting music...
Preprocessing graphical objects...
Finding the ideal number of pages...
Fitting music on 1 page...
Drawing systems...
Layout output to `test.ps'...
Converting to `./test.pdf'...
warning: `(gs -q -dSAFER -dDEVICEWIDTHPOINTS=595.28 -dDEVICEHEIGHTPOINTS=841.89 
-dCompatibilityLevel=1.4 -dNOPAUSE -dBATCH -r1200 -sDEVICE=pdfwrite 
-sOutputFile=./test.pdf -c.setpdfwrite -ftest.ps)' failed (139)

fatal error: failed files: "test.ly"
$ 
```

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

> Masamichi HOSODA  writes:
> 
>> In mingw, lilypond crashes as follows. 
>> Does someone know this reason? 
>>
>> ```
>> C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>type test.ly
>> { c d e f g a b }
>>
>> C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>lilypond test.ly
>> GNU LilyPond 2.19.16
>> Processing `test.ly'
>> Parsing...
>> test.ly:1: warning: no \version statement found, please add
>>
>> \version "2.19.16"
>>
>> for future compatibility
>> Interpreting music...
>> Preprocessing graphical objects...terminate called after throwing an 
>> instance of
>>  'std::bad_alloc'
>>   what():  std::bad_alloc
> 
> No idea.  Looks like out of memory.

I also think so.
I look at the task manager.
The memory that lilypond.exe uses is increased rapidly.
Then, lilypond.exe crashes at about 2 GB.

My Windows environment is 64-bit. The memory is quite larger than 2 GB.
I think that something continues demanding a memory by an infinite loop.
Then, lilyond.exe crashes, when the 32-bit process memory limit is exceeded.

>> In freebsd-x86, gs fails as follows.
>> (I tested this on 64bit FreeBSD. Perhaps it may succeed on 32bit FreeBSD.)
>>
>> ```
>> $ cat test.ly
>> { c d e f g a b }
>>
>> $ ./lilypond test.ly
>> GNU LilyPond 2.19.16
>> Processing `test.ly'
>> Parsing...
>> test.ly:1: warning: no \version statement found, please add
>>
>> \version "2.19.16"
>>
>> for future compatibility
>> Interpreting music...
>> Preprocessing graphical objects...
>> Finding the ideal number of pages...
>> Fitting music on 1 page...
>> Drawing systems...
>> Layout output to `test.ps'...
>> Converting to `./test.pdf'...
>> warning: `(gs -q -dSAFER -dDEVICEWIDTHPOINTS=595.28 
>> -dDEVICEHEIGHTPOINTS=841.89 -dCompatibilityLevel=1.4 -dNOPAUSE -dBATCH 
>> -r1200 -sDEVICE=pdfwrite -sOutputFile=./test.pdf -c.setpdfwrite -ftest.ps)' 
>> failed (139)
>>
>> fatal error: failed files: "test.ly"
> 
> That's a SIGSEGV.  Not something that a GhostScript executable should be
> throwing.  Could it be that the 64-bit cross compilation causes
> problems?

I tried on 32-bit FreeBSD.
The result was just similar.
I tried to execute gs directly.
Then Segmentation fault (core dumped) happened.

linux-x86 binary that I built seems no problem.
I think that it's no problem even if 32-bit binary
is cross compiled by the 64-bit environment.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

> Masamichi HOSODA  writes:
> 
>>> Masamichi HOSODA  writes:
>>> 
>>>> In mingw, lilypond crashes as follows. 
>>>> Does someone know this reason? 
>>>>
>>>> ```
>>>> C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>type test.ly
>>>> { c d e f g a b }
>>>>
>>>> C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>lilypond test.ly
>>>> GNU LilyPond 2.19.16
>>>> Processing `test.ly'
>>>> Parsing...
>>>> test.ly:1: warning: no \version statement found, please add
>>>>
>>>> \version "2.19.16"
>>>>
>>>> for future compatibility
>>>> Interpreting music...
>>>> Preprocessing graphical objects...terminate called after throwing an 
>>>> instance of
>>>>  'std::bad_alloc'
>>>>   what():  std::bad_alloc
>>> 
>>> No idea.  Looks like out of memory.
>>
>> I also think so.
>> I look at the task manager.
>> The memory that lilypond.exe uses is increased rapidly.
>> Then, lilypond.exe crashes at about 2 GB.
>>
>> My Windows environment is 64-bit. The memory is quite larger than 2 GB.
>> I think that something continues demanding a memory by an infinite loop.
>> Then, lilyond.exe crashes, when the 32-bit process memory limit is exceeded.
> 
> My guess would have been that garbage collection is broken on the
> platform, but even without collecting garbage, I should think that 2GB
> should be sufficient for getting a small file like that compiled.

I turned off compilation optimization of mingw::lilypond as follows.
https://github.com/trueroad/gub/commit/1a6bc45fc00fa326270f2be40a5f588083e9db75

As a result, std::bad_alloc didn't occur any more.
But it was the following error.

```
C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>lilypond test.ly
GNU LilyPond 2.19.16
Processing `test.ly'
Parsing...
test.ly:1: warning: no \version statement found, please add

\version "2.19.16"

for future compatibility
Interpreting music...
Preprocessing graphical objects...
Finding the ideal number of pages...
Fitting music on 1 page...
Drawing systems...
Layout output to `test.ps'...
Converting to `./test.pdf'...
warning: `(gs -q -dNOSAFER -dDEVICEWIDTHPOINTS=-0.00 -dDEVICEHEIGHTPOINTS=-0.00
-dCompatibilityLevel=1.4 -dNOPAUSE -dBATCH -r1200 -sDEVICE=pdfwrite -sOutputFile
=./test.pdf -c.setpdfwrite -ftest.ps)' failed (1)

programming error: Parsed object should be dead: #
continuing, cross fingers
programming error: Parsed object should be dead: #) (staff-refpoint-extent -3.776 .
 -3.776) (last-in-score . #t) (page-turn-penalty) (page-break-penalty) (page-tur
n-permission . allow) (page-break-permission . allow) (vertical-skylines . #) (stencil . #))() >

continuing, cross fingers
fatal error: failed files: "test.ly"

C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>
```

test.ps lilypond generated was broken.
test.pdf was also broken of course.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

>> Masamichi HOSODA  writes:
>> 
>>>> Masamichi HOSODA  writes:
>>>> 
>>>>> In mingw, lilypond crashes as follows. 
>>>>> Does someone know this reason? 
>>>>>
>>>>> ```
>>>>> C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>type test.ly
>>>>> { c d e f g a b }
>>>>>
>>>>> C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>lilypond test.ly
>>>>> GNU LilyPond 2.19.16
>>>>> Processing `test.ly'
>>>>> Parsing...
>>>>> test.ly:1: warning: no \version statement found, please add
>>>>>
>>>>> \version "2.19.16"
>>>>>
>>>>> for future compatibility
>>>>> Interpreting music...
>>>>> Preprocessing graphical objects...terminate called after throwing an 
>>>>> instance of
>>>>>  'std::bad_alloc'
>>>>>   what():  std::bad_alloc
>>>> 
>>>> No idea.  Looks like out of memory.
>>>
>>> I also think so.
>>> I look at the task manager.
>>> The memory that lilypond.exe uses is increased rapidly.
>>> Then, lilypond.exe crashes at about 2 GB.
>>>
>>> My Windows environment is 64-bit. The memory is quite larger than 2 GB.
>>> I think that something continues demanding a memory by an infinite loop.
>>> Then, lilyond.exe crashes, when the 32-bit process memory limit is exceeded.
>> 
>> My guess would have been that garbage collection is broken on the
>> platform, but even without collecting garbage, I should think that 2GB
>> should be sufficient for getting a small file like that compiled.
> 
> I turned off compilation optimization of mingw::lilypond as follows.
> https://github.com/trueroad/gub/commit/1a6bc45fc00fa326270f2be40a5f588083e9db75
> 
> As a result, std::bad_alloc didn't occur any more.
> But it was the following error.
> 
> ```
> C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>lilypond test.ly
> GNU LilyPond 2.19.16
> Processing `test.ly'
> Parsing...
> test.ly:1: warning: no \version statement found, please add
> 
> \version "2.19.16"
> 
> for future compatibility
> Interpreting music...
> Preprocessing graphical objects...
> Finding the ideal number of pages...
> Fitting music on 1 page...
> Drawing systems...
> Layout output to `test.ps'...
> Converting to `./test.pdf'...
> warning: `(gs -q -dNOSAFER -dDEVICEWIDTHPOINTS=-0.00 
> -dDEVICEHEIGHTPOINTS=-0.00
> -dCompatibilityLevel=1.4 -dNOPAUSE -dBATCH -r1200 -sDEVICE=pdfwrite 
> -sOutputFile
> =./test.pdf -c.setpdfwrite -ftest.ps)' failed (1)
> 
> programming error: Parsed object should be dead: #
> continuing, cross fingers
> programming error: Parsed object should be dead: # Prob(
> (Y-offset . 1.0) (system-grob . #) (staff-refpoint-extent 
> -3.776 .
>  -3.776) (last-in-score . #t) (page-turn-penalty) (page-break-penalty) 
> (page-tur
> n-permission . allow) (page-break-permission . allow) (vertical-skylines . 
> # line_pair>) (stencil . #))() >
> 
> continuing, cross fingers
> fatal error: failed files: "test.ly"
> 
> C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>
> ```
> 
> test.ps lilypond generated was broken.
> test.pdf was also broken of course.

I changed mingw.org's mingw-runtime to mingw-64 (32-bit).
Then an error didn't occur and correct test.pdf was generated.

https://github.com/trueroad/gub/tree/binutils-2.24

I may pull request if you request it.

Further, even if the runtime is mingw-w64,
bad_alloc occurred when optimization was turned on.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

> I changed mingw.org's mingw-runtime to mingw-64 (32-bit).
> Then an error didn't occur and correct test.pdf was generated.
> 
> https://github.com/trueroad/gub/tree/binutils-2.24
> 
> I may pull request if you request it.
> 
> Further, even if the runtime is mingw-w64,
> bad_alloc occurred when optimization was turned on.

For both of bad_alloc prevented and optimization,
I tried the following setting.
Only one file (lily/skyline.cc) optimization is disabled
and all other files optimization is enabled.

https://github.com/trueroad/gub/tree/mingw-optimize-test

Then bad_alloc didn't occur and correct PDF was generated.
However, I used gcc-4.9.3 because gcc-4.8.2 was Segmentation Fault.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

>>> I changed mingw.org's mingw-runtime to mingw-64 (32-bit).
>>> Then an error didn't occur and correct test.pdf was generated.
>>> 
>>> https://github.com/trueroad/gub/tree/binutils-2.24
>>> 
>>> I may pull request if you request it.
>>> 
>>> Further, even if the runtime is mingw-w64,
>>> bad_alloc occurred when optimization was turned on.
>>
>> For both of bad_alloc prevented and optimization,
>> I tried the following setting.
>> Only one file (lily/skyline.cc) optimization is disabled
>> and all other files optimization is enabled.
> 
> Do you have a backtrace that might give us some more clue about where
> lily/skyline.cc has a problem?
> 
> I thought that I had at one time proposed something to be changed (as
> part of some issue?) order to deal with possible memory corruption, but
> a quick look through the log messages does not turn up either a commit
> from me or a commit message ringing a bell.

I turned off optimization to get correct backtrace,
but bad_alloc didn't occur.
Therefore I could get only incomplete backtrace.

It seems that push_back in Skyline::internal_merge_skyline throws bad_alloc.
I think the problem is Skyline::internal_merge_skyline
and/or first_intersection.

Skyline::internal_merge_skyline has a while loop with push_back.
I think that the loop termination condition may break by optimization.

So, I tried to disable optimization only not one file whole, but one function.

In the case of Skyline::internal_merge_skyline, the result was bad_alloc.
In the case of first_intersection,
the result was that correct PDF was generated.

The source tree when succeeding, is as follows.
https://github.com/trueroad/gub/commit/5e63dbde436bd3fca154557672394987c8d2e385

Masamichi Hosoda

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

> I changed mingw.org's mingw-runtime to mingw-64 (32-bit).
> Then an error didn't occur and correct test.pdf was generated.
> 
> https://github.com/trueroad/gub/tree/binutils-2.24
> 
> I may pull request if you request it.
> 
> Further, even if the runtime is mingw-w64,
> bad_alloc occurred when optimization was turned on.

 For both of bad_alloc prevented and optimization,
 I tried the following setting.
 Only one file (lily/skyline.cc) optimization is disabled
 and all other files optimization is enabled.
>>> 
>>> Do you have a backtrace that might give us some more clue about where
>>> lily/skyline.cc has a problem?
>>> 
>>> I thought that I had at one time proposed something to be changed (as
>>> part of some issue?) order to deal with possible memory corruption, but
>>> a quick look through the log messages does not turn up either a commit
>>> from me or a commit message ringing a bell.
>>
>> I turned off optimization to get correct backtrace,
>> but bad_alloc didn't occur.
>> Therefore I could get only incomplete backtrace.
>>
>> It seems that push_back in Skyline::internal_merge_skyline throws bad_alloc.
>> I think the problem is Skyline::internal_merge_skyline
>> and/or first_intersection.
>>
>> Skyline::internal_merge_skyline has a while loop with push_back.
>> I think that the loop termination condition may break by optimization.
> 
> I need more elements from the backtrace.  The problem most likely is
> that the Skyline is destructed before work with it is finished, and that
> means that somewhere in the call chain the Skyline is not maintained in
> a manner where the Lisp Garbage collector will consider it as still in
> use.
> 
> So internal_merge_skyline is likely only the place where things blow up,
> but the actual cause will be further up in the call chain.

Here is first backtrace.
I stopped lilypond.exe by Ctrl-C
when the memory that lilypond.exe uses was increased rapidly.
I think that it was in the infinite loop.

```
C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>gdb lilypond.exe
GNU gdb (GDB) 7.8.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from lilypond.exe...done.
(gdb) r test.ly
Starting program: C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin\lilypond.exe 
test.ly
[New Thread 7936.0x2234]
GNU LilyPond 2.19.16
Processing `test.ly'
Parsing...
test.ly:1: warning: no \version statement found, please add

\version "2.19.16"

for future compatibility
Interpreting music...
Preprocessing graphical objects...[New Thread 7936.0x256c]

Program received signal SIGINT, Interrupt.
[Switching to Thread 7936.0x256c]
0x76cedbf6 in KERNELBASE!CtrlRoutine ()
   from C:\WINDOWS\SYSTEM32\KernelBase.dll
(gdb) bt
#0  0x76cedbf6 in KERNELBASE!CtrlRoutine ()
   from C:\WINDOWS\SYSTEM32\KernelBase.dll
#1  0x97b0d836 in ?? ()
#2  0x in ?? ()
(gdb) info thread
  Id   Target Id Frame
* 2Thread 7936.0x256c 0x76cedbf6 in KERNELBASE!CtrlRoutine ()
   from C:\WINDOWS\SYSTEM32\KernelBase.dll
  1Thread 7936.0x2234 0x77138fc0 in ntdll!RtlCompareMemoryUlong ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
(gdb) thread 1
[Switching to thread 1 (Thread 7936.0x2234)]
#0  0x77138fc0 in ntdll!RtlCompareMemoryUlong ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
(gdb) bt
#0  0x77138fc0 in ntdll!RtlCompareMemoryUlong ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#1  0x7718f95a in ntdll!RtlFindCharInUnicodeString ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#2  0x7714dfb4 in ntdll!RtlAllocateHeap () from C:\WINDOWS\SYSTEM32\ntdll.dll
#3  0x771eee08 in ntdll!RtlpNtSetValueKey ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#4  0x7718f75a in ntdll!RtlFindCharInUnicodeString ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#5  0x7714dfb4 in ntdll!RtlAllocateHeap () from C:\WINDOWS\SYSTEM32\ntdll.dll
#6  0x76889ad1 in msvcrt!malloc () from C:\WINDOWS\SYSTEM32\msvcrt.dll
#7  0x6fcbddef in operator new (sz=40)
at 
/home/gub/gub/target/mingw/src/cross/gcc-4.8.2/libstdc++-v3/libsupc++/new_op.cc:51
#8  0x0060da2d in Skyline::internal_merge_skyline(std::list >*, std::list >*, 
std::list >*) const ()
#9  0x00611495 in Skyline::padded(double) const ()
#10 0x005fb306 in Separation_item::calc_skylines(scm_unused_struct*) ()
#11 0x6bf921c0 in scm_dapply ()
   from C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin\libguile-17.dll
#12 0x000

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

>>> I changed mingw.org's mingw-runtime to mingw-64 (32-bit).
>>> Then an error didn't occur and correct test.pdf was generated.
>>> 
>>> https://github.com/trueroad/gub/tree/binutils-2.24
>>> 
>>> I may pull request if you request it.
>>> 
>>> Further, even if the runtime is mingw-w64,
>>> bad_alloc occurred when optimization was turned on.
> 
> There is this at the end of skyline.cc:
> 
>   // Should add during ver2.19 (to avoid an endless loop
>   // when  merging identical skylines with a vertical segment)
>   // if (end >= s2->front().end_) s2->pop_front();
> 
> Also, the mention of optimization reminds me of one of the horrors of 
> floating point types: a value in a register can quietly be changed when it is 
> written to memory.  Take a look at some of the “myths” in section 1 of 
> https://hal.archives-ouvertes.fr/hal-00128124/en/ .
> 
> Do the skylines need to deal with the wide range of values that “double" 
> allows?  If not, replacing the Real class with a fixed-point math class would 
> make the code a lot less scaring.

I think that floating point can be used as long as it's used correct.

operator== shouldn't be used for comparison of floating point values.
operator!= shouldn't be used too.
operator< and operator> may be used.
It's better not to use operator<= and operator>=.

I tried the following patch.
But bad_alloc occurred.

Then, I think the floating point problem may be unrelated to this case.

```
--- a/lily/skyline.cc   2014-12-01 21:31:21.353083100 +0900
+++ b/lily/skyline.cc   2014-12-04 22:56:17.740379900 +0900
@@ -184,7 +184,7 @@

   // conceals and intersection_x involve multiplication and
   // division. Avoid that, if we can.
-  if (c.y_intercept_ == -infinity_f)
+  if ((isinf(c.y_intercept_) && signbit(c.y_intercept_)))
 {
   if (c.end_ > b.end_)
 return b.end_;
@@ -197,7 +197,7 @@
 return start_x;

   Real i = b.intersection_x (c);
-  if (i > start_x && i <= b.end_ && i <= c.end_)
+  if (i > start_x && (i - b.end_) < DBL_EPSILON && (i - c.end_) < 
DBL_EPSILON)
 return i;

   start_x = c.end_;
```

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

>> There is this at the end of skyline.cc:
>> 
>>  // Should add during ver2.19 (to avoid an endless loop
>>  // when  merging identical skylines with a vertical segment)
>>  // if (end >= s2->front().end_) s2->pop_front();
> 
> I meant at the end of internal_merge_skyline() in skyline.cc. (Fatigue 
> strikes again.)
> 
>> make the code a lot less scaring.
> 
> scary (derp)

I tried the following patch, too.
But bad_alloc occurred.

```
--- a/lily/skyline.cc   2014-12-01 21:31:21.353083100 +0900
+++ b/lily/skyline.cc   2014-12-04 23:46:29.488379900 +0900
@@ -329,7 +329,7 @@
 s1->pop_front ();
   // Should add during ver2.19 (to avoid an endless loop
   // when  merging identical skylines with a vertical segment)
-  // if (end >= s2->front().end_) s2->pop_front();
+  if (end >= s2->front().end_) s2->pop_front();

   x = end;
 }
```

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

>>> Also, the mention of optimization reminds me of one of the horrors of 
>>> floating point types: a value in a register can quietly be changed when it 
>>> is written to memory.  Take a look at some of the “myths” in section 1 of 
>>> https://hal.archives-ouvertes.fr/hal-00128124/en/ .
> […]
>> I think that floating point can be used as long as it's used correct.
>> 
>> operator== shouldn't be used for comparison of floating point values.
>> operator!= shouldn't be used too.
>> operator< and operator> may be used.
> 
> The issue is more complex than that.  Sit down and try this:
[...]
> Add looping and you’ve got fatal issues.  (And if you’re like me, steam will 
> erupt from your ears any second now...)

You're right of course.
A floating point operation has a margin of error always.
When the optimization is different, the margin of error may be also different,
because the bit width of floating point value is different
in a register and a memory.

Therefore, robust treating for margin of error is necessary.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB again

[Masamichi HOSODA]

>> I've seen a proposed patch from Masamichi, but as David says, this may
>> fix the issue but doesn't shed any light on what is the root cause.
>> Is it worth trying to go back to an earlier version of gcc?  If so,
>> how would I go about that?
> 
> I lean towards just using Masamichi's patch.

I'll pull request following branch if it's necessary.

https://github.com/trueroad/gub/tree/fix-librestrict

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi HOSODA]

> I agree that changing the algorithms is preferred; I didn’t mean to suggest 
> otherwise.  But if that’s not going to happen overnight, and there is a way 
> to mitigate the problem in the meantime without touching the code, the people 
> affected would value it.

I tried "-mfpmath=sse -msse2".
It worked fine. bad_alloc didn't occur.
Correct PDF was generated.

It can be a workaround until changing the algorithms.
___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB and mpfr/mpc

[Masamichi Hosoda]

> >>> Converting to `./test.pdf'...
> >>> warning: `(gs -q -dSAFER -dDEVICEWIDTHPOINTS=595.28
> >>> -dDEVICEHEIGHTPOINTS=841.89 -dCompatibilityLevel=1.4 -dNOPAUSE
> >>> -dBATCH -r1200 -sDEVICE=pdfwrite -sOutputFile=./test.pdf
> >>> -c.setpdfwrite -ftest.ps)' failed (139)
> >>>
> >>> fatal error: failed files: "test.ly"
> >> 
> >> That's a SIGSEGV.  Not something that a GhostScript executable should be
> >> throwing.  Could it be that the 64-bit cross compilation causes
> >> problems?
> >
> > I tried on 32-bit FreeBSD.
> > The result was just similar.
> > I tried to execute gs directly.
> > Then Segmentation fault (core dumped) happened.
> >
> > linux-x86 binary that I built seems no problem.
> > I think that it's no problem even if 32-bit binary
> > is cross compiled by the 64-bit environment.
> 
> No idea.  Phil, what is your experience with that?

I tried to update freebsd-x86::freebsd-runtime from 4.11 to 6.2.
(It becomes same version as freebsd-64::freebsd-runtime.)

https://github.com/trueroad/gub/tree/gcc-4.8

And, I executed 'lilypond test.ly' on 32-bit FreeBSD.
Then SIGSEGV didn't occur and correct pdf was generated.

I think FreeBSD 4.11 is too old.


___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

When environment variable PATH is undefined

[Masamichi HOSODA]

When environment variable PATH is undefined, the following error occurs.

```
C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>set PATH=

C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>lilypond
GNU LilyPond 2.19.16
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_S_construct null not valid

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

C:\tmp\lilypond-2.19.16-0.mingw\$_OUTDIR\usr\bin>
```

The following patch can fix this problem.

```
--- a/flower/file-path.cc   2014-03-18 00:29:16.0 +0900
+++ b/flower/file-path.cc   2014-12-29 17:18:06.887602900 +0900
@@ -52,6 +52,13 @@
   concat (dirs_, string_split (p, PATHSEP));
 }
 
+void
+File_path::parse_path (const char *p)
+{
+  if (p)
+parse_path (static_cast(p));
+}
+
 bool
 is_file (const string &file_name)
 {
--- a/flower/include/file-path.hh   2014-03-18 00:29:16.0 +0900
+++ b/flower/include/file-path.hh   2014-12-29 17:28:53.070834400 +0900
@@ -44,6 +44,7 @@
   bool try_append (string str);
   void append (const string&);
   void parse_path (const string&);
+  void parse_path (const char*);
   void prepend (const string&);
 };
 
```

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB fail with smob templates

[Masamichi HOSODA]

> The changes from Masamichi to librestrict look safe,
> and the later floating-point-endless-loop-eating-all-memory should be
> gone now that I've re-worked the skyline merge code.
> 
> If gcc 4.8.2 still looks difficult, I'll look into problem with the
> templates.

Now, in this branch,
https://github.com/trueroad/gub/tree/gcc-4.8

The following commands have been succeed by gcc-4.8.2.

bin/gub mingw::lilypond-installer
bin/gub linux-x86::lilypond-installer
bin/gub linux-64::lilypond-installer
bin/gub freebsd-x86::lilypond-installer
bin/gub freebsd-64::lilypond-insatller

They work fine in my environments.

mingw: bad_alloc don't occur.
freebsd-x86: SIGSEGV don't occur.

Correct PDFs are generated.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB fail with smob templates

[Masamichi HOSODA]

>>  How does the upgrade to gcc 4.8.2 look now ?
> 
> Just wondering: There is already a gcc 4.9.x series, so why are we
> trying to update to 4.8.x?

In this branch, I've tried gcc-4.9.
https://github.com/trueroad/gub/tree/gcc-4.9

I've succeed to build lilypond-installer
for mingw, linux-x86, linux-64, freebsd-x86, freebsd-64 platforms.
And they work fine.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: GUB update

[Masamichi HOSODA]

> The principle with GUB is that it has details of all the packages it uses,
> and by issuing the 'make bootstrap' command, it goes and gets all the
> packages it needs, and all their dependencies, and builds them all from
> scratch.  The problem I believe I now have is that gcc 4.8 has a new
> dependency: GMP, the multi-precision library. I believe this is a dependency
> from the newer version of MPFR that gcc 4.8 requires.
...
> I assume what is happening here is that the manual install places the mpc
> libraries where the gcc configur can't find them.  In any case, doing this
> manually defeats the object of a self-building package builder.  So what
> would be really helpful would be for someone who understands all the python
> stuff that GUB does could point me to how to add the new dependency to GUB
> for MPC.

How about this branch?
https://github.com/trueroad/gub/tree/gcc-4.8

I've uninstalled Ubuntu's gmp/mpfr/mpc by the following commands.

```
sudo apt-get --purge remove libmpc-dev
sudo apt-get --purge remove libmpfr-dev
sudo apt-get --purge remove libgmp-dev
```

And, I've changed gub to use tools::gmp, tools::mpfr, tools::mpc
for gcc building.
Then, I've succeed to build mingw::cross/gcc and linux-x86::cross/gcc.

I haven't build for other platforms yet.
However, I'd succeed to build for linux-64, freebsd-x86, freebsd-64 platforms.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Lilypond build: version

[Masamichi HOSODA]

> Upload is now proceeding.  Takes about 5 hours and slows internet
> access for the house to a crawl :-(

Thank you for uploading official binaries.
I've tested the binaries in my environments.
The results are as follows.

linux-x86: Ubuntu 14.04 LTS x86_64 (with 32bit libs)
  Fine

linux-64: Ubuntu 14.04 LTS x86_64
  Fine

linux-ppc: Debian wheezy PowerPC (on qemu-system-ppc on Windows 8.1)
  Fine

freebsd-x86: FreeBSD 10.1-RELEASE i386 (with compat libs)
  Fine

freebsd-64: FreeBSD 10.1-RELEASE amd64 (with compat libs)
  Fine

mingw (Windows): Windows 8.1 64bit
  Fine
  And, this issue
  http://code.google.com/p/lilypond/issues/detail?id=431
  has been fixed. Because GUB's binutils has been updated to 2.25.

darwin-x86:
  Untested
  My Intel Mac environment was quite a temporary.
  Now, I don't have it.

darwin-ppc:
  Untested
  I couldn't prepare PowerPC Mac.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Lilypond build: version

[Masamichi HOSODA]

> On Mar 2, 2015, at 09:03 , Masamichi HOSODA  wrote:
>> 
>> darwin-x86:
>>  Untested
>>  My Intel Mac environment was quite a temporary.
>>  Now, I don't have it.
> 
> It runs without crashing on 10.10.2.

Thank you for reporting.
I'm happy to hear it.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Unicode filename support for Windows

[Masamichi HOSODA]

On Windows, lilypond can't treat unicode filenames.
On linux, it can (UTF-8 unicode filenames).

So, I'm trying to add a unicode filename support for Windows lilypond.
I attach the patch.

It replaces main(), and hooks filename related functions.
This converts between UTF-16 unicode (Windows) and
UTF-8 unicode (lilypond, libguile etc.).

As a result, lilypond can treat unicode filenames of *.ly, *.mid, *.ps.

However, it can't treat *.pdf, yet.
Ghostscript-8.70 that are included in the binary distribution of lilypond
can't treat the unicode filenames.
Ghostscript-9.10 or later can treat the unicode filenames.
>From bb3f8bd6bcaf41de57e8981ac57a3ac0ef8c Mon Sep 17 00:00:00 2001
From: Masamichi Hosoda 
Date: Sat, 7 Mar 2015 18:42:57 +0900
Subject: [PATCH] Add unicode filename support for Windows

---
 flower/include/mingw-utf8-conv.hh |   7 ++
 flower/include/mingw-utf8-func.hh |  16 
 flower/include/mingw-utf8-hook.hh |   6 ++
 flower/include/mingw-utf8.hh  |  11 +++
 flower/mingw-utf8-conv.cc |  58 
 flower/mingw-utf8-func.cc | 157 +
 flower/mingw-utf8-hook.cc | 179 ++
 flower/mingw-utf8-main.cc |  84 ++
 lily/GNUmakefile  |   1 +
 lily/main.cc  |   4 +
 10 files changed, 523 insertions(+)
 create mode 100644 flower/include/mingw-utf8-conv.hh
 create mode 100644 flower/include/mingw-utf8-func.hh
 create mode 100644 flower/include/mingw-utf8-hook.hh
 create mode 100644 flower/include/mingw-utf8.hh
 create mode 100644 flower/mingw-utf8-conv.cc
 create mode 100644 flower/mingw-utf8-func.cc
 create mode 100644 flower/mingw-utf8-hook.cc
 create mode 100644 flower/mingw-utf8-main.cc

diff --git a/flower/include/mingw-utf8-conv.hh b/flower/include/mingw-utf8-conv.hh
new file mode 100644
index 000..1cabe21
--- /dev/null
+++ b/flower/include/mingw-utf8-conv.hh
@@ -0,0 +1,7 @@
+#ifndef __MINGW_UTF8_CONV_H__
+#define __MINGW_UTF8_CONV_H__
+
+std::vector mingw_utf8_16to8 (const wchar_t *wc);
+std::vector mingw_utf8_8to16 (const char *c);
+
+#endif // __MINGW_UTF8_CONV_H__
diff --git a/flower/include/mingw-utf8-func.hh b/flower/include/mingw-utf8-func.hh
new file mode 100644
index 000..d4a0a29
--- /dev/null
+++ b/flower/include/mingw-utf8-func.hh
@@ -0,0 +1,16 @@
+#ifndef __MINGW_UTF8_FUNC_H__
+#define __MINGW_UTF8_FUNC_H__
+
+FILE *utf8_fopen(const char *path, const char *mode);
+FILE *utf8_freopen (const char *path, const char *mode, FILE *stream);
+int utf8__stat (const char *path, struct _stat *buff);
+char *utf8_getcwd(char *buff, int size);
+char *utf8_getenv(const char *var);
+int utf8_open (const char *path, int flag, ...);
+int utf8_system (const char *command);
+int utf8__unlink (const char *path);
+gchar *utf8_g_locale_to_from_utf8 (const gchar *string, gssize len,
+   gsize *bytes_read, gssize *bytes_written,
+   GError **error);
+
+#endif // __MINGW_UTF8_FUNC_H__
diff --git a/flower/include/mingw-utf8-hook.hh b/flower/include/mingw-utf8-hook.hh
new file mode 100644
index 000..0eea976
--- /dev/null
+++ b/flower/include/mingw-utf8-hook.hh
@@ -0,0 +1,6 @@
+#ifndef __MINGW_UTF8_HOOK_H__
+#define __MINGW_UTF8_HOOK_H__
+
+void mingw_utf8_hook(void);
+
+#endif // __MINGW_UTF8_HOOK_H__
diff --git a/flower/include/mingw-utf8.hh b/flower/include/mingw-utf8.hh
new file mode 100644
index 000..feb2fab
--- /dev/null
+++ b/flower/include/mingw-utf8.hh
@@ -0,0 +1,11 @@
+#ifdef __MINGW32__
+
+#ifndef __MINGW_UTF8_H__
+#define __MINGW_UTF8_H__
+
+#define main(...) utf8_main(__VA_ARGS__)
+
+int utf8_main(int argc, char **argv, char **envp);
+
+#endif // __MINGW_UTF8_H__
+#endif // __MINGW32__
diff --git a/flower/mingw-utf8-conv.cc b/flower/mingw-utf8-conv.cc
new file mode 100644
index 000..fa04da9
--- /dev/null
+++ b/flower/mingw-utf8-conv.cc
@@ -0,0 +1,58 @@
+/*
+  This file is part of LilyPond, the GNU music typesetter.
+
+  Copyright (C) 2015 Masamichi Hosoda 
+
+  LilyPond is free software: you can redistribute it and/or modify
+  it under the terms of the GNU General Public License as published by
+  the Free Software Foundation, either version 3 of the License, or
+  (at your option) any later version.
+
+  LilyPond is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+  GNU General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with LilyPond.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#ifdef __MINGW32__
+
+#include 
+#include 
+#include 
+
+#include "mingw-utf8-conv.hh"
+
+// Convert from UTF-16 to UTF-8
+std::vector mingw_utf8_16to8 (const wchar_t *wc)
+{
+  int size = WideCharToMultiByte (CP_UTF8, 0, wc, -1, NULL, 0, N

Re: Add support for unicode filenames on Windows (issue 206640043 by pkx1...@gmail.com)

[Masamichi HOSODA]

Thank you for the mail.
I've written reply to the tracker web page.

https://code.google.com/p/lilypond/issues/detail?id=4317

> Reviewers: ,
> 
> Message:
> Pasting update in Tracker by David K for Masamichi's benefit:
> 
> --snip--
> 
> by d...@gnu.org: Patch: Add support for unicode filenames on Windows
> https://code.google.com/p/lilypond/issues/detail?id=4317
> 
> Ok, I've been stalling on this patch for about a week because I don't
> have a good idea how to address it properly.  So let's start with the
> worst first: at the current point of time, I don't see it making
> sense,
> and in addition I will not be able to spend significant amount of time
> addressing the problems arising in connection with it for about a
> month,
> so I'd like a moratorium on it for about that time.
> 
> The main problem is that the next step needed for GUILEv2 migration
> (which nobody is really interested in working on apart from myself) is
> to get the coding mess sorted out.  GUILEv1 is basically working with
> byte streams in its strings and string ports and files.  GUILEv2 is
> coding system aware, uses either Latin-1 or UTF-32 as its string
> internals, uses UTF-8 for string ports (necessitating copies and
> conversions rather than being able to work in-place) and converts back
> and forth all the time.
> 
> Since LilyPond has a lexer working on an UTF-8 coded byte stream and
> data is liberally bounced around between files, strings, and string
> ports, and the respective position pointers are not distinguished.
> 
> It does not help that GUILEv2 changes its ways to pick particular
> encodings basically every 3 or 4 "stable" versions in the 2.0 series,
> and 2.1 is different again.  So we need to carefully refactor the code
> in order to have a chance of it working in both GUILEv1 and GUILEv2
> (and
> at all in GUILEv2).
> 
> This is an ugly and ungrateful task that has been at the front of my
> this-is-really-up-next-so-let's-procrastinate-instead queue the last
> month or so.  Letting a patch like this in right now would make the
> situation worse.
> 
> Viewed realistically, this patch also needs Ghostscript updates in GUB
> to make any sense.  There is no reason _those_ cannot go ahead first:
> having a buffer of a few developer releases where we might discover
> and
> sort out possible problems with newer versions of GhostScript on our
> crosscompiled distributions seems sensible.
> 
> Personally, I don't really like the look of this patch at all as it is
> very specifically useful only for one platform, and apart from that
> I'd
> very much want to avoid seeing any wide character functions in
> LilyPond
> code: that's rarely exercised library code of often dubious
> reliability
> or availability, and most programmers are not overly comfortable
> utilizing it.  I'd very much prefer that we'd find a better
> encapsulated
> solution, probably by utilizing the encoding support of GUILEv2.
> Which
> again suggests that a moratorium on this patch makes sense, but that
> one
> would not be "until David gets encodings sorted out so that we can
> compile and test on GUILEv2" but rather on "until we are able to ditch
> GUILEv1 altogether".  And given the current enthusiasm for working on
> the GUILEv2 migration and the likelihood of necessary bug fixes in
> GUILE
> itself becoming available as we discover problems, the latter is
> likely
> quite longer than a month away.
> 
> The problem this patch tries to address is not a recent problem, and
> the
> workaround ("don't use special Unicode characters in filenames") is
> obvious.  So the urgency seems limited.  I very much agree that
> LilyPond
> should not balk at file and directory names containing accented
> characters on all platforms.  But I think we will be doing ourselves
> our
> favor to postpone addressing this problem until we have moved LilyPond
> to GUILEv2, and then look for the best-matching solution within _this_
> framework.  Otherwise we complicate the coding issue work for GUILEv2
> migration and that's something we can afford less than prolonging the
> current filename encoding situation.
> 
> -- 
> You received this message because you are the owner of the issue.
> You may adjust your notification preferences at:
> https://code.google.com/hosting/settings
> 
> Reply to this email to add a comment or make updates.
> 
> Description:
> Add support for unicode filenames on Windows
> 
> On Windows, LilyPond can't handle
> unicode filenames.
> 
> The patch replaces main(), and
> hooks filename related functions.
> This converts between UTF-16
> unicode (Windows) and UTF-8 unicode
> (LilyPond, libguile etc.).
> 
> LilyPond can handle unicode filenames
> for *.ly, *.mid, *.ps.
> 
> *.pdf is not supported yet.
> Ghostscript-8.70 that is included
> in the binary distribution of LilyPond
> can't handle unicode filenames.
> This requires Ghostscript-9.10 or later
> 
> Please review this at https://codereview.appspot.com/206640043/
> 
> Affected files (+523, -0 lin

Ghostscript 9.15

[Masamichi HOSODA]

I've succeed to upgrade GUB's ghostscript to 9.15 in this branch.
https://github.com/trueroad/gub/tree/ghostscript-9.15

I've succeed GUB's ``make lilypond'' by ghostscript-9.15.
All lilypond installers have been build.

In mingw (Windows):
Ghostscript can handle unicode filenames.
I've tested lilypond installer and lilypond. They work fine.
Correct PDF is generated.

In linux-64, linux-x86, linux-ppc, freebsd-64, freebsd-x86, darwin-x86:
I've tested lilypond installer and lilypond. They work fine.
Correct PDF is generated.

In darwin-ppc (PowerPC Mac):
It is untested.
I can't prepare PowerPC Mac.

Would you pull this branch for next lilypond developer build?

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Ghostscript 9.15

[Masamichi HOSODA]

> Have you checked ligatures?  I think we were able to correlate our
> ligature problems (don't know the issue right now) to the use of 64bit
> architecture.  It may be related to GhostScript.

I haven't.
Is it here?
http://code.google.com/p/lilypond/issues/detail?id=2656

I'll check it.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Ghostscript 9.15

[Masamichi HOSODA]

>>> Have you checked ligatures?  I think we were able to correlate our
>>> ligature problems (don't know the issue right now) to the use of 64bit
>>> architecture.  It may be related to GhostScript.
>>
>> I haven't.
>> Is it here?
>> http://code.google.com/p/lilypond/issues/detail?id=2656
>>
>> I'll check it.
> 
> Yes, that's the one.  We still haven't a clue what is really causing it,
> so updating GhostScript is a good opportunity to check whether this
> makes a difference.  Of course, our update of the crosscompilers could
> conceivably also have made a difference here.

I've checked some environments.

  linux-64: Ubuntu 14.04 LTS 64 bit
ligatured pdf is generated.

  linux-x86: Ubuntu 14.04 LTS 32 bit (minimal install, no GUI, console only)
non-ligatured pdf is generated.

  mingw (Windows): Windows 8.1 64 bit
non-ligatured pdf is generated.

I think that ghostscript is transparent about ligature.
It convert from ligatured PostScript files to ligatured PDFs,
from non-ligatured PostScript files to non-ligatured PDFs.

So, I've tried to the following.

  In linux-64: .ly to .ps
$ ./lilypond -f ps -o linux-64 2656-font-ligatures.ly

  In linux-x86: .ly to .ps
$ ./lilypond -f ps -o linux-x86 2656-font-ligatures.ly

  Then, I copied the generated PostScript files to cygwin environment.

  In cygwin: .ps to .pdf
$ ps2pdf14 linux-64.ps linux-64.pdf
$ ps2pdf14 linux-x86.ps linux-x86.pdf

  The result:
linux-64.pdf is ligatured.
linux-x86.pdf is non-ligatured.

This result shows that ghostscript isn't the cause of the ligature problem.
I think that ligature problem caused by system's installed fonts
or libraries etc.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Ghostscript 9.15

[Masamichi HOSODA]

>>   linux-64: Ubuntu 14.04 LTS 64 bit
>> ligatured pdf is generated.
>> 
>>   linux-x86: Ubuntu 14.04 LTS 32 bit (minimal install, no GUI, console only)
>> non-ligatured pdf is generated.
> 
> Ah, this is surprising, since up to now exactly the opposite behaviour
> was reported (this is, working on 32bit and failing on 64bit).

I've tried lilypond linux-x86 binary on Ubuntu 64 bit environment.
The result is non-ligatured.
In this same environment, linux-64 binary is ligatured.
Therefore, system's installed fonts etc. are no relation of this problem.

I think that the reason of opposite result is different compiler version.
I use GUB's gcc-4.9.2.
Lilypond 2.19.16 and 2.19.17 official binary is compiled by GUB's gcc-4.9.2.
Therefore, it might be the same result as mine
when you use the latest official binary.

> I still suspect a Pango bug that has probably been fixed in the not
> too distant past.  What version are you using?  Can you upgrade to the
> most recent one?  On my 32bit GNU/Linux I use a quite recent git
> version of Pango, which should behave identically to the last release
> (1.36.8), and everything's just fine.

GUB's pango is 1.26.0.
https://github.com/gperciva/gub/blob/master/gub/specs/pango.py#L18
I think that it is too old.

Pango 1.26.2 and 1.28.0 had some ligature related bugfixes.
https://git.gnome.org/browse/pango/tree/NEWS#n306
https://git.gnome.org/browse/pango/tree/NEWS#n288

I'll try to upgrade GUB's pango to 1.28.3 or 1.26.2.
Pango 1.28.4 requires glib 2.24.0.
However, GUB's glib is 2.21.5.
Therefore, in order to upgrade to newer pango,
it is necessary to also upgrade glib.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Ghostscript 9.15

[Masamichi HOSODA]

>> I still suspect a Pango bug that has probably been fixed in the not
>> too distant past.  What version are you using?  Can you upgrade to the
>> most recent one?  On my 32bit GNU/Linux I use a quite recent git
>> version of Pango, which should behave identically to the last release
>> (1.36.8), and everything's just fine.
> 
> GUB's pango is 1.26.0.
> https://github.com/gperciva/gub/blob/master/gub/specs/pango.py#L18
> I think that it is too old.
> 
> Pango 1.26.2 and 1.28.0 had some ligature related bugfixes.
> https://git.gnome.org/browse/pango/tree/NEWS#n306
> https://git.gnome.org/browse/pango/tree/NEWS#n288
> 
> I'll try to upgrade GUB's pango to 1.28.3 or 1.26.2.
> Pango 1.28.4 requires glib 2.24.0.
> However, GUB's glib is 2.21.5.
> Therefore, in order to upgrade to newer pango,
> it is necessary to also upgrade glib.

I've succeed to upgrade GUB's pango to 1.28.3.
https://github.com/trueroad/gub/tree/pango-1.28

Then, I've checked some environments.
The results are following:

  linux-64 binary on Ubuntu 14.04 LTS 64 bit:
ligatured pdf is generated.

  linux-x86 binary on Ubuntu 14.04 LTS 64 bit:
ligatured pdf is generated.

  linux-x86 binary on Ubuntu 14.04 LTS 32 bit (minimal install):
Serif font (CenturySchL-Roma) is ligatured.
Sans-serif font (TakaoPGothic) is non-ligatured.

I think that the reason is that
the system doesn't have ligature possible sans-serif fonts.
In other environment, selected sans-serif font is DejaVuSans.

  mingw (Windows) on Windows 8.1 64 bit:
ligatured pdf is generated.

Note that I haven't build all platform's lilypond installer, yet.

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Ghostscript 9.15

[Masamichi HOSODA]

> So that’s probably a matter of the font, not of its style - not every font 
> defines ligatures, and the name „TakaoPGothic“ tells me its main focus would 
> be Japanese (is this true?), so the designers probably didn’t put so much 
> work in features of Latin script.
> 
> Would you care to try a different font?

TakaoPGothic is Japanese sans-serif font.
It is only one sans-serif font in the system.
The system doesn't have any other sans-serif fonts.

I'll try to install DejaVuSans and check again.
___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: Ghostscript 9.15

[Masamichi HOSODA]

>> So that’s probably a matter of the font, not of its style - not every font 
>> defines ligatures, and the name „TakaoPGothic“ tells me its main focus would 
>> be Japanese (is this true?), so the designers probably didn’t put so much 
>> work in features of Latin script.
>> 
>> Would you care to try a different font?
> 
> TakaoPGothic is Japanese sans-serif font.
> It is only one sans-serif font in the system.
> The system doesn't have any other sans-serif fonts.
> 
> I'll try to install DejaVuSans and check again.

I've checked.

The result is ligatured.
Lilypond select DejaVuSans as sans-serif font.
___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel