[
https://issues.apache.org/jira/browse/FOP-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Harrop updated FOP-3341:
------------------------------
Description:
GlyphTable.matchLookups falls back to the default script when a lookup match
comes up empty:
{noformat}
if (lm.isEmpty() && !OTFScript.isDefault(script) &&
!OTFScript.isWildCard(script)) {
return matchLookups(OTFScript.DEFAULT, OTFLanguage.DEFAULT, feature);
}
{noformat}
The fallback targets DFLT. So for a font that has no DFLT entry in its script
list, a script of DFLT finds nothing, and the fallback looks for DFLT again and
finds nothing again. Nothing is applied and nothing is reported. The font is
loaded, its tables are read, performsSubstitution and performsPositioning both
return true, and every lookup silently misses.
Fonts without a DFLT script table are not unusual. Carlito, the
metric-compatible Calibri substitute packaged by most Linux distributions,
declares cyrl, grek and latn and no DFLT. DejaVu declares DFLT alongside them,
which is why the two behave differently on the same code path and why this is
easy to mistake for a font-file or a loader problem.
Measured, on the same font objects with only the script argument changing:
{noformat}
script Carlito 1.104 DejaVuLGCSerif
latn 'fi' -> U+FB01 'fi' -> U+FB01
* 'fi' -> U+FB01 'fi' -> U+FB01
DFLT 'fi' -> 'f','i' 'fi' -> U+FB01
zyyy 'fi' -> 'f','i' 'fi' -> U+FB01
auto 'fi' -> 'f','i' 'fi' -> U+FB01
{noformat}
The language argument opens the same hole. A language with no langsys entry
under the script makes the (script, language) match come up empty, and the
fallback then goes to (DFLT, dflt), skipping the script's own default langsys.
Measured 2026-09-27 through the command line, one fo:block, only the language
attribute changing, read back with pdftotext:
{noformat}
attributes Carlito DejaVu Sans
script=latn, no language ffi ffl fi fl ft ti ligated ffi ffl fi fl
ligated
script=latn language="en" nothing ligated nothing ligated
{noformat}
And the language never matches in the first place. The FO language property
carries an ISO 639 code ("en", "tr"); the font's language systems are OpenType
tags ("ENG", "TRK"); FOP passes the code through untranslated. So a language
the font does distinguish is treated like one it does not: DejaVuLGCSerif's TRK
system omits liga, and performSubstitution("fi", "latn", "tr") ligates while
("fi", "latn", "TRK") does not.
Neither font has an ENG langsys under latn. DejaVu Sans does have DFLT, but its
DFLT langsys carries case, ccmp, dlig and kern and no liga, so the fallback
lands on a script table that lacks the feature. DejaVu Serif's DFLT langsys
does carry liga, and we measured that it ligates under the same attributes. So
whether a run ligates today is font-accidental: it depends on which features
the font's author happened to put in DFLT, not on what the font offers for the
script.
This is not a corner case for a consumer. docx4j writes language="en"
country="US" on every block and inline it produces, from w:lang and the
document defaults, and it cannot drop the attribute because FOP selects
hyphenation patterns by it. The first point of FOP-2092 (Thai shapes only under
language="dflt") is this language half. The fix belongs in the fallback order,
as OpenType layout engines do it: (script, language), then (script, dflt), then
(DFLT, dflt). GlyphTable.matchLookups goes straight from the first to the third.
It is not only substitution. The same script matching governs GPOS, so kerning
disappears too, and that is the wider harm because it reaches every letter pair
rather than a handful of ligature sequences:
{noformat}
Carlito legacy kern table: absent, 0 entries
GPOS 'AV' latn kerned * kerned DFLT null
DejaVu legacy kern table: present, 101 entries
GPOS 'AV' latn kerned * kerned DFLT kerned
{noformat}
Carlito has no legacy kern table at all, so under a default script it gets no
kerning from either mechanism. A consumer that sets kerning="true" on such a
font gets nothing and is told nothing.
Note that the wildcard script "*" works, because the wildcard matches the
font's real script tables. GlyphMapping maps "auto" and "zyyy" to "*", so the
common FO path is safe; it is a caller that passes DFLT, or "auto"/"zyyy"
unnormalised, that loses everything.
h3. Why it matters
The failure is silent and invisible in rendering: the text is drawn, just
unshaped and unkerned. Nobody notices until they compare advance widths against
another renderer. It also produces exactly the misleading symptom this was
first mistaken for: "FOP reads this font's tables but not that one's".
h3. Proposed fix
Done on the fork, branch CR-003-script-fallback (GlyphTable, OTFLanguage,
OTFScript; tests GlyphTableFallbackTestCase on synthetic tables shaped like
Carlito and DejaVu Sans, and LanguageSystemTestCase on the DejaVuLGCSerif in
the test tree, whose TRK system omits liga):
# Map the language to its OpenType tag first (a table of ISO 639 codes against
the tags OTFLanguage already names: "tr" to TRK, "en-US" to ENG, "zh-TW" to
ZHT; a tag is accepted as given; unknown codes match nothing and take the next
step), then fall back in the order OpenType layout engines use: (script,
language), then (script, dflt), then (DFLT, dflt). Each step is taken only when
the one before found nothing.
# OTFScript.isWildCard compares with the wildcard rather than DEFAULT (harmless
today, corrected alongside).
Measured after the change, same command line: Carlito under language="en"
ligates (ffi fi fl ft ti, 26 glyphs to 21) and kerns (AVATAR 43.19 pt, was
46.62); DejaVu Sans under "en" ligates; DejaVuLGCSerif
performSubstitution("fi", "latn", "tr") no longer ligates, which is what its
TRK system says; Noto Sans Arabic under "ar" unchanged.
This changes shaping and kerning for existing documents wherever a language was
set and the font had no system for it, so it wants the release note. A safer
first step, if preferred: an event when a font with GSUB or GPOS present yields
no lookups for the requested script, turning the silent miss into a diagnosable
one.
Note that the kerning half is also gated by the font configuration's kerning
flag, which did not reach GPOS at all (a separate report, FOP-3343), and by a
reader defect that drops a script's default language system when a named
language shares its table (FOP-3342): DejaVu Sans needs all three before it
kerns under "en".
h3. Provenance
Found while implementing the docx4j FO renderer's gsub-features hook, tracing
why one font shaped and another did not through the same pipeline. All the
numbers above are from loading each font directly with FontLoader and calling
performSubstitution and performPositioning, so no renderer or consumer pipeline
is involved.
was:
GlyphTable.matchLookups falls back to the default script when a lookup match
comes up empty:
{noformat}
if (lm.isEmpty() && !OTFScript.isDefault(script) &&
!OTFScript.isWildCard(script)) {
return matchLookups(OTFScript.DEFAULT, OTFLanguage.DEFAULT, feature);
}
{noformat}
The fallback targets DFLT. So for a font that has no DFLT entry in its script
list, a script of DFLT finds nothing, and the fallback looks for DFLT again and
finds nothing again. Nothing is applied and nothing is reported. The font is
loaded, its tables are read, performsSubstitution and performsPositioning both
return true, and every lookup silently misses.
Fonts without a DFLT script table are not unusual. Carlito, the
metric-compatible Calibri substitute packaged by most Linux distributions,
declares cyrl, grek and latn and no DFLT. DejaVu declares DFLT alongside them,
which is why the two behave differently on the same code path and why this is
easy to mistake for a font-file or a loader problem.
Measured, on the same font objects with only the script argument changing:
{noformat}
script Carlito 1.104 DejaVuLGCSerif
latn 'fi' -> U+FB01 'fi' -> U+FB01
* 'fi' -> U+FB01 'fi' -> U+FB01
DFLT 'fi' -> 'f','i' 'fi' -> U+FB01
zyyy 'fi' -> 'f','i' 'fi' -> U+FB01
auto 'fi' -> 'f','i' 'fi' -> U+FB01
{noformat}
The language argument opens the same hole. A language with no langsys entry
under the script makes the (script, language) match come up empty, and the
fallback then goes to (DFLT, dflt), skipping the script's own default langsys.
Measured 2026-09-27 through the command line, one fo:block, only the language
attribute changing, read back with pdftotext:
{noformat}
attributes Carlito DejaVu Sans
script=latn, no language ffi ffl fi fl ft ti ligated ffi ffl fi fl
ligated
script=latn language="en" nothing ligated nothing ligated
{noformat}
And the language never matches in the first place. The FO language property
carries an ISO 639 code ("en", "tr"); the font's language systems are OpenType
tags ("ENG", "TRK"); FOP passes the code through untranslated. So a language
the font does distinguish is treated like one it does not: DejaVuLGCSerif's TRK
system omits liga, and performSubstitution("fi", "latn", "tr") ligates while
("fi", "latn", "TRK") does not.
Neither font has an ENG langsys under latn. DejaVu Sans does have DFLT, but its
DFLT langsys carries case, ccmp, dlig and kern and no liga, so the fallback
lands on a script table that lacks the feature. DejaVu Serif's DFLT langsys
does carry liga, and we measured that it ligates under the same attributes. So
whether a run ligates today is font-accidental: it depends on which features
the font's author happened to put in DFLT, not on what the font offers for the
script.
This is not a corner case for a consumer. docx4j writes language="en"
country="US" on every block and inline it produces, from w:lang and the
document defaults, and it cannot drop the attribute because FOP selects
hyphenation patterns by it. The first point of FOP-2092 (Thai shapes only under
language="dflt") is this language half. The fix belongs in the fallback order,
as OpenType layout engines do it: (script, language), then (script, dflt), then
(DFLT, dflt). GlyphTable.matchLookups goes straight from the first to the third.
It is not only substitution. The same script matching governs GPOS, so kerning
disappears too, and that is the wider harm because it reaches every letter pair
rather than a handful of ligature sequences:
{noformat}
Carlito legacy kern table: absent, 0 entries
GPOS 'AV' latn kerned * kerned DFLT null
DejaVu legacy kern table: present, 101 entries
GPOS 'AV' latn kerned * kerned DFLT kerned
{noformat}
Carlito has no legacy kern table at all, so under a default script it gets no
kerning from either mechanism. A consumer that sets kerning="true" on such a
font gets nothing and is told nothing.
Note that the wildcard script "*" works, because the wildcard matches the
font's real script tables. GlyphMapping maps "auto" and "zyyy" to "*", so the
common FO path is safe; it is a caller that passes DFLT, or "auto"/"zyyy"
unnormalised, that loses everything.
h3. Why it matters
The failure is silent and invisible in rendering: the text is drawn, just
unshaped and unkerned. Nobody notices until they compare advance widths against
another renderer. It also produces exactly the misleading symptom this was
first mistaken for: "FOP reads this font's tables but not that one's".
h3. Proposed fix
Done on the fork, branch CR-003-script-fallback (GlyphTable, OTFLanguage,
OTFScript; tests GlyphTableFallbackTestCase on synthetic tables shaped like
Carlito and DejaVu Sans, and LanguageSystemTestCase on the DejaVuLGCSerif in
the test tree, whose TRK system omits liga):
# Map the language to its OpenType tag first (a table of ISO 639 codes against
the tags OTFLanguage already names: "tr" to TRK, "en-US" to ENG, "zh-TW" to
ZHT; a tag is accepted as given; unknown codes match nothing and take the next
step), then fall back in the order OpenType layout engines use: (script,
language), then (script, dflt), then (DFLT, dflt). Each step is taken only when
the one before found nothing.
# OTFScript.isWildCard compares with the wildcard rather than DEFAULT (harmless
today, corrected alongside).
Measured after the change, same command line: Carlito under language="en"
ligates (ffi fi fl ft ti, 26 glyphs to 21) and kerns (AVATAR 43.19 pt, was
46.62); DejaVu Sans under "en" ligates; DejaVuLGCSerif
performSubstitution("fi", "latn", "tr") no longer ligates, which is what its
TRK system says; Noto Sans Arabic under "ar" unchanged.
This changes shaping and kerning for existing documents wherever a language was
set and the font had no system for it, so it wants the release note. A safer
first step, if preferred: an event when a font with GSUB or GPOS present yields
no lookups for the requested script, turning the silent miss into a diagnosable
one.
Note that the kerning half is also gated by the font configuration's kerning
flag, which did not reach GPOS at all (a separate report, a companion issue
filed alongside this one), and by a reader defect that drops a script's default
language system when a named language shares its table (a companion issue filed
alongside this one): DejaVu Sans needs all three before it kerns under "en".
h3. Provenance
Found while implementing the docx4j FO renderer's gsub-features hook, tracing
why one font shaped and another did not through the same pipeline. All the
numbers above are from loading each font directly with FontLoader and calling
performSubstitution and performPositioning, so no renderer or consumer pipeline
is involved.
> A font whose GSUB or GPOS has no DFLT script table silently gets no
> substitution and no kerning under a default script, with no warning
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FOP-3341
> URL: https://issues.apache.org/jira/browse/FOP-3341
> Project: FOP
> Issue Type: Bug
> Components: font/opentype
> Affects Versions: 2.11
> Reporter: Jason Harrop
> Priority: Minor
>
> GlyphTable.matchLookups falls back to the default script when a lookup match
> comes up empty:
> {noformat}
> if (lm.isEmpty() && !OTFScript.isDefault(script) &&
> !OTFScript.isWildCard(script)) {
> return matchLookups(OTFScript.DEFAULT, OTFLanguage.DEFAULT, feature);
> }
> {noformat}
> The fallback targets DFLT. So for a font that has no DFLT entry in its script
> list, a script of DFLT finds nothing, and the fallback looks for DFLT again
> and finds nothing again. Nothing is applied and nothing is reported. The font
> is loaded, its tables are read, performsSubstitution and performsPositioning
> both return true, and every lookup silently misses.
> Fonts without a DFLT script table are not unusual. Carlito, the
> metric-compatible Calibri substitute packaged by most Linux distributions,
> declares cyrl, grek and latn and no DFLT. DejaVu declares DFLT alongside
> them, which is why the two behave differently on the same code path and why
> this is easy to mistake for a font-file or a loader problem.
> Measured, on the same font objects with only the script argument changing:
> {noformat}
> script Carlito 1.104 DejaVuLGCSerif
> latn 'fi' -> U+FB01 'fi' -> U+FB01
> * 'fi' -> U+FB01 'fi' -> U+FB01
> DFLT 'fi' -> 'f','i' 'fi' -> U+FB01
> zyyy 'fi' -> 'f','i' 'fi' -> U+FB01
> auto 'fi' -> 'f','i' 'fi' -> U+FB01
> {noformat}
> The language argument opens the same hole. A language with no langsys entry
> under the script makes the (script, language) match come up empty, and the
> fallback then goes to (DFLT, dflt), skipping the script's own default
> langsys. Measured 2026-09-27 through the command line, one fo:block, only the
> language attribute changing, read back with pdftotext:
> {noformat}
> attributes Carlito DejaVu Sans
> script=latn, no language ffi ffl fi fl ft ti ligated ffi ffl fi fl
> ligated
> script=latn language="en" nothing ligated nothing ligated
> {noformat}
> And the language never matches in the first place. The FO language property
> carries an ISO 639 code ("en", "tr"); the font's language systems are
> OpenType tags ("ENG", "TRK"); FOP passes the code through untranslated. So a
> language the font does distinguish is treated like one it does not:
> DejaVuLGCSerif's TRK system omits liga, and performSubstitution("fi", "latn",
> "tr") ligates while ("fi", "latn", "TRK") does not.
> Neither font has an ENG langsys under latn. DejaVu Sans does have DFLT, but
> its DFLT langsys carries case, ccmp, dlig and kern and no liga, so the
> fallback lands on a script table that lacks the feature. DejaVu Serif's DFLT
> langsys does carry liga, and we measured that it ligates under the same
> attributes. So whether a run ligates today is font-accidental: it depends on
> which features the font's author happened to put in DFLT, not on what the
> font offers for the script.
> This is not a corner case for a consumer. docx4j writes language="en"
> country="US" on every block and inline it produces, from w:lang and the
> document defaults, and it cannot drop the attribute because FOP selects
> hyphenation patterns by it. The first point of FOP-2092 (Thai shapes only
> under language="dflt") is this language half. The fix belongs in the fallback
> order, as OpenType layout engines do it: (script, language), then (script,
> dflt), then (DFLT, dflt). GlyphTable.matchLookups goes straight from the
> first to the third.
> It is not only substitution. The same script matching governs GPOS, so
> kerning disappears too, and that is the wider harm because it reaches every
> letter pair rather than a handful of ligature sequences:
> {noformat}
> Carlito legacy kern table: absent, 0 entries
> GPOS 'AV' latn kerned * kerned DFLT null
> DejaVu legacy kern table: present, 101 entries
> GPOS 'AV' latn kerned * kerned DFLT kerned
> {noformat}
> Carlito has no legacy kern table at all, so under a default script it gets no
> kerning from either mechanism. A consumer that sets kerning="true" on such a
> font gets nothing and is told nothing.
> Note that the wildcard script "*" works, because the wildcard matches the
> font's real script tables. GlyphMapping maps "auto" and "zyyy" to "*", so the
> common FO path is safe; it is a caller that passes DFLT, or "auto"/"zyyy"
> unnormalised, that loses everything.
> h3. Why it matters
> The failure is silent and invisible in rendering: the text is drawn, just
> unshaped and unkerned. Nobody notices until they compare advance widths
> against another renderer. It also produces exactly the misleading symptom
> this was first mistaken for: "FOP reads this font's tables but not that
> one's".
> h3. Proposed fix
> Done on the fork, branch CR-003-script-fallback (GlyphTable, OTFLanguage,
> OTFScript; tests GlyphTableFallbackTestCase on synthetic tables shaped like
> Carlito and DejaVu Sans, and LanguageSystemTestCase on the DejaVuLGCSerif in
> the test tree, whose TRK system omits liga):
> # Map the language to its OpenType tag first (a table of ISO 639 codes
> against the tags OTFLanguage already names: "tr" to TRK, "en-US" to ENG,
> "zh-TW" to ZHT; a tag is accepted as given; unknown codes match nothing and
> take the next step), then fall back in the order OpenType layout engines use:
> (script, language), then (script, dflt), then (DFLT, dflt). Each step is
> taken only when the one before found nothing.
> # OTFScript.isWildCard compares with the wildcard rather than DEFAULT
> (harmless today, corrected alongside).
> Measured after the change, same command line: Carlito under language="en"
> ligates (ffi fi fl ft ti, 26 glyphs to 21) and kerns (AVATAR 43.19 pt, was
> 46.62); DejaVu Sans under "en" ligates; DejaVuLGCSerif
> performSubstitution("fi", "latn", "tr") no longer ligates, which is what its
> TRK system says; Noto Sans Arabic under "ar" unchanged.
> This changes shaping and kerning for existing documents wherever a language
> was set and the font had no system for it, so it wants the release note. A
> safer first step, if preferred: an event when a font with GSUB or GPOS
> present yields no lookups for the requested script, turning the silent miss
> into a diagnosable one.
> Note that the kerning half is also gated by the font configuration's kerning
> flag, which did not reach GPOS at all (a separate report, FOP-3343), and by a
> reader defect that drops a script's default language system when a named
> language shares its table (FOP-3342): DejaVu Sans needs all three before it
> kerns under "en".
> h3. Provenance
> Found while implementing the docx4j FO renderer's gsub-features hook, tracing
> why one font shaped and another did not through the same pipeline. All the
> numbers above are from loading each font directly with FontLoader and calling
> performSubstitution and performPositioning, so no renderer or consumer
> pipeline is involved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
