[
https://issues.apache.org/jira/browse/PDFBOX-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo A. updated PDFBOX-6252:
-------------------------------
Description:
h2. Summary
{{PDDocument#saveIncremental}} derives the update section's trailer and
cross-reference entries
from the objects it holds in memory rather than from the combination of its own
section with the
ones already in the file. On a document whose cross-reference marks an object
free, two things
follow: {{/Size}} shrinks below the address space the file already claims, and
that free number is
handed to a new object without the writer knowing it was ever spoken for.
No signing is involved, no cross-reference stream, and no file produced by an
older PDFBox: the
input is a five-object PDF written by hand. Neither is a regression. Measured
identically on
3.0.6, 3.0.7, 3.0.8 and on the 3.0.9-SNAPSHOT and 4.0.0-SNAPSHOT builds of
2026-09-03 (JDK 25).
h2. Status
Both readings are fixed on the 3.0 branch and trunk by r1937934 and r1937935,
committed for
PDFBOX-6236, which seed the writer from the previous trailer {{/Size}}. On the
3.0.9-SNAPSHOT and
4.0.0-SNAPSHOT builds of 2026-09-07 the title-only save keeps {{/Size 7}} and
the added page is
written as {{7 0 obj}}, leaving {{/Outlines 6 0 R}} dangling as it was. The 2.0
branch is not
covered: 2.0.38-SNAPSHOT of the same day still writes {{/Size 6}}.
h2. Steps to reproduce
Input ({{free-object-input.pdf}}, attached): five objects, table covering 0..6
in one subsection,
{{/Size 7}}, object 6 free with the list 0 -> 6 -> end, catalog {{/Outlines 6 0
R}}.
{code:java}
try (PDDocument doc = Loader.loadPDF(input))
{
doc.getDocumentInformation().setTitle("a change, so the save has something
to write");
// doc.addPage(new PDPage()); // second run, for reading 2
doc.saveIncremental(out);
}
{code}
h2. 1. /Size shrinks (title-only save)
{noformat}
input /Size 7
update /Size 6
{noformat}
Table 15 defines it over "the combination of the original section and all
update sections", and
says any object numbered above it "shall be ignored and defined to be missing
by a conforming
reader" -- so the catalog's {{/Outlines 6 0 R}} is instructed to be missing.
The same happens on a
cross-reference stream input ({{XrefStreamTopFree.java}}: 9 -> 8), where table
17 puts it even more
directly: "the number one greater than the highest object number used in this
section _or in any
section for which this shall be an update_".
h2. 2. The free number is handed to a new object (save that adds a page)
The added page is written as {{6 0 obj}} -- the number the previous section
marks free and the
catalog still references. Read back, {{/Outlines 6 0 R}} from the earlier
revision resolves to the
new page, in PDFBox and in pypdf 6.16.2:
{noformat}
BEFORE /Outlines -> 6 0 R "Object 6 0 not defined." (dangling)
AFTER /Outlines -> 6 0 R {'/Type': '/Page', '/MediaBox': [...]}
{noformat}
Reusing a free number is legal on its own. The objection is that PDFBox is not
choosing to reuse
anything: the number simply looks unallocated. When the earlier revision is
signed, what it says
changes underneath it.
h2. The mechanism
Both readings follow from one seed. {{COSWriter#write}} starts numbering at
{{getHighestXRefObjectNumber()}}, which counts only numbers that have an
*in-use* entry, and
{{/Size}} is then written as that ceiling + 1. A number the file's own trailer
covers but that
carries no in-use entry is invisible to both. Trunk as of 2026-09-03:
* {{COSWriter#write(PDDocument, SignatureInterface)}}:
{{number = pdDocument.getDocument().getHighestXRefObjectNumber()}}
* {{COSWriter#doWriteTrailer(COSDocument)}}: {{trailer.setLong(COSName.SIZE,
number + 1)}}; the
stream path does the equivalent in {{doWriteXRefInc}} with
{{pdfxRefStream.setSize(number + 1)}}.
h2. Suggested fix
* Seed with {{max(getHighestXRefObjectNumber(), previous trailer /Size - 1)}}.
The previous trailer
already declares the address space the file claims, whether or not every number
in it has an entry,
so the writer needs to know nothing about free lists. On a well-formed file the
two agree and
nothing changes.
* Write {{/Size}} as {{max(previous trailer /Size, highest object number
written + 1)}}. An
incremental update can never lower the address space of the file it extends.
r1937934 and r1937935 implement the first. The second follows from it, since
{{/Size}} is written
as the seed + 1.
Not suggested: repairing the document before the save. Rewriting existing bytes
breaks every
signature already in the file, which is what an incremental save exists to
avoid. A repair has to
be appended as its own revision, by the caller.
h2. Relation to PDFBOX-6236 and PDFBOX-5382
6236 is the same "the number looks unallocated" condition reached from a
different direction: there
the invisible number belonged to the increment's own {{/XRef}} stream, which
had no entry in its
own revision. PDFBOX-6176 fixed that entry in 3.0.8, and the fix for 6236
covers the free-entry
direction as well.
5382 is the read side of the same missing information -- {{COSDocument}}'s
cross-reference map
holds no entries for free objects, as Michael Klink pointed out there in 2022.
So
{{SigUtils.checkCrossReferenceTable}} does *not* warn on either file, before or
after the save:
{{getXrefTable()}} comes back with keys 1..5 and object 6 is absent from the
model. A control with
an actual hole does produce the warning, so this is not a logging artefact.
h2. 2.0 branch
2.0.37 writes the same update section as 3.0.8 for the same input, byte offsets
included, so
reading 1 holds there. Reading 2 does not: with the changed objects flagged,
2.0.37 numbers the
added page 7 and writes the free object out as {{6 0 obj null}}.
2.0.38-SNAPSHOT of 2026-09-07
still writes {{/Size 6}}, so the fix is not on that branch.
h2. Attachments
{{free-object-input.pdf}} / {{free-object-after-addpage.pdf}} and
{{topfree-input.pdf}} /
{{topfree-after-incremental.pdf}} are the inputs and outputs of the two
readings, a few hundred
bytes each, synthetic, no data of any kind. The reproducers --
{{IncrementalXrefRepro.java}},
{{XrefReuseRepro.java}}, {{XrefStreamRepro.java}}, {{XrefStreamTopFree.java}}
and
{{IncrementalXrefRepro20.java}} for the 2.0 API -- build their input in memory
and need only PDFBox
and its runtime dependencies, plus {{log4j-api}} on 4.0.0-SNAPSHOT.
{{ANALYSIS-full.txt}} is the long-form analysis behind the above: the full
walkthrough of the
clauses involved, the measurements build by build, and what the appended
section does to the free
list.
was:
h2. Summary
{{PDDocument#saveIncremental}} derives the update section's trailer and
cross-reference entries
from the objects it holds in memory rather than from the combination of its own
section with the
ones already in the file. On a document whose cross-reference marks an object
free, two things
follow: {{/Size}} shrinks below the address space the file already claims, and
that free number is
handed to a new object without the writer knowing it was ever spoken for.
No signing is involved, no cross-reference stream, and no file produced by an
older PDFBox: the
input is a five-object PDF written by hand. Measured identically on 3.0.6,
3.0.7, 3.0.8 and on the
3.0.9-SNAPSHOT and 4.0.0-SNAPSHOT builds of 2026-09-03 (JDK 25). Neither is a
regression; neither
is fixed in trunk.
h2. Steps to reproduce
Input ({{free-object-input.pdf}}, attached): five objects, table covering 0..6
in one subsection,
{{/Size 7}}, object 6 free with the list 0 -> 6 -> end, catalog {{/Outlines 6 0
R}}.
{code:java}
try (PDDocument doc = Loader.loadPDF(input))
{
doc.getDocumentInformation().setTitle("a change, so the save has something
to write");
// doc.addPage(new PDPage()); // second run, for reading 2
doc.saveIncremental(out);
}
{code}
h2. 1. /Size shrinks (title-only save)
{noformat}
input /Size 7
update /Size 6
{noformat}
Table 15 defines it over "the combination of the original section and all
update sections", and
says any object numbered above it "shall be ignored and defined to be missing
by a conforming
reader" -- so the catalog's {{/Outlines 6 0 R}} is instructed to be missing.
The same happens on a
cross-reference stream input ({{XrefStreamTopFree.java}}: 9 -> 8), where table
17 puts it even more
directly: "the number one greater than the highest object number used in this
section _or in any
section for which this shall be an update_".
h2. 2. The free number is handed to a new object (save that adds a page)
The added page is written as {{6 0 obj}} -- the number the previous section
marks free and the
catalog still references. Read back, {{/Outlines 6 0 R}} from the earlier
revision resolves to the
new page, in PDFBox and in pypdf 6.16.2:
{noformat}
BEFORE /Outlines -> 6 0 R "Object 6 0 not defined." (dangling)
AFTER /Outlines -> 6 0 R {'/Type': '/Page', '/MediaBox': [...]}
{noformat}
Reusing a free number is legal on its own. The objection is that PDFBox is not
choosing to reuse
anything: the number simply looks unallocated. When the earlier revision is
signed, what it says
changes underneath it.
h2. The mechanism
Both readings follow from one seed. {{COSWriter#write}} starts numbering at
{{getHighestXRefObjectNumber()}}, which counts only numbers that have an
*in-use* entry, and
{{/Size}} is then written as that ceiling + 1. A number the file's own trailer
covers but that
carries no in-use entry is invisible to both. Trunk as of 2026-09-03:
* {{COSWriter#write(PDDocument, SignatureInterface)}}:
{{number = pdDocument.getDocument().getHighestXRefObjectNumber()}}
* {{COSWriter#doWriteTrailer(COSDocument)}}: {{trailer.setLong(COSName.SIZE,
number + 1)}}; the
stream path does the equivalent in {{doWriteXRefInc}} with
{{pdfxRefStream.setSize(number + 1)}}.
h2. Suggested fix
* Seed with {{max(getHighestXRefObjectNumber(), previous trailer /Size - 1)}}.
The previous trailer
already declares the address space the file claims, whether or not every number
in it has an entry,
so the writer needs to know nothing about free lists. On a well-formed file the
two agree and
nothing changes.
* Write {{/Size}} as {{max(previous trailer /Size, highest object number
written + 1)}}. An
incremental update can never lower the address space of the file it extends.
Not suggested: repairing the document before the save. Rewriting existing bytes
breaks every
signature already in the file, which is what an incremental save exists to
avoid. A repair has to
be appended as its own revision, by the caller.
h2. Relation to PDFBOX-6236 and PDFBOX-5382
6236 is the same "the number looks unallocated" condition reached from a
different direction: there
the invisible number belonged to the increment's own {{/XRef}} stream, which
had no entry in its
own revision. PDFBOX-6176 fixed that entry in 3.0.8; the free-entry direction
is not fixed. The
first suggestion covers both.
5382 is the read side of the same missing information -- {{COSDocument}}'s
cross-reference map
holds no entries for free objects, as Michael Klink pointed out there in 2022.
So
{{SigUtils.checkCrossReferenceTable}} does *not* warn on either file, before or
after the save:
{{getXrefTable()}} comes back with keys 1..5 and object 6 is absent from the
model. A control with
an actual hole does produce the warning, so this is not a logging artefact.
h2. 2.0 branch
2.0.37 writes the same update section as 3.0.8 for the same input, byte offsets
included, so
reading 1 holds there. Reading 2 does not: with the changed objects flagged,
2.0.37 numbers the
added page 7 and writes the free object out as {{6 0 obj null}}.
h2. Attachments
{{free-object-input.pdf}} / {{free-object-after-addpage.pdf}} and
{{topfree-input.pdf}} /
{{topfree-after-incremental.pdf}} are the inputs and outputs of the two
readings, a few hundred
bytes each, synthetic, no data of any kind. The reproducers --
{{IncrementalXrefRepro.java}},
{{XrefReuseRepro.java}}, {{XrefStreamRepro.java}}, {{XrefStreamTopFree.java}}
and
{{IncrementalXrefRepro20.java}} for the 2.0 API -- build their input in memory
and need only PDFBox
and its runtime dependencies, plus {{log4j-api}} on 4.0.0-SNAPSHOT.
{{ANALYSIS-full.txt}} is the long-form analysis behind the above: the full
walkthrough of the
clauses involved, the measurements build by build, and what the appended
section does to the free
list.
> PDDocument.saveIncremental() writes an update section that contradicts the
> sections it extends, on a document whose cross-reference marks an object free
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: PDFBOX-6252
> URL: https://issues.apache.org/jira/browse/PDFBOX-6252
> Project: PDFBox
> Issue Type: Bug
> Components: Writing
> Affects Versions: 2.0.37, 3.0.8 PDFBox, 4.0.0
> Reporter: Gustavo A.
> Priority: Major
> Attachments: ANALYSIS-full.txt, IncrementalXrefRepro.java,
> IncrementalXrefRepro20.java, XrefReuseRepro.java, XrefStreamRepro.java,
> XrefStreamTopFree.java, free-object-after-addpage.pdf, free-object-input.pdf,
> topfree-after-incremental.pdf, topfree-input.pdf
>
>
> h2. Summary
> {{PDDocument#saveIncremental}} derives the update section's trailer and
> cross-reference entries
> from the objects it holds in memory rather than from the combination of its
> own section with the
> ones already in the file. On a document whose cross-reference marks an object
> free, two things
> follow: {{/Size}} shrinks below the address space the file already claims,
> and that free number is
> handed to a new object without the writer knowing it was ever spoken for.
> No signing is involved, no cross-reference stream, and no file produced by an
> older PDFBox: the
> input is a five-object PDF written by hand. Neither is a regression. Measured
> identically on
> 3.0.6, 3.0.7, 3.0.8 and on the 3.0.9-SNAPSHOT and 4.0.0-SNAPSHOT builds of
> 2026-09-03 (JDK 25).
> h2. Status
> Both readings are fixed on the 3.0 branch and trunk by r1937934 and r1937935,
> committed for
> PDFBOX-6236, which seed the writer from the previous trailer {{/Size}}. On
> the 3.0.9-SNAPSHOT and
> 4.0.0-SNAPSHOT builds of 2026-09-07 the title-only save keeps {{/Size 7}} and
> the added page is
> written as {{7 0 obj}}, leaving {{/Outlines 6 0 R}} dangling as it was. The
> 2.0 branch is not
> covered: 2.0.38-SNAPSHOT of the same day still writes {{/Size 6}}.
> h2. Steps to reproduce
> Input ({{free-object-input.pdf}}, attached): five objects, table covering
> 0..6 in one subsection,
> {{/Size 7}}, object 6 free with the list 0 -> 6 -> end, catalog {{/Outlines 6
> 0 R}}.
> {code:java}
> try (PDDocument doc = Loader.loadPDF(input))
> {
> doc.getDocumentInformation().setTitle("a change, so the save has
> something to write");
> // doc.addPage(new PDPage()); // second run, for reading 2
> doc.saveIncremental(out);
> }
> {code}
> h2. 1. /Size shrinks (title-only save)
> {noformat}
> input /Size 7
> update /Size 6
> {noformat}
> Table 15 defines it over "the combination of the original section and all
> update sections", and
> says any object numbered above it "shall be ignored and defined to be missing
> by a conforming
> reader" -- so the catalog's {{/Outlines 6 0 R}} is instructed to be missing.
> The same happens on a
> cross-reference stream input ({{XrefStreamTopFree.java}}: 9 -> 8), where
> table 17 puts it even more
> directly: "the number one greater than the highest object number used in this
> section _or in any
> section for which this shall be an update_".
> h2. 2. The free number is handed to a new object (save that adds a page)
> The added page is written as {{6 0 obj}} -- the number the previous section
> marks free and the
> catalog still references. Read back, {{/Outlines 6 0 R}} from the earlier
> revision resolves to the
> new page, in PDFBox and in pypdf 6.16.2:
> {noformat}
> BEFORE /Outlines -> 6 0 R "Object 6 0 not defined." (dangling)
> AFTER /Outlines -> 6 0 R {'/Type': '/Page', '/MediaBox': [...]}
> {noformat}
> Reusing a free number is legal on its own. The objection is that PDFBox is
> not choosing to reuse
> anything: the number simply looks unallocated. When the earlier revision is
> signed, what it says
> changes underneath it.
> h2. The mechanism
> Both readings follow from one seed. {{COSWriter#write}} starts numbering at
> {{getHighestXRefObjectNumber()}}, which counts only numbers that have an
> *in-use* entry, and
> {{/Size}} is then written as that ceiling + 1. A number the file's own
> trailer covers but that
> carries no in-use entry is invisible to both. Trunk as of 2026-09-03:
> * {{COSWriter#write(PDDocument, SignatureInterface)}}:
> {{number = pdDocument.getDocument().getHighestXRefObjectNumber()}}
> * {{COSWriter#doWriteTrailer(COSDocument)}}: {{trailer.setLong(COSName.SIZE,
> number + 1)}}; the
> stream path does the equivalent in {{doWriteXRefInc}} with
> {{pdfxRefStream.setSize(number + 1)}}.
> h2. Suggested fix
> * Seed with {{max(getHighestXRefObjectNumber(), previous trailer /Size -
> 1)}}. The previous trailer
> already declares the address space the file claims, whether or not every
> number in it has an entry,
> so the writer needs to know nothing about free lists. On a well-formed file
> the two agree and
> nothing changes.
> * Write {{/Size}} as {{max(previous trailer /Size, highest object number
> written + 1)}}. An
> incremental update can never lower the address space of the file it extends.
> r1937934 and r1937935 implement the first. The second follows from it, since
> {{/Size}} is written
> as the seed + 1.
> Not suggested: repairing the document before the save. Rewriting existing
> bytes breaks every
> signature already in the file, which is what an incremental save exists to
> avoid. A repair has to
> be appended as its own revision, by the caller.
> h2. Relation to PDFBOX-6236 and PDFBOX-5382
> 6236 is the same "the number looks unallocated" condition reached from a
> different direction: there
> the invisible number belonged to the increment's own {{/XRef}} stream, which
> had no entry in its
> own revision. PDFBOX-6176 fixed that entry in 3.0.8, and the fix for 6236
> covers the free-entry
> direction as well.
> 5382 is the read side of the same missing information -- {{COSDocument}}'s
> cross-reference map
> holds no entries for free objects, as Michael Klink pointed out there in
> 2022. So
> {{SigUtils.checkCrossReferenceTable}} does *not* warn on either file, before
> or after the save:
> {{getXrefTable()}} comes back with keys 1..5 and object 6 is absent from the
> model. A control with
> an actual hole does produce the warning, so this is not a logging artefact.
> h2. 2.0 branch
> 2.0.37 writes the same update section as 3.0.8 for the same input, byte
> offsets included, so
> reading 1 holds there. Reading 2 does not: with the changed objects flagged,
> 2.0.37 numbers the
> added page 7 and writes the free object out as {{6 0 obj null}}.
> 2.0.38-SNAPSHOT of 2026-09-07
> still writes {{/Size 6}}, so the fix is not on that branch.
> h2. Attachments
> {{free-object-input.pdf}} / {{free-object-after-addpage.pdf}} and
> {{topfree-input.pdf}} /
> {{topfree-after-incremental.pdf}} are the inputs and outputs of the two
> readings, a few hundred
> bytes each, synthetic, no data of any kind. The reproducers --
> {{IncrementalXrefRepro.java}},
> {{XrefReuseRepro.java}}, {{XrefStreamRepro.java}}, {{XrefStreamTopFree.java}}
> and
> {{IncrementalXrefRepro20.java}} for the 2.0 API -- build their input in
> memory and need only PDFBox
> and its runtime dependencies, plus {{log4j-api}} on 4.0.0-SNAPSHOT.
> {{ANALYSIS-full.txt}} is the long-form analysis behind the above: the full
> walkthrough of the
> clauses involved, the measurements build by build, and what the appended
> section does to the free
> list.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
