[
https://issues.apache.org/jira/browse/PDFBOX-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111405#comment-18111405
]
Michael Klink commented on PDFBOX-6252:
---------------------------------------
I doubt you do your issue a great service by presenting it using this much
text.
I'm not part of the PDFBox development team. But if i was and if I was the
person to deal with that issue, I'd have to wait until I have a lot of time to
spare, enough time to study the whole text in detail. And i would not have that
much time this year anymore.
I quickly glanced over the text. And at first glance this appears to be a bug
that needs to be fixed. Reducing the *Size* in an incremental update is a no-go.
Concerning the free object lists, though, there is no guarantee that they will
remain unchanged. Remember that there may be multiple free object lists in the
cross reference of a PDF, and an incremental update may reorganize them
completely.
> PDDocument.saveIncremental() writes an update section that contradicts the
> sections it extends, on a document whose cross-reference marks an object free
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: PDFBOX-6252
> URL: https://issues.apache.org/jira/browse/PDFBOX-6252
> Project: PDFBox
> Issue Type: Bug
> Components: Writing
> Affects Versions: 2.0.37, 3.0.8 PDFBox, 4.0.0
> Reporter: Gustavo A.
> Priority: Major
> Attachments: IncrementalXrefRepro.java, IncrementalXrefRepro20.java,
> XrefReuseRepro.java, XrefStreamRepro.java, XrefStreamTopFree.java,
> free-object-after-addpage.pdf, free-object-input.pdf,
> topfree-after-incremental.pdf, topfree-input.pdf
>
>
> h2. Summary
> {{PDDocument#saveIncremental}} derives the update section's trailer and
> cross-reference entries
> from the objects it holds in memory, rather than from the combination of its
> own section with the
> ones already in the file. On a document whose cross-reference marks an object
> free, the section it
> appends contradicts the sections it extends in three ways: {{/Size}} shrinks,
> the free list is
> discarded, and a number the previous section maps into that free list is
> handed out to a new
> object.
> No signing is involved, and neither is a cross-reference stream nor a file
> produced by an older
> PDFBox: the input built by the attached reproducers is a five-object PDF
> written by hand. Measured
> identically on 3.0.6, 3.0.7, 3.0.8, and on the 3.0.9-SNAPSHOT and
> 4.0.0-SNAPSHOT builds published
> on 2026-09-03 (JDK 25). None of the three is a regression, and none is fixed
> in trunk.
> On the 2.0 branch the picture is partial, and worth stating separately rather
> than folding in.
> 2.0.37 writes the same update section as 3.0.8 does for the same input --
> {{/Size 6}}, the
> free-list head reset, the same byte offsets -- so readings 1 and 2 hold there
> too. Reading 3 does
> not: with the changed objects flagged, 2.0.37 numbers the added page 7 rather
> than 6, and writes
> the free object out as {{6 0 obj null}}. {{IncrementalXrefRepro20.java}} is
> attached for that
> branch; the 2.0 API needs {{PDDocument.load}} and {{setNeedToBeUpdated}} on
> the changed objects,
> and is otherwise the same file.
> h2. Steps to reproduce
> All the save does is this, on the attached {{free-object-input.pdf}}:
> {code:java}
> try (PDDocument doc = Loader.loadPDF(input))
> {
> doc.getDocumentInformation().setTitle("a change, so the save has
> something to write");
> // doc.addPage(new PDPage()); // uncomment for reading 3 below
> doc.saveIncremental(out);
> }
> {code}
> {{IncrementalXrefRepro.java}} is attached and does exactly that. It needs
> nothing but PDFBox and
> its own runtime dependencies, and builds the same input in memory, so it can
> be run on its own:
> {noformat}
> java -cp
> "pdfbox-3.0.8.jar;pdfbox-io-3.0.8.jar;fontbox-3.0.8.jar;commons-logging-1.4.0.jar"
> IncrementalXrefRepro.java
> {noformat}
> The input is a five-object PDF whose cross-reference table covers 0..6 in a
> single subsection,
> declares {{/Size 7}}, marks object 6 free with the free list linked 0 -> 6 ->
> end, and whose
> catalog references it as {{/Outlines 6 0 R}}. That is the shape a document
> takes when an exporter
> drops an outline tree but leaves the reference behind; it is common enough in
> office-suite output
> that we ran into it in production. The program changes only the
> document-information title, so the
> update section is as small as an update section gets.
> The reproducer runs the same input a second time with nothing referencing
> object 6 at all. The
> update section comes out identical, so the dangling reference is not what is
> at fault.
> There is no exception and so no stack trace. With logging active, nothing is
> written at INFO level
> or above during either the load or the save; that silence is part of what is
> being reported, and
> the PDFBOX-5382 section below covers why
> {{SigUtils.checkCrossReferenceTable}} does not catch it
> either.
> h2. Expected result
> The appended section, combined with the one it extends, still describes the
> same address space:
> {{/Size}} stays at least 7, object 6 stays free and reachable from the head
> of the free list, and
> any object the save adds gets a number no previous section has already spoken
> for.
> h2. Actual result
> {noformat}
> original cross-reference section
> /Size : 7
> free-list head : 0000000006 65535 f (object 0 -> object 6)
> object 6 : free, end of the list
> catalog : /Outlines 6 0 R
> update section written by saveIncremental
> /Size : 6
> free-list head : 0000000000 65535 f
> {noformat}
> h3. 1. /Size shrinks
> The input declares {{/Size 7}}; the update section declares {{/Size 6}}.
> Table 15 defines it as
> "the total number of entries in the file's cross-reference table, as defined
> by the combination of
> the original section and all update sections" -- seven entries here, numbered
> 0 to 6 -- and goes on
> to say that "any object in a cross-reference section whose number is greater
> than this value shall
> be ignored and defined to be missing by a conforming reader". So this is not
> a cosmetic
> discrepancy: the spec instructs readers to treat object 6 as gone, while the
> catalog still
> references it as {{/Outlines 6 0 R}}.
> 7.5.6 says the same thing from the other side: "The added trailer shall
> contain all the entries
> except the Prev entry (if present) from the previous trailer, whether
> modified or not". And 7.5.4
> requires the combined table to "contain one entry for each object number from
> 0 to the maximum
> object number defined in the file, even if one or more of the object numbers
> in this range do not
> actually occur in the file", which a lowered {{/Size}} silently undoes.
> h3. 2. The free list is broken
> The update section writes {{0 1 / 0000000000 65535 f}}, declaring the free
> list empty, while the
> previous section still marks object 6 free with generation 0. 7.5.4 gives
> free entries exactly two
> lawful shapes: members of the linked list headed by object 0, or entries that
> "link back to object
> number 0 and have a generation number of 65,535, even though these entries
> are not in the linked
> list itself". After the update, object 6 is neither: it is not reachable from
> the head, and its
> generation is 0, not 65,535.
> Annex H.7 shows what that entry is _for_. In H.7.3, an update section that
> frees two objects writes
> {{0 1 / 0000000008 65535 f}}, moving the head onto the object it just freed;
> in H.7.4, which reuses
> both of them, it writes {{0000000000 65535 f}}, because by then the list
> really is empty. That
> entry is how the head is carried across revisions. PDFBox writes the second
> form unconditionally.
> The same example never lowers {{/Size}} either: 12, 12, 12, then 13.
> h3. 3. A number the previous section maps into its free list is handed out to
> a new object
> The two readings above come from a save that adds nothing. If the same save
> adds any object -- a
> {{PDPage}} in the attached {{XrefReuseRepro.java}}, though the type is
> irrelevant -- that object is
> written as {{6 0 obj}}, the number the previous section marks free and the
> catalog still
> references. Read back, {{/Outlines 6 0 R}} from the earlier revision resolves
> to the new page, in
> PDFBox itself and in pypdf 6.16.2:
> {noformat}
> BEFORE /Outlines -> IndirectObject(6, 0) "Object 6 0 not defined."
> (dangling)
> AFTER /Outlines -> IndirectObject(6, 0) {'/Type': '/Page', '/MediaBox':
> [...]}
> {noformat}
> Reusing a free number is legal on its own -- 7.5.4 makes the free entry's
> generation the one to use
> on reuse -- so the objection is not the reuse. It is that PDFBox is not
> choosing to reuse anything:
> the number looks unallocated, because the free entry is not in the model.
> When the earlier revision
> is signed, what it says changes underneath it.
> h2. What decides it is the seed, not the kind of section
> {{/Size}} is written as _highest object number written_ + 1, and the
> numbering starts from
> {{COSDocument#getHighestXRefObjectNumber()}}, which counts only numbers that
> have an in-use entry.
> So all three readings follow from one thing: numbers the previous sections
> put in the free list are
> invisible, and everything the writer decides is measured from a ceiling that
> is therefore too low.
> Two further reproducers make that concrete rather than asserted:
> * {{XrefStreamRepro.java}} -- a cross-reference _stream_ input with object 6
> free but the xref
> stream object (7) above it. {{/Size}} comes out right (8 -> 9) and the one
> new object the save
> writes, its own {{/XRef}} stream, is numbered 8, because the visible ceiling
> happens to be above
> the free number. Only reading 2 shows.
> * {{XrefStreamTopFree.java}} -- the same, with the free objects at the _top_
> of the address space
> (1..5 in use, 6 the xref stream, 7 and 8 free, {{/Size 9}}, catalog
> {{/Outlines 7 0 R}}). Here
> {{/Size}} drops to 8 on every build tested, and the update's own
> cross-reference stream is written
> as object *7*, so {{/Outlines 7 0 R}} resolves to the {{/XRef}} dictionary
> itself -- the same
> reading PDFBOX-6236 describes, reached from a free entry instead of a missing
> one. For a
> cross-reference stream the requirement is stated even more directly than in
> table 15: table 17
> defines {{/Size}} as "the number one greater than the highest object number
> used in this section
> _or in any section for which this shall be an update_" -- 9 here, not 8.
> So reading 1 is not specific to cross-reference tables; it appears wherever
> free numbers sit above
> the highest in-use one.
> Incidentally, {{XrefStreamRepro.java}} also shows PDFBOX-6176 taking effect.
> The update's own xref
> stream is object 8, and 3.0.6 and 3.0.7 omit its entry:
> {noformat}
> 3.0.6, 3.0.7 /Index [0 1 5 1]
> 3.0.8, 3.0.9-SNAPSHOT, 4.0.0-SNAP /Index [0 1 5 1 8 1]
> {noformat}
> h2. Suggested area to investigate
> Reading trunk as of 2026-09-03:
> * {{COSWriter#write(PDDocument, SignatureInterface)}} seeds the allocator with
> {{number = pdDocument.getDocument().getHighestXRefObjectNumber()}}.
> * {{COSWriter#doWriteTrailer(COSDocument)}} writes
> {{trailer.setLong(COSName.SIZE, number + 1)}}
> for the non-xref-stream case; the stream path does the equivalent in
> {{doWriteXRefInc}} with
> {{pdfxRefStream.setSize(number + 1)}}.
> * {{COSWriter#doWriteXRefTable()}} writes the free-list entry unconditionally
> on the incremental
> branch: {{addXRefEntry(FreeXReference.NULL_ENTRY)}}. The
> {{fillGapsWithFreeEntries()}} next to it
> runs only when the save is _not_ incremental.
> * The cross-reference stream path arrives at the same entry by a different
> route:
> {{PDFXRefStream#getIndexEntry()}} always adds object 0 to the index, and
> {{PDFXRefStream#writeStreamData()}} always writes
> {{FreeXReference.NULL_ENTRY}} as the first entry
> of the stream. So reading 2 is not specific to tables either.
> h2. Suggested fix
> All three come from the same place -- the update section is computed from
> PDFBox's object model
> rather than from the file it extends -- and each has a narrow fix:
> * *Number allocation*: seed it with {{max(getHighestXRefObjectNumber(),
> previous trailer /Size - 1)}}
> rather than with {{getHighestXRefObjectNumber()}} alone. The previous trailer
> already declares the
> address space the file claims, whether or not every number in it has an
> entry, so object 6 stops
> looking unallocated without the writer having to learn what the free list
> holds. On a well-formed
> file the two values agree and nothing changes. The same rule also covers
> PDFBOX-6236, where the
> invisible number is a cross-reference stream object rather than a free one.
> * *The Size entry*: take {{max(previous trailer /Size, highest object number
> written + 1)}}
> instead of a count of the in-memory objects. An incremental update can never
> lower the address space of the file
> it extends.
> * *Free list*: carry the previous head forward instead of writing
> {{0000000000 65535 f}}
> unconditionally, which is what annex H.7's own update sections do. Omitting
> the entry altogether is
> equally correct and even smaller: 7.5.6 asks an update section for "entries
> only for objects that
> have been changed, replaced, or deleted", and object 0 has not changed.
> h2. What is not being suggested
> Since it is the natural next question: please do not repair the document
> before an incremental
> save. Rewriting anything in the existing bytes would break every signature
> already in the file,
> which is exactly what an incremental save exists to avoid. Any repair has to
> be appended as its own
> revision, by the caller, before the first signature. That is what we ended up
> doing on our side,
> and it is out of scope for PDFBox.
> Likewise, a strict mode that refuses malformed input would run against
> PDFBox's own design;
> tolerating broken documents is a large part of why people reach for it. The
> ask here is only that
> the section PDFBox itself writes not contradict the ones it is extending.
> h2. Relation to PDFBOX-6236
> That issue is the closest sibling, and the mechanism looks like the same one.
> There, a new field
> object is given the number of the increment's own {{/XRef}} stream, because
> that object had no
> entry in its own revision's cross-reference, so to the writer the number
> looked unallocated.
> PDFBOX-6176 fixed the missing entry in 3.0.8.
> What is reported here is the same "the number looks unallocated" condition
> arriving from a
> different direction -- a free entry rather than a missing one -- and that
> direction is _not_ fixed
> in 3.0.8.
> Measured on that issue's own attachment, for what it is worth: loading
> {{already-signed.pdf}} with
> 3.0.8 reports {{getHighestXRefObjectNumber() = 42}} while the newest trailer
> of the file says
> {{/Size 44}}, so {{addPage}} plus {{saveIncremental}} writes the new page as
> object 43 -- the number
> revision 4's cross-reference stream already occupies at offset 76040. PDFBox
> reads the result as
> two pages, pypdf 6.16.2 as one. Identical on all five builds tested. Both
> cases share the seed, and
> the first suggestion above fixes both.
> h2. Relation to PDFBOX-5382
> All three readings look like the write-side consequence of the limitation
> discussed there. In that
> thread, {{SigUtils.checkCrossReferenceTable}} was added in r1898484/r1898485
> and wired into the
> three signing examples in r1898487/r1898488; r1899085/r1899086 then reverted
> the call sites only
> ("revert, but keep a comment for now"), after a discussion in which Michael
> Klink pointed out that
> {{COSDocument}}'s cross-reference {{Map}} holds no entries for free objects,
> so an object number
> mapped into the free list cannot be told apart from one that is absent
> altogether. The method
> itself was never removed; it is in trunk today, and {{CreateSignature}} still
> carries the line
> {{// call SigUtils.checkCrossReferenceTable(document) if Adobe complains}}.
> That is the same missing information, one step later: because the free
> entries are gone by the time
> the update section is written, the writer has no free list to carry forward
> and no object 6 to
> count towards {{/Size}}.
> The difference worth drawing out is the direction. PDFBOX-5382 is about
> _noticing_ a gap in a
> document that arrives with one. What is reported here is on the other side:
> given a document where
> object 6 is mapped into the free list, which is a legitimate mapping, an
> incremental save hands
> back one where it is not, because the newest {{/Size}} no longer covers it
> and the newest free-list
> head no longer leads to it.
> Worth stating plainly, since it is the obvious question:
> {{SigUtils.checkCrossReferenceTable}} does
> *not* warn on either file, before or after the save.
> {{COSDocument#getXrefTable()}} comes back with
> keys 1..5 and {{getHighestXRefObjectNumber()}} with 5, so object 6 is absent
> from the model and
> nothing looks missing to the check. A control with an actual hole (object 4
> defined in no section)
> does produce the warning, so this is not a logging artefact. The blind spot
> is the same one Michael
> Klink described in 2022; the point here is only that the writer sits behind
> it too.
> h2. Impact
> A conforming reader is instructed by table 15 to treat object 6 as missing
> after the save, and the
> free list the file used to have is gone. Where it bites hardest is a signed
> document: references
> made by an earlier, signed revision can be made to resolve to different
> objects by a later
> incremental save, without any of the existing bytes being touched.
> How much that matters to a validator is the part PDFBOX-5382 already frames,
> and this report adds
> nothing to it. That issue's description says a gap "may fail validation in
> Acrobat Reader", citing
> mkl's accepted answer to SO 71267471, which notes that Adobe Reader "normally
> ignores violations of
> these requirements, but in the context of signature validation it is
> stricter", and that this
> "usually shows in situations where the PDF in question is signed and then
> some arbitrary
> incremental update is added" -- which is exactly the shape here. What is
> squarely PDFBox's is the
> other half: given a document whose {{/Size}} is 7, an incremental save hands
> back one whose newest
> section says 6.
> A last note for completeness, since it bears on how much of this is worth
> chasing: the document
> that started our investigation broke identically under a signing stack that
> uses no PDFBox at all.
> The defect that caused it was in the source PDF, and we fixed it on our side
> by appending a repair
> revision before the first signature. Nothing in this report is offered as the
> cause of that.
> h2. Attachments
> The PDFs are the inputs and outputs of the two table-based readings, in case
> it is easier to look
> at the bytes than to run anything. They are synthetic, a few hundred bytes
> each, and contain no
> data of any kind.
> * {{free-object-input.pdf}} -- the five-object input: {{/Size 7}}, object 6
> free, list 0 -> 6 ->
> end, catalog {{/Outlines 6 0 R}}.
> * {{free-object-after-addpage.pdf}} -- the same file after {{addPage}} plus
> {{saveIncremental}} on
> 3.0.8, where {{/Outlines 6 0 R}} now resolves to the new page.
> * {{topfree-input.pdf}} and {{topfree-after-incremental.pdf}} -- the
> cross-reference stream pair,
> where {{/Size}} drops from 9 to 8 and {{/Outlines 7 0 R}} ends up resolving
> to the update's own
> {{/XRef}} dictionary.
> * {{IncrementalXrefRepro.java}} -- readings 1 and 2, cross-reference table
> input, run twice (with
> and without a reference to the free object).
> * {{XrefReuseRepro.java}} -- reading 3, same input, the save adds a
> {{PDPage}}.
> * {{XrefStreamRepro.java}} -- cross-reference stream input whose free number
> is below the highest
> in-use one; only reading 2 shows.
> * {{XrefStreamTopFree.java}} -- cross-reference stream input whose free
> numbers are above it; the
> update's own {{/XRef}} stream takes a referenced free number.
> * {{IncrementalXrefRepro20.java}} -- the same input on the 2.0 API, showing
> readings 1 and 2 on
> 2.0.37 and the different numbering behaviour described above.
> All four run the same way as the first, needing only PDFBox and its runtime
> dependencies -- on
> 4.0.0-SNAPSHOT, {{log4j-api}} as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
