arib06 opened a new pull request, #3946: URL: https://github.com/apache/avro/pull/3946
## What is the purpose of the change AVRO-4241 made the zero-byte collection-element allocation cap (SystemLimitException) cumulative across a decoded datum, so a record with many `array<null>`-style fields cannot over-allocate in aggregate even when each field stays under the per-collection limit. That cumulative accounting only holds while an allocation scope is open. `GenericDatumReader.read` opens one for the whole datum, and the fast array reader opens one around its block loop, but the datum-level fast readers do not. When the fast reader is used standalone via `FastReaderBuilder.createDatumReader(...)` (i.e. not reached through `GenericDatumReader.read`), decoding a record resets the running total per array field, because each field's array reader opens and closes its own outermost scope. Each field then passes the per-collection check while the aggregate exceeds the cap, so a small payload can drive a large allocation. The fix wraps the reader returned by `createDatumReader` in a scope, matching `GenericDatum Reader.read`. Scopes nest, so the delegated path (where `GenericDatumReader.read` already opened the outer scope) is unchanged, and valid input decodes exactly as before. ## Verifying this change This change added tests and can be verified as follows: - Added `standaloneFastReaderRecordRejectedCumulativelyAcrossDatum`: a record with two `array<null>` fields of 600 nulls each (cap set to 1000) decoded through the standalone fast reader is rejected on the second field with `SystemLimitException`. Before the change both fields decoded, bypassing the cap. - Added `standaloneFastReaderRecordWithinCumulativeLimitStillDecodes`: two fields totalling 800 elements (under the cap) still decode, and the per-datum budget resets between reads. ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
