Knowing the footer offset would be really useful if passed down to whatever is implementing the input stream, along with the actual file size.
This can be used for prefetching the footer, as well as caching it (Azure ABFS, google GCS connectors): right now they guess that about 1MB is all they need. while readTail() can get bytes off the end, it doesn't pass that information down to the stream, to do its own thing. The Analytics stream which the AWS S3 team are getting into the s3a code ( https://issues.apache.org/jira/browse/HADOOP-19363) goes one step further than the others: it parses that footer itself and tries to predict where application code is going to read next: as you read one rowgroup it speculatively fetch the next one, even as the first one is downloaded. Again, it guesses on footer size: pass that in and they will know what to fetch and store. Ideally this should be accompanied by file type (parquet, avro) and your actual read plans (vectored, random, sequential, whole-file). With this information you can cut out a number of wasted/inefficient S3 calls, and tune fetching/caching policy appropriately. Anyway: +1 to footer length, and if already known, file length should come down too, along with that read plan. saying "parque, vectored, randomt" will be enough, which is what a draft PR i have for hadoop fileIO does. On Wed, 22 Jan 2025 at 03:39, Sreeram Garlapati <gsreeramku...@gmail.com> wrote: > Thanks for the nice idea/suggestion, Dan. > Yes, we have been employing a similar technique that you noted below and > kinda arrived at the conclusion that there is no deterministic way to > achieve that most optimal situation, ie., single i/o call to S3 to read the > parquet footer. > > Best, > Sreeram > > On Tue, Jan 21, 2025 at 4:20 PM Daniel Weeks <dwe...@apache.org> wrote: > >> Hey Sreeram, >> >> I think it's worthwhile to consider what value would be added by tracking >> the footer size in metadata, but there are other options to address these >> optimization use cases. >> >> For example, if you take a look at the RangeReadable >> <https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/io/RangeReadable.java#L68> >> interface >> for FileIO implementations, there's a readTail method so that you can >> optimistically read from the tail end of the file to try to fetch the full >> footer in a single read. This is even optimized in some of the >> implementations (like S3InputStream) to leverage backward reads as opposed >> to seek operations which might have overhead. >> >> Depending on the size of the file, you may want to load just the tail or >> the whole file to avoid all reads. Having the exact value definitely will >> make this more exact, but I feel like using the above approach can >> approximate the same performance benefits. >> >> Just a thought, >> -Dan >> >> On Tue, Jan 21, 2025 at 12:17 PM Sreeram Garlapati < >> gsreeramku...@gmail.com> wrote: >> >>> Hello Team! >>> >>> This is a small improvement proposal to store the *parquet footer size* >>> as part of the *data_file* metadata in the iceberg manifest >>> <https://iceberg.apache.org/spec/#manifests>. >>> *manifest_entry > (2) data_file > (146 Optional) >>> footer_size_in_bytes* >>> >>> *Motivation*: >>> >>> - We have several sub-second read use cases on iceberg tables. We >>> store icebergs and parquets on S3. Every hop to S3 is v.expensive (P99 of >>> >200 milliseconds). Hence we are trying to see if we can optimize by >>> cutting down any of these hops. One such hop is during the Parquet file >>> read., the first read to the parquet, which is to read the last 8 bytes - >>> to read the - footer size and par1 sequence. >>> - Iceberg metadata already includes the file_size_in_bytes. >>> Including the footer size benefits all the readers. ie., readers can >>> directly issue 1 I/O call to read the footer - >>> *read_parquet_footer(filehandle, >>> offset=file_size_in_bytes-footer_size_in_bytes-1)* >>> - This is similar to what we have in the iceberg specification in >>> the case of storing Table statistics >>> <https://iceberg.apache.org/spec/#table-statistics>, puffins > >>> *file-footer-size-in-bytes*. >>> - This can be easily extended to ORC as needed too. Perhaps, in the >>> ORC case, an additional property to store the postscript length is also >>> needed. >>> >>> Truly appreciate your thoughts, >>> Sreeram <https://www.linkedin.com/in/sreeramgarlapati> >>> >>>
