Hey Sreeram, Sounds like there's a fair amount of interest/support for this. Anton also mentioned that having this information would help estimate orphaned DVs, so there's multiple cases where this would be beneficial.
We might want to tie this change to a format version release (even if just an optional field) because any metadata rewrites may result in dropping the value. Did you want to put together a proposal for the changes? Best, -Dan On Sat, Feb 8, 2025 at 11:31 AM Micah Kornfield <emkornfi...@gmail.com> wrote: > +1 I think this is probably useful for wider schemas that typically have > larger footers that go past the heuristic. > > It would be good to have some concrete numbers on how much this impacts > workloads before committing to it. > > Cheers, > Micah > > On Thu, Jan 30, 2025 at 7:10 AM Steve Loughran <ste...@cloudera.com.invalid> > wrote: > >> >> Knowing the footer offset would be really useful if passed down to >> whatever is implementing the input stream, along with the actual file size. >> >> This can be used for prefetching the footer, as well as caching it (Azure >> ABFS, google GCS connectors): right now they guess that about 1MB is all >> they need. >> >> while readTail() can get bytes off the end, it doesn't pass that >> information down to the stream, to do its own thing. >> >> The Analytics stream which the AWS S3 team are getting into the s3a code ( >> https://issues.apache.org/jira/browse/HADOOP-19363) goes one step >> further than the others: it parses that footer itself and tries to predict >> where application code is going to read next: as you read one rowgroup it >> speculatively fetch the next one, even as the first one is downloaded. >> >> Again, it guesses on footer size: pass that in and they will know what to >> fetch and store. Ideally this should be accompanied by file type (parquet, >> avro) and your actual read plans (vectored, random, sequential, >> whole-file). With this information you can cut out a number of >> wasted/inefficient S3 calls, and tune fetching/caching policy appropriately. >> >> Anyway: >> +1 to footer length, and if already known, file length should come down >> too, along with that read plan. saying "parque, vectored, randomt" will be >> enough, which is what a draft PR i have for hadoop fileIO does. >> >> On Wed, 22 Jan 2025 at 03:39, Sreeram Garlapati <gsreeramku...@gmail.com> >> wrote: >> >>> Thanks for the nice idea/suggestion, Dan. >>> Yes, we have been employing a similar technique that you noted below and >>> kinda arrived at the conclusion that there is no deterministic way to >>> achieve that most optimal situation, ie., single i/o call to S3 to read the >>> parquet footer. >>> >>> Best, >>> Sreeram >>> >>> On Tue, Jan 21, 2025 at 4:20 PM Daniel Weeks <dwe...@apache.org> wrote: >>> >>>> Hey Sreeram, >>>> >>>> I think it's worthwhile to consider what value would be added by >>>> tracking the footer size in metadata, but there are other options to >>>> address these optimization use cases. >>>> >>>> For example, if you take a look at the RangeReadable >>>> <https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/io/RangeReadable.java#L68> >>>> interface >>>> for FileIO implementations, there's a readTail method so that you can >>>> optimistically read from the tail end of the file to try to fetch the full >>>> footer in a single read. This is even optimized in some of the >>>> implementations (like S3InputStream) to leverage backward reads as opposed >>>> to seek operations which might have overhead. >>>> >>>> Depending on the size of the file, you may want to load just the tail >>>> or the whole file to avoid all reads. Having the exact value >>>> definitely will make this more exact, but I feel like using the above >>>> approach can approximate the same performance benefits. >>>> >>>> Just a thought, >>>> -Dan >>>> >>>> On Tue, Jan 21, 2025 at 12:17 PM Sreeram Garlapati < >>>> gsreeramku...@gmail.com> wrote: >>>> >>>>> Hello Team! >>>>> >>>>> This is a small improvement proposal to store the *parquet footer >>>>> size* as part of the *data_file* metadata in the iceberg manifest >>>>> <https://iceberg.apache.org/spec/#manifests>. >>>>> *manifest_entry > (2) data_file > (146 Optional) >>>>> footer_size_in_bytes* >>>>> >>>>> *Motivation*: >>>>> >>>>> - We have several sub-second read use cases on iceberg tables. We >>>>> store icebergs and parquets on S3. Every hop to S3 is v.expensive (P99 >>>>> of >>>>> >200 milliseconds). Hence we are trying to see if we can optimize by >>>>> cutting down any of these hops. One such hop is during the Parquet file >>>>> read., the first read to the parquet, which is to read the last 8 >>>>> bytes - >>>>> to read the - footer size and par1 sequence. >>>>> - Iceberg metadata already includes the file_size_in_bytes. >>>>> Including the footer size benefits all the readers. ie., readers can >>>>> directly issue 1 I/O call to read the footer - >>>>> *read_parquet_footer(filehandle, >>>>> offset=file_size_in_bytes-footer_size_in_bytes-1)* >>>>> - This is similar to what we have in the iceberg specification in >>>>> the case of storing Table statistics >>>>> <https://iceberg.apache.org/spec/#table-statistics>, puffins > >>>>> *file-footer-size-in-bytes*. >>>>> - This can be easily extended to ORC as needed too. Perhaps, in >>>>> the ORC case, an additional property to store the postscript length is >>>>> also >>>>> needed. >>>>> >>>>> Truly appreciate your thoughts, >>>>> Sreeram <https://www.linkedin.com/in/sreeramgarlapati> >>>>> >>>>>
