Re: AvroFileAppender metrics

2020-03-13 Thread Ryan Blue
Yeah, I would probably ignore the column size metric. That's really more for columnar formats, where we could use it to estimate how much data from a row group is being projected. For Avro, we'd have to read the same amount either way. For this, I'd probably create an appender that wraps another a

Re: AvroFileAppender metrics

2020-03-13 Thread Luis Otero
Feedback/guidance request: Byte size info in avro is encapsulated in encoder (org.apache.avro.io.BufferedBinaryEncoder) and is not exposed by avro api. Should we carry on with the task ignoring that metric (gathering as much info as we can inside Iceberg)? Is it feasible to get Avro modified (to