Re: truncate transform over binary columns

2024-04-04 Thread Brian Hulette
> > Quick question: do you actually have an issue with truncate on binary > columns ? No issue - as a consumer of Iceberg metadata I'd just like to clarify if we should expect to see partition fields with truncated binary. I was initially coding against the spec and planned to reject a truncate

Re: truncate transform over binary columns

2024-04-04 Thread Jean-Baptiste Onofré
Hi Brian, Welcome to this list :) Quick question: do you actually have an issue with truncate on binary columns ? The Truncate transform (in Iceberg API) supports BINARY using TruncateByteBuffer, so it would make sense to clearly state this in the spec. Regards JB On Thu, Apr 4, 2024 at 2:45 AM

Re: truncate transform over binary columns

2024-04-03 Thread Amogh Jahagirdar
Another aspect that just occurred to me, there could be concern on just directly updating the spec as is since it's ultimately the source of truth (not the reference implementation). If there's concern that such a change would make other implementations technically non-compliant because they don't

Re: truncate transform over binary columns

2024-04-03 Thread Amogh Jahagirdar
Thanks for bringing this up Brian! In my view, the decision tree for this would look something like: 1.) Is there anything incorrect with supporting truncate on the basis of width for binary columns? I can't really think of any reason, it seems legitimate to me (handling characters outside of utf-