Re: Need clarification on max number of columns a Iceberg table can have

2024-10-31 Thread Pucheng Yang
Thank you very much! I will try to document this on the website. On Thu, Oct 31, 2024 at 6:06 PM Szehon Ho wrote: > Yes, that is correct! > > Thanks > Szehon > > On Thu, Oct 31, 2024 at 5:58 PM Pucheng Yang > wrote: > >> Hi Szehon, >> >> Thanks for getting back to me so quickly! >> >> No, I don

Re: Need clarification on max number of columns a Iceberg table can have

2024-10-31 Thread Szehon Ho
Yes, that is correct! Thanks Szehon On Thu, Oct 31, 2024 at 5:58 PM Pucheng Yang wrote: > Hi Szehon, > > Thanks for getting back to me so quickly! > > No, I don't see anywhere that is failing. My question is more of a general > question after browsing all the issues. So from what you said, it s

Re: Need clarification on max number of columns a Iceberg table can have

2024-10-31 Thread Pucheng Yang
Hi Szehon, Thanks for getting back to me so quickly! No, I don't see anywhere that is failing. My question is more of a general question after browsing all the issues. So from what you said, it seems Iceberg in theory can support a very large number of columns (say 100K) w/o hitting any hard limi

Re: Need clarification on max number of columns a Iceberg table can have

2024-10-31 Thread Szehon Ho
Hi Pucheng There were some parts in the implementation where column field ids collided with partition field ids. https://github.com/apache/iceberg/pull/10020 introduced mechanisms for affected code to get unique ids, and known places have been fixed. (Particularly the Spark procedure rewrite_posit

Need clarification on max number of columns a Iceberg table can have

2024-10-31 Thread Pucheng Yang
Hey community, I was following https://github.com/apache/iceberg/issues/9220 (Max number of columns) and down the rabbit hole and I found there are a lot of discussions about issues with tables having more than 1k columns. However, after reviewing discussions, it is still a little confusing to me