Thank you, understood.This is really neat.
I think I had trouble grasping this because systems are not normally so
flexible!
On Thu, 10 Oct 2019 at 17:36, Ryan Blue wrote:
> The old data would continue to exist in daily partitions, new data would
> be written into hourly partitions.
>
> Icebe
The old data would continue to exist in daily partitions, new data would be
written into hourly partitions.
Iceberg as a format doesn't require you to do anything here. You can go
back and rewrite the data that you think is worth moving to hourly, or you
can leave it as it is. We want actions like
Thank you. I now appreciate the significant benefit of the decoupling the
query from the partition scheme.
In terms of physical layout, what would an evolution of partitioning from
daily to hourly look like? Would one need rewrite the whole table to
achieve smaller groupings in files, or does Iceb
> It is not clear to me how partition keys are distributed with respect to
actual files and what constraints exist for partition evolution.
The requirement is that a file contains rows that have the same values for
all partition columns. If you partition by log_level and date(ts), then for
any giv
‘If would’ → ‘it would’
‘original schema’ → ‘original scheme’
On Tue, 8 Oct 2019 at 18:00, Elliot West wrote:
> Hello,
>
> I'm trying to understand the underlying partitioning model in Iceberg. It
> is not clear to me how partition keys are distributed with respect to
> actual files and what con
Hello,
I'm trying to understand the underlying partitioning model in Iceberg. It
is not clear to me how partition keys are distributed with respect to
actual files and what constraints exist for partition evolution. My
expectation is that to achieve reasonable read performance, sets of keys
must b
