Re: Time-based partitioning on long column type

Bumping this thread again. I received this SQL from a user today. CREATE TABLE partition_by_epoch_timestamp ( > `id` STRING, > `data` DECIMAL(10,0), > `start_ts` DECIMAL(10,0), > `end_ts` DECIMAL(10,0) > ) > USING iceberg > PARTITIONED BY ( > days(from_unixtime(start_ts)) > ) Of c

Re: Time-based partitioning on long column type

I was thinking about adding to the specification how the current date/time transforms work with source columns that are longs. I don't think it helps much to introduce a new transform, although that would make the interpretation of the source data more obvious to users applying the transform. On W

Re: Time-based partitioning on long column type

> > Maybe we could update the time-based partition functions to be applied to > a long column directly. It would treat that column like a timestamp in > milliseconds. Would that work? I need to think more about the implications > of doing that, but I don't think that we currently have an issue with

Re: Time-based partitioning on long column type

Maybe we could update the time-based partition functions to be applied to a long column directly. It would treat that column like a timestamp in milliseconds. Would that work? I need to think more about the implications of doing that, but I don't think that we currently have an issue with extending

Re: Time-based partitioning on long column type

Ah yes, milestone is fine. Thanks ! All good. Regards JB On Thu, Apr 4, 2024 at 5:12 PM Eduard Tudenhoefner wrote: > > There is the V3 Spec milestone where it's tracked (amongst other things). > > On Thu, Apr 4, 2024 at 9:44 AM Jean-Baptiste Onofré wrote: >> >> Hi Eduard, >> >> Thanks for the

Re: Time-based partitioning on long column type

There is the V3 Spec milestone where it's tracked (amongst other things). On Thu, Apr 4, 2024 at 9:44 AM Jean-Baptiste Onofré wrote: > Hi Eduard, > > Thanks for the update ! It makes sense to me. > > Maybe a GH label with spec or v3_spec would hel

Re: Time-based partitioning on long column type

Hi Eduard, Thanks for the update ! It makes sense to me. Maybe a GH label with spec or v3_spec would help to see what is planned for v3 ? Regards JB On Thu, Apr 4, 2024 at 9:36 AM Eduard Tudenhoefner wrote: > > Type promotion from Long to Timestamp is on the roadmap for the V3 Spec, so > that

Re: Time-based partitioning on long column type

Type promotion from Long to Timestamp is on the roadmap for the V3 Spec, so that would be the preferred way. On Wed, Apr 3, 2024 at 10:38 AM Jean-Baptiste Onofré wrote: > Hi Manu > > TIMESTAMP_LONG type promotion could be the easiest way, it would

Re: Time-based partitioning on long column type

Hi Manu TIMESTAMP_LONG type promotion could be the easiest way, it would work with the existing transform. Would it work for you? Regards JB On Wed, Apr 3, 2024 at 5:56 AM Manu Zhang wrote: > > Hi all, > > We have source data with a timestamp field in LONG type to land in an Iceberg > table.

Time-based partitioning on long column type

Hi all, We have source data with a timestamp field in LONG type to land in an Iceberg table. We want to partition the table with the timestamp field such that we can query recent data more efficiently. However, LONG is not supported as the source type of time-based transform (hour, day, etc) I fi