Re: Re: [Discuss] SPIP: Support NanoSecond Timestamps

Hello team, I have already updated the google doc: https://docs.google.com/document/d/1wjFsBdlV2YK75x7UOk2HhDOqWVA0yC7iEiqOMnNnxlA/edit?usp=sharing. If all looks good, I will raise a vote later this week. Thank you! Qi Tan 于2025年3月18日周二 21:39写道： > Hello Reynold, I truly appreciate your time and

RE: Re: [Discuss] SPIP: Support NanoSecond Timestamps

Hello Reynold, I truly appreciate your time and attention to this feature. For the performance, here are my thoughts: * As Serge mentioned above, Apache Spark needs to be aligned with other competitive products. We should not overlook potential benefits just because of performance regression.

RE: Re: [Discuss] SPIP: Support NanoSecond Timestamps

Thank you all for your comments and suggestions! Let me rewrite the document. On 2025/03/17 22:28:35 Gengliang Wang wrote: > Hi Qi, > > Thanks for the proposal. I am generally +1 with the idea. Could you clarify > which option is preferred in “Q1. What are you trying to do?”? > Understanding this

Re: [Discuss] SPIP: Support NanoSecond Timestamps

IMHO that’s not a good comparison. By that logic we shouldn’t have double because it’s slower than int. We should compare against the competition first. Maybe as part of this effort we’ll need to prototype two competing solutions. The vast majority of differences should be related to storage cos

Re: [Discuss] SPIP: Support NanoSecond Timestamps

Hi Qi, Thanks for the proposal. I am generally +1 with the idea. Could you clarify which option is preferred in “Q1. What are you trying to do?”? Understanding this will help us align our discussion. On Mon, Mar 17, 2025 at 3:05 PM Reynold Xin wrote: > Pretty much anything (say vs current tim

Re: [Discuss] SPIP: Support NanoSecond Timestamps

Pretty much anything (say vs current timestamp operations in Spark). On Mon, Mar 17, 2025 at 2:51 PM serge rielau.com wrote: > What are you comparing performance against? > On Mar 17, 2025 at 11:54 AM -0700, Reynold Xin , > wrote: > > Any thoughts on how to deal with performance here? Initially

Re: [Discuss] SPIP: Support NanoSecond Timestamps

What are you comparing performance against? On Mar 17, 2025 at 11:54 AM -0700, Reynold Xin , wrote: Any thoughts on how to deal with performance here? Initially we didn't do nano level precision because of performance (would not be able to fit everything into a 64 bit int). On Mon, Mar 17, 2025

Re: [Discuss] SPIP: Support NanoSecond Timestamps

Any thoughts on how to deal with performance here? Initially we didn't do nano level precision because of performance (would not be able to fit everything into a 64 bit int). On Mon, Mar 17, 2025 at 11:34 AM Sakthi wrote: > +1 (non-binding) > > On Mon, Mar 17, 2025 at 11:32 AM Zhou Jiang > wrot

Re: [Discuss] SPIP: Support NanoSecond Timestamps

+1 (non-binding) On Mon, Mar 17, 2025 at 11:32 AM Zhou Jiang wrote: > +1 for the nanosecond support > > > > On Mar 16, 2025, at 16:03, Dongjoon Hyun wrote: > > > > +1 for supporting NanoSecond Timestamps. > > > > Thank you, Qi. > > > > Dongjoon. > > > >

Re: [Discuss] SPIP: Support NanoSecond Timestamps

+1 for the nanosecond support > On Mar 16, 2025, at 16:03, Dongjoon Hyun wrote: > > +1 for supporting NanoSecond Timestamps. > > Thank you, Qi. > > Dongjoon. > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [Discuss] SPIP: Support NanoSecond Timestamps

+1 for supporting NanoSecond Timestamps. Thank you, Qi. Dongjoon. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [Discuss] SPIP: Support NanoSecond Timestamps

+1Sent from my iPhoneOn Mar 14, 2025, at 11:28 PM, Szehon Ho wrote:+1 to the idea as well, as Iceberg V3 is coming with time with nanos, and Spark would not be able to read this type without this.ThanksSzehonOn Fri, Mar 14, 2025 at 3:34 PM Wenchen Fan wrote:In general, I thi

Re: Re: [Discuss] SPIP: Support NanoSecond Timestamps

In general, I think it's good for Spark to support the common data types in the ecosystem, as it's the only way to fully integrate with the ecosystem. So +1. On Fri, Mar 14, 2025 at 8:56 AM 谭琦 wrote: > Updated. Thanks. > > On 2025/03/13 23:56:20 Jungtaek Lim wrote: > > Hi, would you mind allowin

RE: Re: [Discuss] SPIP: Support NanoSecond Timestamps

Updated. Thanks. On 2025/03/13 23:56:20 Jungtaek Lim wrote: > Hi, would you mind allowing comments on the doc? Thanks! > > On Fri, Mar 14, 2025 at 8:50 AM Qi Tan wrote: > > > Hello everybody, > > > > I would like to start a discussion on SPARK-50532 > >

Re: Re: [Discuss] SPIP: Support NanoSecond Timestamps

+1 to the idea as well, as Iceberg V3 is coming with time with nanos, and Spark would not be able to read this type without this. Thanks Szehon On Fri, Mar 14, 2025 at 3:34 PM Wenchen Fan wrote: > In general, I think it's good for Spark to support the common data types > in the ecosystem, as it

Re: [Discuss] SPIP: Support NanoSecond Timestamps

Will this nanosecond timestamp be a fixed-size (10 bytes) binary in UnsafeRow and ColumnVector? On Thu, Mar 13, 2025 at 4:57 PM Jungtaek Lim wrote: > Hi, would you mind allowing comments on the doc? Thanks! > > On Fri, Mar 14, 2025 at 8:50 AM Qi Tan wrote: > >> Hello everybody, >> >> I would li

Re: [Discuss] SPIP: Support NanoSecond Timestamps

Hi, would you mind allowing comments on the doc? Thanks! On Fri, Mar 14, 2025 at 8:50 AM Qi Tan wrote: > Hello everybody, > > I would like to start a discussion on SPARK-50532 > to enable Spark to > support nanoseconds. Here attached the spip d

[Discuss] SPIP: Support NanoSecond Timestamps

Hello everybody, I would like to start a discussion on SPARK-50532 to enable Spark to support nanoseconds. Here attached the spip doc . Huaxin was