Trying to catch up on this, Serge's suggestion in the doc seems the best
way forward,
https://docs.google.com/document/d/1wjFsBdlV2YK75x7UOk2HhDOqWVA0yC7iEiqOMnNnxlA/edit?disco=AAABe5AUnWU.
Spark would support the full ANSI SQL timestamp range, and Iceberg /
Parquet/ other data source will throw ru
Seems we all agree that adding a timestamp with nanosecond precision is
necessary. And we need to store it in 10 bytes. Additionally, the spark
side will not wait for the change of Parquet as well. I will continue to
address all the comments for implementation details.
huaxin gao 于2025年3月30日周日 19
Thanks all for the discussion!
I agree that we first need to reach a consensus on adding the
TIMESTAMP(nanosecond) data type to Apache Spark. It's a standard data type
supported by major databases like Oracle and IBM DB2, making it a necessary
inclusion in Spark to align with industry practices.
>
> I think the key issue is the format. The proposed 10-byte format doesn't
> seem like a standard and the one in Iceberg/Parquet does not support the
> required range by ANSI SQL: year 0001 to year . We should address this
> issue first. Note that Parquet has an INT96 timestamp that supports
Maybe we should discuss the key issues on the dev list as it's easy to lose
track of Google Doc comments.
I think all the proposals for adding new data types need to prove that the
new data type is common/standard in the ecosystem. This means 3 things:
- it has common/standard semantic. TIMESTAMP
Thanks!!!
DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
> On Mar 27, 2025, at 3:56 PM, Qi Tan wrote:
>
> Thanks DB,
>
> I just noticed a few more comments came in after I initiated the vote. I'm
> going to postpone the voting process and address those outstanding comments.
>
>
Thanks DB,
I just noticed a few more comments came in after I initiated the vote. I'm
going to postpone the voting process and address those outstanding comments.
Qi Tan
DB Tsai 于2025年3月27日周四 15:12写道：
> Hello Qi,
>
> I'm supportive of the NanoSecond Timestamps proposal; however, before we
> in
Hello Qi,
I'm supportive of the NanoSecond Timestamps proposal; however, before we
initiate the vote, there are a few outstanding comments in the SPIP document
that haven't been addressed yet. Since the vote is on the document itself,
could we resolve these items beforehand?
For example:
The d
+1
On Thu, Mar 27, 2025 at 1:22 PM Qi Tan wrote:
> Hi all,
>
> I would like to start a vote on adding support for nanoseconds timestamps.
>
> *Discussion thread: *
> https://lists.apache.org/thread/y2vzrjl1499j5dvbpg3m81jxdhf4b6of
> *SPIP:*
> https://docs.google.com/document/d/1wjFsBdlV2YK75x7UO
Hi all,
I would like to start a vote on adding support for nanoseconds timestamps.
*Discussion thread: *
https://lists.apache.org/thread/y2vzrjl1499j5dvbpg3m81jxdhf4b6of
*SPIP:*
https://docs.google.com/document/d/1wjFsBdlV2YK75x7UOk2HhDOqWVA0yC7iEiqOMnNnxlA/edit?usp=sharing
*JIRA:* https://issue
10 matches
Mail list logo
