Hi, Once we are through the discussion phase and hopefully have reached agreement, I support moving the document to a more permanent place. I'm unsure about what the best place would be though. Since it needs the agreement of three communities, it does not strictly belong to any single one of them (although the Hive Metastore is certainly a central component in this ecosystem, so we could put it in the Hive documentation based on that). I am also uncertain about whether we should use a wiki page, because it is too easily editable and after reaching an agreement it should not be modified without asking or notifying the same communities again.
Is there a documentation space where modifications are subject to review? Or can we protect a wiki page to achieve that? I'm open to your suggestions. Thanks, Zoltan On Wed, Jan 9, 2019 at 9:32 PM Owen O'Malley <owen.omal...@gmail.com> wrote: > > From an Apache point of view, we really need to move this document and the > discussion to the Apache wiki and mailing lists. > > Did you want to take a first pass at moving it to Hive's wiki? > > .. Owen > > On Tue, Dec 11, 2018 at 10:40 AM Zoltan Ivanfi <z...@cloudera.com> wrote: >> >> Hi Owen, >> >> Thanks, I think your email contains a great summary of the problems tackled >> in the proposal. I would like highlight two particular topics from the >> discussion that we are having in the comments (details can be read in the >> document): >> >> It seems that we have agreement on the desired semantics of the more >> explicit SQL types. In particular, I was glad to hear that the TIMESTAMP >> WITH LOCAL TIME ZONE type that is already implemented in Hive is supposed to >> have Instant semantics. (In fact, it already does have Instant semantics, >> but it also has additional time zone information that is unused at this >> moment, and I wasn't sure whether that will be utilized, changing the >> semantics, or whether the semantics will remain and the superflous time zone >> data will be removed.) >> We are still discussing what is the best course of action to take with the >> plain TIMESTAMP type, which behaved differently in different file formats in >> Hive 2 and was made to behave the same way in a compatibility-breaking >> manner in Hive 3. My take on this type is that it has already been used to >> write huge amounts of data and for this reason we should restore its Avro- >> and Parquet-specific incosistent behaviour (possibly controlled by a feature >> flag), so that legacy data remains readable and legacy workarounds remain >> functional. The new, more explicit SQL types will provide a clear migration >> path away from the messy TIMESTAMP type. >> >> All in all, I feel that we are converging towards a common goal and I have >> high hopes that the more explicit timestamp types will have much better >> interoperability and consistency across different Hadoop SQL engines. >> >> Thanks, >> >> Zoltan >> >> >> On Mon, Dec 10, 2018 at 7:54 PM Owen O'Malley <owen.omal...@gmail.com> wrote: >>> >>> Thank you for starting this discussion. Clearly the Hive semantics on >>> timestamp are very messed up, but has been moving in the right direction of >>> becoming more SQL standard compliant. I'm pulling this discussion back to >>> the list rather than the personal GoogleDoc, which isn't very collaborative. >>> >>> I like your breakdown of the semantics: >>> >>> Instant - point in time that will appear different depending on the reader >>> time zone >>> LocalDateTime - consistent hour and minute regardless of the reader time >>> zone. >>> OffsetDateTime - consistent hour and minute with the offset of the writer >>> time zone >>> >>> The SQL standard has: >>> >>> Timestamp & Timestamp without time zone = LocalDateTime >>> Timestamp with time zone = OffsetDateTime >>> >>> Hive 2 had very confused semantics for timestamp: >>> >>> When storage was ORC, text, or RCFile with a text serde it was LocalDateTime >>> When storage was Avro, Parquet, or RCFile with a binary serde it was Instant >>> >>> Hive 3.1 has moved toward the SQL standard extended with Oracles' timestamp >>> with local time zone: >>> >>> Timestamp = LocalDateTime >>> Timestamp with local time zone = Instant >>> >>> This leaves us with a few problems: >>> >>> The Hive bindings to Parquet and Avro don't handle timestamps correctly. >>> ORC doesn't support timestamps with local time zone. I start working on it >>> in ORC-189. >>> We don't have timestamp with time zone support. >>> >>> .. Owen >>> >>> On Thu, Dec 6, 2018 at 7:55 AM Marta Kuczora >>> <kuczo...@cloudera.com.invalid> wrote: >>>> >>>> Hi Hive Community, >>>> >>>> I would like to share the following document on our "Consistent Timestamp >>>> types in Hadoop" plans for review. >>>> https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit >>>> >>>> With this plan we would like to get an agreement on consistent timestamp >>>> behavior on Hive, Spark and Impala and in order to achieve this, we are >>>> sharing this document with all three communities. >>>> >>>> Please review and comment, any feedback is much appreciated! >>>> >>>> Regards, >>>> Marta
