Re: Moving forward with the timestamp proposal

most urgent tasks are probably the ones dealing with the backward incompatible change introduced in Hive 3.1. Please let me know if you have any questions or concerns. Thanks, Zoltan On Wed, Feb 20, 2019 at 3:56 PM Zoltan Ivanfi wrote: > Hi, > > Last december we shared a times

[jira] [Created] (HIVE-21361) ORC support for TIMESTAMP WITHOUT TIME ZONE

Zoltan Ivanfi created HIVE-21361: Summary: ORC support for TIMESTAMP WITHOUT TIME ZONE Key: HIVE-21361 URL: https://issues.apache.org/jira/browse/HIVE-21361 Project: Hive Issue Type: Task

[jira] [Created] (HIVE-21360) Avro support for TIMESTAMP WITHOUT TIME ZONE

Zoltan Ivanfi created HIVE-21360: Summary: Avro support for TIMESTAMP WITHOUT TIME ZONE Key: HIVE-21360 URL: https://issues.apache.org/jira/browse/HIVE-21360 Project: Hive Issue Type: Task

[jira] [Created] (HIVE-21359) Parquet support for TIMESTAMP WITHOUT TIME ZONE

Zoltan Ivanfi created HIVE-21359: Summary: Parquet support for TIMESTAMP WITHOUT TIME ZONE Key: HIVE-21359 URL: https://issues.apache.org/jira/browse/HIVE-21359 Project: Hive Issue Type

[jira] [Created] (HIVE-21358) ORC support for TIMESTAMP WITH LOCAL TIME ZONE

Zoltan Ivanfi created HIVE-21358: Summary: ORC support for TIMESTAMP WITH LOCAL TIME ZONE Key: HIVE-21358 URL: https://issues.apache.org/jira/browse/HIVE-21358 Project: Hive Issue Type: Task

[jira] [Created] (HIVE-21353) Use Instant instead of ZonedDateTime as the internal type for TIMESTAMP WITH LOCAL TIME ZONE

Zoltan Ivanfi created HIVE-21353: Summary: Use Instant instead of ZonedDateTime as the internal type for TIMESTAMP WITH LOCAL TIME ZONE Key: HIVE-21353 URL: https://issues.apache.org/jira/browse/HIVE-21353

[jira] [Created] (HIVE-21357) Avro support for TIMESTAMP WITH LOCAL TIME ZONE

Zoltan Ivanfi created HIVE-21357: Summary: Avro support for TIMESTAMP WITH LOCAL TIME ZONE Key: HIVE-21357 URL: https://issues.apache.org/jira/browse/HIVE-21357 Project: Hive Issue Type

[jira] [Created] (HIVE-21355) Parquet support for TIMESTAMP WITH LOCAL TIME ZONE

Zoltan Ivanfi created HIVE-21355: Summary: Parquet support for TIMESTAMP WITH LOCAL TIME ZONE Key: HIVE-21355 URL: https://issues.apache.org/jira/browse/HIVE-21355 Project: Hive Issue Type

[jira] [Created] (HIVE-21351) TIMESTAMP WITH TIME ZONE

Zoltan Ivanfi created HIVE-21351: Summary: TIMESTAMP WITH TIME ZONE Key: HIVE-21351 URL: https://issues.apache.org/jira/browse/HIVE-21351 Project: Hive Issue Type: Task Reporter

[jira] [Created] (HIVE-21350) TIMESTAMP WITH LOCAL TIME ZONE

Zoltan Ivanfi created HIVE-21350: Summary: TIMESTAMP WITH LOCAL TIME ZONE Key: HIVE-21350 URL: https://issues.apache.org/jira/browse/HIVE-21350 Project: Hive Issue Type: Task

[jira] [Created] (HIVE-21349) TIMESTAMP WITHOUT TIME ZONE

Zoltan Ivanfi created HIVE-21349: Summary: TIMESTAMP WITHOUT TIME ZONE Key: HIVE-21349 URL: https://issues.apache.org/jira/browse/HIVE-21349 Project: Hive Issue Type: Task

[jira] [Created] (HIVE-21348) Execute the TIMESTAMP types roadmap

Zoltan Ivanfi created HIVE-21348: Summary: Execute the TIMESTAMP types roadmap Key: HIVE-21348 URL: https://issues.apache.org/jira/browse/HIVE-21348 Project: Hive Issue Type: Task

Re: Moving forward with the timestamp proposal

in Parquet/Orc > first? Or are we going to use low-level physical types directly and add > Spark-specific metadata to Parquet/Orc files? > > On Wed, Feb 20, 2019 at 10:57 PM Zoltan Ivanfi > wrote: > > > Hi, > > > > Last december we shared a timestamp harmon

Moving forward with the timestamp proposal

Hi, Last december we shared a timestamp harmonization proposal with the Hive, Spark and Impala communities. This was followed by an extensive discussion in January that lead to various updates and improvements to the proposal, as well as the creation of a new document for f

Requesting write access to the Hive wiki

Dear Hive Developers, I would like to create a wiki page for the timestamp plan , could someone please grant me write access for doing so? According to this wiki page , I have to request access on the mailing li

[jira] [Created] (HIVE-21291) Restore historical way of handling timestamps in Avro while keeping the new semantics at the same time

Zoltan Ivanfi created HIVE-21291: Summary: Restore historical way of handling timestamps in Avro while keeping the new semantics at the same time Key: HIVE-21291 URL: https://issues.apache.org/jira/browse/HIVE

[jira] [Created] (HIVE-21290) Restore historical way of handling timestamps in Parquet while keeping the new semantics at the same time

Zoltan Ivanfi created HIVE-21290: Summary: Restore historical way of handling timestamps in Parquet while keeping the new semantics at the same time Key: HIVE-21290 URL: https://issues.apache.org/jira/browse/HIVE

Re: [DISCUSS] Consistent Timestamps across Hadoop

: Owen O'Malley and Jesús Camacho Rodríguez from Hive, Anna Szonyi and Zoltan Ivanfi representing the original proposal. Owen and Jesús reasoned that the TIMESTAMP type must have the same semantics in all file formats in Hive. Anna and Zoltan reasoned that different Hive versions (and other compone

Adding more timestamp types to on-disk storage formats

Hi, One of the feedbacks I got for the SQL timestamp type harmonization proposal was that I should reach out the file format communities as well. For this purpose I created a separate document from their perspective and sent it to the Avro, ORC, Parquet, Arrow, Kudu and Iceberg developer lists. Pl

Re: [DISCUSS] Consistent Timestamps across Hadoop

From: Owen O'Malley > Date: Thu, Jan 10, 2019 at 3:09 PM > Subject: Re: [DISCUSS] Consistent Timestamps across Hadoop > To: Zoltan Ivanfi > > > No, that isn't right. > > The discussion for Apache projects needs to happen in the open and not the > private google

[jira] [Created] (HIVE-21117) A day may belong to a different year than the week it is a part of

Zoltan Ivanfi created HIVE-21117: Summary: A day may belong to a different year than the week it is a part of Key: HIVE-21117 URL: https://issues.apache.org/jira/browse/HIVE-21117 Project: Hive

Re: [DISCUSS] Consistent Timestamps across Hadoop

; > From an Apache point of view, we really need to move this document and the > discussion to the Apache wiki and mailing lists. > > Did you want to take a first pass at moving it to Hive's wiki? > > .. Owen > > On Tue, Dec 11, 2018 at 10:40 AM Zoltan Ivanfi wrote:

Updated proposal: Consistent timestamp types in Hadoop SQL engines

Dear All, I would like to thank every reviewer of the consistent timestamps proposal[1] for their time and valuable comments. Based on your feedback, I have updated the proposal. The changes include clarifications, fixes and other improvements as summarized at the end of the document, in the Chang

Re: [DISCUSS] Consistent Timestamps across Hadoop

Hi Owen, Thanks, I think your email contains a great summary of the problems tackled in the proposal. I would like highlight two particular topics from the discussion that we are having in the comments (details can be read in the document

[jira] [Created] (HIVE-21002) Backwards incompatible change: Hive 3.1 reads back Avro and Parquet timestamps written by Hive 2.x incorrectly

Zoltan Ivanfi created HIVE-21002: Summary: Backwards incompatible change: Hive 3.1 reads back Avro and Parquet timestamps written by Hive 2.x incorrectly Key: HIVE-21002 URL: https://issues.apache.org/jira/browse

[jira] [Created] (HIVE-20374) Write Hive version information to Parquet footer

Zoltan Ivanfi created HIVE-20374: Summary: Write Hive version information to Parquet footer Key: HIVE-20374 URL: https://issues.apache.org/jira/browse/HIVE-20374 Project: Hive Issue Type

[jira] [Created] (HIVE-17844) Hive JDBC driver converts floats to doubles, changing values slightly

Zoltan Ivanfi created HIVE-17844: Summary: Hive JDBC driver converts floats to doubles, changing values slightly Key: HIVE-17844 URL: https://issues.apache.org/jira/browse/HIVE-17844 Project: Hive

[jira] [Created] (HIVE-17843) UINT32 Parquet columns are handled as signed INT32-s, silently reading incorrect data

Zoltan Ivanfi created HIVE-17843: Summary: UINT32 Parquet columns are handled as signed INT32-s, silently reading incorrect data Key: HIVE-17843 URL: https://issues.apache.org/jira/browse/HIVE-17843

[jira] [Created] (HIVE-17618) Extend ANALYZE TABLE / DESCRIBE FORMATTED functionality with distribution of selected file-level metadata fields

Zoltan Ivanfi created HIVE-17618: Summary: Extend ANALYZE TABLE / DESCRIBE FORMATTED functionality with distribution of selected file-level metadata fields Key: HIVE-17618 URL: https://issues.apache.org/jira

Re: Review Request 58501: HIVE-16469: Parquet timestamp table property is not always taken into account

/MapredParquetInputFormat.java Lines 86 (patched) <https://reviews.apache.org/r/58501/#comment245296> Nit: You should put a space before "because" as well, both here and 4 lines below. - Zoltan Ivanfi On April 18, 2017, 1:14 p.m., Barna Zso

Re: Review Request 56334: HIVE-12767: Implement table property to address Parquet int96 timestamp bug

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56334/#review165334 --- Ship it! Thanks! - Zoltan Ivanfi On Feb. 13, 2017, 3:21 p.m

Re: Review Request 56334: HIVE-12767: Implement table property to address Parquet int96 timestamp bug

n that it should be added as a comment. - Zoltan Ivanfi On Feb. 13, 2017, 1:59 p.m., Barna Zsombor Klara wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https:

Re: Review Request 56334: HIVE-12767: Implement table property to address Parquet int96 timestamp bug

that's a different thing.) ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java (line 43) <https://reviews.apache.org/r/56334/#comment236948> getUTCCalendar should always return a UTC calendar, shouldn't it? - Zoltan Ivanfi On Feb. 10, 2017, 1:

Re: Review Request 56334: HIVE-12767: Implement table property to address Parquet int96 timestamp bug

f no timezone is specified. If a timezone is specified but not available, that must trigger an error. ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedPrimitiveColumnReader.java (line 419) <https://reviews.apache.org/r/56334/#comment236805> ... if no timezon

[jira] [Created] (HIVE-14846) Char encoding does not apply to newline chars

Zoltan Ivanfi created HIVE-14846: Summary: Char encoding does not apply to newline chars Key: HIVE-14846 URL: https://issues.apache.org/jira/browse/HIVE-14846 Project: Hive Issue Type: Bug