> On May 7, 2017, 11:22 p.m., Xuefu Zhang wrote: > > common/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java > > Lines 138 (patched) > > <https://reviews.apache.org/r/50787/diff/7/?file=1706764#file1706764line138> > > > > Not sure if I understand this, but why cannot we get seconds/nanos from > > date/timestamp and then convert to TimestapTZ? I assume this is a faster > > way.
Hi Xuefu, the reason why I did this: 1. As Ashutosh suggested, we will use LocalDate and LocalDateTime for Date and Timestamp in the future. When that happens, date/timestamp won't have seconds/nanos part, instead they're only descriptions of time. So the conversion should be done based on text format. 2. At the moment, the seconds/nanos of date/timestamp is computed using system timezone. So the conversion can have different results in different systems. I noted Carter also suggested that SQL standard requires session zone should be taken into consideration in the conversion. Consolidating your suggestions with Carter's, I think we can: make the conversion text-wise, and append the system zone (Hive currently doesn't have session zone). For example, a date of '2017-01-01' in LA will be converted to timestamptz as '2017-01-01 00:00:00 America/Los_Angeles', which in turn converted to '2017-01-01 08:00:00.0 Z'. Does this make sense? > On May 7, 2017, 11:22 p.m., Xuefu Zhang wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java > > Lines 204 (patched) > > <https://reviews.apache.org/r/50787/diff/7/?file=1706785#file1706785line204> > > > > What does this imply? The method converts our primitive type to a SqlTypeName in calcite. But SqlTypeName currently doesn't have timestamp with time zone. This will have some impact when calcite does the optimization, e.g. computing average value sizes. But I think we have to live with it untile SqlTypeName supports timestamp with time zone. > On May 7, 2017, 11:22 p.m., Xuefu Zhang wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestampTZ.java > > Lines 32 (patched) > > <https://reviews.apache.org/r/50787/diff/7/?file=1706827#file1706827line32> > > > > Can you also make a note about the source of the code, like > > TimeStampTZWritable? sure, will do - Rui ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50787/#review174136 ----------------------------------------------------------- On May 8, 2017, 6:51 a.m., Rui Li wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50787/ > ----------------------------------------------------------- > > (Updated May 8, 2017, 6:51 a.m.) > > > Review request for hive, pengcheng xiong and Xuefu Zhang. > > > Bugs: HIVE-14412 > https://issues.apache.org/jira/browse/HIVE-14412 > > > Repository: hive-git > > > Description > ------- > > The 1st patch to add timezone-aware timestamp. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java > PRE-CREATION > common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java > PRE-CREATION > contrib/src/test/queries/clientnegative/serde_regex.q a676338 > contrib/src/test/queries/clientpositive/serde_regex.q d75d607 > contrib/src/test/results/clientnegative/serde_regex.q.out 58b1c02 > contrib/src/test/results/clientpositive/serde_regex.q.out 2984293 > hbase-handler/src/test/queries/positive/hbase_timestamp.q 0350afe > hbase-handler/src/test/results/positive/hbase_timestamp.q.out 3918121 > itests/hive-blobstore/src/test/queries/clientpositive/orc_format_part.q > 358eccd > > itests/hive-blobstore/src/test/queries/clientpositive/orc_nonstd_partitions_loc.q > c462538 > itests/hive-blobstore/src/test/queries/clientpositive/rcfile_format_part.q > c563d3a > > itests/hive-blobstore/src/test/queries/clientpositive/rcfile_nonstd_partitions_loc.q > d17c281 > itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out > 5d1319f > > itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out > 70e72f7 > > itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out > bed10ab > > itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out > c6442f9 > jdbc/src/java/org/apache/hive/jdbc/HiveBaseResultSet.java ade1900 > jdbc/src/java/org/apache/hive/jdbc/JdbcColumn.java 38918f0 > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 8dc5f2e > ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java f8b55da > ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java > 01a652d > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java > 38308c9 > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 0cf9205 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 0721b92 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8598fae > ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java > 8f8eab0 > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java bda2050 > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 7cdf2c3 > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 5cacd59 > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 68d98f5 > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDate.java > 5a31e61 > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToTimestampTZ.java > PRE-CREATION > > ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java > 0dc6b19 > ql/src/test/queries/clientnegative/serde_regex.q c9cfc7d > ql/src/test/queries/clientnegative/serde_regex2.q a29bb9c > ql/src/test/queries/clientnegative/serde_regex3.q 4e91f06 > ql/src/test/queries/clientpositive/create_like.q bd39731 > ql/src/test/queries/clientpositive/join43.q 12c45a6 > ql/src/test/queries/clientpositive/serde_regex.q e21c6e1 > ql/src/test/queries/clientpositive/timestamptz.q PRE-CREATION > ql/src/test/queries/clientpositive/timestamptz_1.q PRE-CREATION > ql/src/test/queries/clientpositive/timestamptz_2.q PRE-CREATION > ql/src/test/results/clientnegative/serde_regex.q.out a1ec5ca > ql/src/test/results/clientnegative/serde_regex2.q.out 374675d > ql/src/test/results/clientnegative/serde_regex3.q.out dc0a0e2 > ql/src/test/results/clientpositive/create_like.q.out ff2e752 > ql/src/test/results/clientpositive/join43.q.out e8c7278 > ql/src/test/results/clientpositive/serde_regex.q.out 7bebb0c > ql/src/test/results/clientpositive/timestamptz.q.out PRE-CREATION > ql/src/test/results/clientpositive/timestamptz_1.q.out PRE-CREATION > ql/src/test/results/clientpositive/timestamptz_2.q.out PRE-CREATION > serde/if/serde.thrift 1d40d5a > serde/src/gen/thrift/gen-cpp/serde_constants.h 8785bd2 > serde/src/gen/thrift/gen-cpp/serde_constants.cpp 907acf2 > > serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java > 2578d3e > serde/src/gen/thrift/gen-php/org/apache/hadoop/hive/serde/Types.php ea2dbbe > serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py > e3b24eb > serde/src/gen/thrift/gen-rb/serde_constants.rb 15efaea > serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 5ecfbca > > serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java > 89e15c3 > serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampTZWritable.java > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 23dbe6a > serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestampTZ.java > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 73c72e1 > > serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java > 5601734 > > serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampTZObjectInspector.java > PRE-CREATION > > serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java > 52f3527 > > serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java > 56b4ca3 > > serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestampTZ.java > PRE-CREATION > > serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java > 8237b64 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java > 24b3d4e > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java > ba44bae > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java > 70633f3 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampTZObjectInspector.java > PRE-CREATION > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java > e08ad43 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java > 2ed0843 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java > 9642a7e > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampTZObjectInspector.java > PRE-CREATION > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampTZObjectInspector.java > PRE-CREATION > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampTZObjectInspector.java > PRE-CREATION > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampTZObjectInspector.java > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/thrift/Type.java 0ad8c02 > serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java > 43c4819 > > serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampTZWritable.java > PRE-CREATION > service-rpc/if/TCLIService.thrift 824b049 > service-rpc/src/gen/thrift/gen-cpp/TCLIService_constants.cpp 991cb2e > service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 8accf66 > service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp b6995c4 > > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java > 930bed7 > > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java > 18a7825 > > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TTypeId.java > a3735eb > service-rpc/src/gen/thrift/gen-php/Types.php ee5acd2 > service-rpc/src/gen/thrift/gen-py/TCLIService/constants.py c8d4f8f > service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py e9faa2a > service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_constants.rb 25adbb4 > service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 714309c > service/src/java/org/apache/hive/service/cli/ColumnValue.java 76e8c03 > service/src/java/org/apache/hive/service/cli/TypeDescriptor.java d634bef > > > Diff: https://reviews.apache.org/r/50787/diff/7/ > > > Testing > ------- > > > Thanks, > > Rui Li > >
