[
https://issues.apache.org/jira/browse/HIVE-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072494#comment-13072494
]
[email protected] commented on HIVE-2272:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1135/
-----------------------------------------------------------
(Updated 2011-07-28 19:17:59.868991)
Review request for hive.
Changes
-------
Decided to store timestamps as UTC on disk and add offsets (localize) if user
needs through UDF. This simplifies use cases (compared to MySQL) where users
from different time zones querying the same data would receive different
results.
Will file a follow up JIRA ticket to add table level timezone defaults. This
would allow a user to import existing data that is already offset from UTC and
write new data to the table with that offset.
Summary:
-Added config var for default timezone
-Added GenericUDF for converting from UTC to localized timezone (either
default, or supplied from list of Java timezones)
Ex: utc_to_localized_timestamp(<timestamp>) OR
utc_to_localized_timestamp(<timestamp>, "America/Chicago")
-Fixed DateDiff bug I introduced
Summary
-------
Adds TIMESTAMP type to serde2 with both string (LazySimple) and binary
(LazyBinary) serialization.
Supports SQL style jdbc timestamps of the format with nanosecond precision
YYYY-MM-DD HH:MM:SS[.fff...]
This addresses bug HIVE-2272.
https://issues.apache.org/jira/browse/HIVE-2272
Diffs (updated)
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1151571
trunk/conf/hive-default.xml 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDate.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateAdd.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateDiff.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateSub.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDayOfMonth.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHour.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMinute.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMonth.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSecond.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFWeekOfYear.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFYear.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovarianceSample.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStd.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStdSample.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVarianceSample.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java
1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java
PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtcToLocalizedTimestamp.java
PRE-CREATION
trunk/ql/src/test/queries/clientnegative/invalid_t_create3.q 1151571
trunk/ql/src/test/queries/clientpositive/timestamp_1.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/timestamp_2.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/timestamp_3.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/timestamp_comparison.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/timestamp_udf.q PRE-CREATION
trunk/ql/src/test/results/clientnegative/invalid_create_tbl1.q.out 1151571
trunk/ql/src/test/results/clientnegative/invalid_t_alter1.q.out 1151571
trunk/ql/src/test/results/clientnegative/invalid_t_alter2.q.out 1151571
trunk/ql/src/test/results/clientnegative/invalid_t_create1.q.out 1151571
trunk/ql/src/test/results/clientnegative/invalid_t_create2.q.out 1151571
trunk/ql/src/test/results/clientnegative/invalid_t_transform.q.out 1151571
trunk/ql/src/test/results/clientnegative/wrong_column_type.q.out 1151571
trunk/ql/src/test/results/clientpositive/show_functions.q.out 1151571
trunk/ql/src/test/results/clientpositive/timestamp_1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/timestamp_2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/timestamp_3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/timestamp_comparison.q.out
PRE-CREATION
trunk/ql/src/test/results/clientpositive/timestamp_udf.q.out PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java
PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestamp.java
PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampObjectInspector.java
PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestamp.java
PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampObjectInspector.java
PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampObjectInspector.java
PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampObjectInspector.java
PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampObjectInspector.java
PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java
1151571
Diff: https://reviews.apache.org/r/1135/diff
Testing
-------
Wrote some unit tests for string/binary serialization. Also ran against test
data and looked consistent
Thanks,
Franklin
> add TIMESTAMP data type
> -----------------------
>
> Key: HIVE-2272
> URL: https://issues.apache.org/jira/browse/HIVE-2272
> Project: Hive
> Issue Type: New Feature
> Reporter: Franklin Hu
> Assignee: Franklin Hu
> Attachments: hive-2272.1.patch, hive-2272.2.patch, hive-2272.3.patch,
> hive-2272.4.patch, hive-2272.5.patch, hive-2272.6.patch, hive-2272.7.patch
>
>
> Add TIMESTAMP type to serde2 that supports unix timestamp (1970-01-01
> 00:00:01 UTC to 2038-01-19 03:14:07 UTC) with optional nanosecond precision
> using both LazyBinary and LazySimple SerDes.
> For LazySimpleSerDe, the data is stored in jdbc compliant java.sql.Timestamp
> parsable strings.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
