----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49766/ -----------------------------------------------------------
(Updated July 27, 2016, 2:54 p.m.) Review request for hive and Eugene Koifman. Changes ------- Refactor the way delete event writers are created for compaction case in favor of a better abstraction. Repository: hive-git Description ------- https://issues.apache.org/jira/browse/HIVE-14035 In current Hive version, delta files created by ACID transactions do not allow predicate pushdown if they contain any update/delete events. This is done to preserve correctness when following a multi-version approach during event collapsing, where an update event overwrites an existing insert event. This JIRA proposes to split an update event into a combination of a delete event followed by a new insert event, that can enable predicate push down to all delta files without breaking correctness. To support backward compatibility for this feature, this JIRA also proposes to add some sort of versioning to ACID that can allow different versions of ACID transactions to co-exist together. Diffs (updated) ----- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e92466f hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java 14f7316 hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java 974c6b8 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java ca2a912 metastore/if/hive_metastore.thrift 4d92b73 metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h ae14bd1 metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp f982bf2 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java 5a666f2 metastore/src/gen/thrift/gen-php/metastore/Types.php f505208 metastore/src/gen/thrift/gen-py/hive_metastore/constants.py d1c07a5 metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb eeccc84 metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalValidationListener.java 3e74675 orc/src/java/org/apache/orc/impl/TreeReaderFactory.java c4a2093 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java db6848a ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 57b6c67 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 23a13d6 ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java dd90a95 ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java c150ec5 ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 945b828 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 63d02fb ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 1a1af28 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 8cf261d ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 6caca98 ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java d48e441 ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java b83cea4 Diff: https://reviews.apache.org/r/49766/diff/ Testing ------- Tests for the feature are in ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java. These are mostly integration tests that test end-to-end insert/update/delete scenarios followed by compaction and cleaning. Thanks, Saket Saurabh
