----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49766/ -----------------------------------------------------------
(Updated Aug. 8, 2016, 5:53 p.m.) Review request for hive and Eugene Koifman. Changes ------- Updated patch with comments/feedback from code review. Major changes are refactoring of AcidUtils.getAcidState(), CompactorMR, fixing of flush length related changes in OrcRecordUpdater, lazy initialization of deleteEventWriters and other minor changes. The patch adds additional unit tests to test schema evolution with ACID. It also adds a subclass to TestTxnCommands2 for replicating all the existing ACID test cases with split-update turned on. Repository: hive-git Description ------- https://issues.apache.org/jira/browse/HIVE-14035 In current Hive version, delta files created by ACID transactions do not allow predicate pushdown if they contain any update/delete events. This is done to preserve correctness when following a multi-version approach during event collapsing, where an update event overwrites an existing insert event. This JIRA proposes to split an update event into a combination of a delete event followed by a new insert event, that can enable predicate push down to all delta files without breaking correctness. To support backward compatibility for this feature, this JIRA also proposes to add some sort of versioning to ACID that can allow different versions of ACID transactions to co-exist together. Diffs (updated) ----- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 70816bd hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java 14f7316 hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java 974c6b8 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java 731caa8 metastore/if/hive_metastore.thrift a2e35b8 metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h ae14bd1 metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp f982bf2 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java 5a666f2 metastore/src/gen/thrift/gen-php/metastore/Types.php d6f7f49 metastore/src/gen/thrift/gen-py/hive_metastore/constants.py d1c07a5 metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb eeccc84 metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalValidationListener.java 3e74675 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java db6848a ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 57b6c67 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 26e6443 ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java dd90a95 ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 449d889 ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 945b828 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 334cb31 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java efde2db ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 1a1af28 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 8cb5e8a ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 8cf261d ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 6caca98 ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java af192fb ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 556df18 ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 6648829 Diff: https://reviews.apache.org/r/49766/diff/ Testing ------- Tests for the feature are in ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java. These are mostly integration tests that test end-to-end insert/update/delete scenarios followed by compaction and cleaning. Thanks, Saket Saurabh
