[ https://issues.apache.org/jira/browse/HIVE-26997?focusedWorklogId=854546&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854546 ]
ASF GitHub Bot logged work on HIVE-26997: ----------------------------------------- Author: ASF GitHub Bot Created on: 03/Apr/23 13:03 Start Date: 03/Apr/23 13:03 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #4162: URL: https://github.com/apache/hive/pull/4162#discussion_r1155935183 ########## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergAcidUtil.java: ########## @@ -93,10 +95,16 @@ public static Schema createFileReadSchemaWithVirtualColums(List<Types.NestedFiel * @param dataCols The columns of the serde projection schema * @return The schema for SerDe operations, extended with metadata columns needed for deletes */ - public static Schema createSerdeSchemaForDelete(List<Types.NestedField> dataCols) { - List<Types.NestedField> cols = Lists.newArrayListWithCapacity(dataCols.size() + SERDE_META_COLS.size()); + public static Schema createSerdeSchemaForDelete(List<Types.NestedField> dataCols, boolean partitioned, + Properties serDeProperties) { + boolean skipRowData = Boolean.parseBoolean(serDeProperties.getProperty(WriterBuilder.ICEBERG_DELETE_SKIPROWDATA, + WriterBuilder.ICEBERG_DELETE_SKIPROWDATA_DEFAULT)); + List<Types.NestedField> cols = Lists.newArrayListWithCapacity( + SERDE_META_COLS.size() + (skipRowData || partitioned ? 0 : dataCols.size())); Review Comment: completely changed the approach, not valid anymore Issue Time Tracking ------------------- Worklog Id: (was: 854546) Time Spent: 2h 20m (was: 2h 10m) > Iceberg: Vectorization gets disabled at runtime in merge-into statements > ------------------------------------------------------------------------ > > Key: HIVE-26997 > URL: https://issues.apache.org/jira/browse/HIVE-26997 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration > Reporter: Rajesh Balamohan > Assignee: Zsolt Miskolczi > Priority: Major > Labels: pull-request-available > Attachments: explain_merge_into.txt > > Time Spent: 2h 20m > Remaining Estimate: 0h > > *Query:* > Think of "ssv" table as a table containing trickle feed data in the following > query. "store_sales_delete_1" is the destination table. > > {noformat} > MERGE INTO tpcds_1000_iceberg_mor_v4.store_sales_delete_1 t USING > tpcds_1000_update.ssv s ON (t.ss_item_sk = s.ss_item_sk > > AND t.ss_customer_sk=s.ss_customer_sk > > AND t.ss_sold_date_sk = "2451181" > > AND ((Floor((s.ss_item_sk) / 1000) * 1000) BETWEEN 1000 AND > 2000) > > AND s.ss_ext_discount_amt < 0.0) WHEN matched > AND t.ss_ext_discount_amt IS NULL THEN > UPDATE > SET ss_ext_discount_amt = 0.0 WHEN NOT matched THEN > INSERT (ss_sold_time_sk, > ss_item_sk, > ss_customer_sk, > ss_cdemo_sk, > ss_hdemo_sk, > ss_addr_sk, > ss_store_sk, > ss_promo_sk, > ss_ticket_number, > ss_quantity, > ss_wholesale_cost, > ss_list_price, > ss_sales_price, > ss_ext_discount_amt, > ss_ext_sales_price, > ss_ext_wholesale_cost, > ss_ext_list_price, > ss_ext_tax, > ss_coupon_amt, > ss_net_paid, > ss_net_paid_inc_tax, > ss_net_profit, > ss_sold_date_sk) > VALUES (s.ss_sold_time_sk, > s.ss_item_sk, > s.ss_customer_sk, > s.ss_cdemo_sk, > s.ss_hdemo_sk, > s.ss_addr_sk, > s.ss_store_sk, > s.ss_promo_sk, > s.ss_ticket_number, > s.ss_quantity, > s.ss_wholesale_cost, > s.ss_list_price, > s.ss_sales_price, > s.ss_ext_discount_amt, > s.ss_ext_sales_price, > s.ss_ext_wholesale_cost, > s.ss_ext_list_price, > s.ss_ext_tax, > s.ss_coupon_amt, > s.ss_net_paid, > s.ss_net_paid_inc_tax, > s.ss_net_profit, > "2451181") > {noformat} > > > *Issue:* > # Map phase is not getting vectorized due to "PARTITION_{_}SPEC{_}_ID" column > {noformat} > Map notVectorizedReason: Select expression for SELECT operator: Virtual > column PARTITION__SPEC__ID is not supported {noformat} > > 2. "Reducer 2" stage isn't vectorized. > {noformat} > Reduce notVectorizedReason: exception: java.lang.RuntimeException: Full Outer > Small Table Key Mapping duplicate column 0 in ordered column map {0=(value > column: 30, type info: int), 1=(value column: 31, type info: int)} when > adding value column 53, type into int stack trace: > org.apache.hadoop.hive.ql.exec.vector.VectorColumnOrderedMap.add(VectorColumnOrderedMap.java:102), > > org.apache.hadoop.hive.ql.exec.vector.VectorColumnSourceMapping.add(VectorColumnSourceMapping.java:41), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.canSpecializeMapJoin(Vectorizer.java:3865), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperator(Vectorizer.java:5246), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChild(Vectorizer.java:988), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChildren(Vectorizer.java:874), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperatorTree(Vectorizer.java:841), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.access$2400(Vectorizer.java:251), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeReduceOperators(Vectorizer.java:2298), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeReduceOperators(Vectorizer.java:2246), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeReduceWork(Vectorizer.java:2224), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertReduceWork(Vectorizer.java:2206), > > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:1038), > > org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111), > > org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180), > ... {noformat} > > I have attached the explain plan for this, which has details on this. -- This message was sent by Atlassian Jira (v8.20.10#820010)