[
https://issues.apache.org/jira/browse/IMPALA-14051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17956456#comment-17956456
]
ASF subversion and git services commented on IMPALA-14051:
----------------------------------------------------------
Commit f545a0828df9874881a224b49575e0aebb195dab in impala's branch
refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f545a0828 ]
IMPALA-14051: Use batch insert HMS API to fire insert events
Firing insert statements from Impala can be improved by leveraging the
batch insert HMS API addWriteNotificationLogInBatch() available (via
HIVE-25205) starting from Hive 4. Right now, Impala is using a single
call for each partition, so firing insert statements on a huge
partitioned table can become a performance bottleneck.
This patch addresses the above concern by leveraging
addWriteNotificationLogInBatch() API to fire batch insert partition
events.
Note: This optimization is applicable only for transactional partitioned
tables. The external partitioned table case is already handled.
Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Reviewed-on: http://gerrit.cloudera.org:8080/22869
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Riza Suminto <[email protected]>
> Use batch insert HMS API to fire insert events from Impala
> ----------------------------------------------------------
>
> Key: IMPALA-14051
> URL: https://issues.apache.org/jira/browse/IMPALA-14051
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Sai Hemanth Gantasala
> Assignee: Sai Hemanth Gantasala
> Priority: Major
>
> Firing insert statements from Impala can be improved by leveraging the batch
> insert HMS API. Right now, Impala is using a single call for each partition,
> so firing insert statements on a huge partitioned table (look at logs below)
> can become a performance bottleneck.
> {code:java}
> I0430 21:08:39.927217 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 6]
> I0430 21:08:40.134413 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 9]
> I0430 21:08:40.284446 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 8]
> I0430 21:08:40.409381 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 3]
> I0430 21:08:40.534400 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 2]
> I0430 21:08:40.659636 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 5]
> I0430 21:08:40.884773 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2014, Sep, 19]
> I0430 21:08:41.076419 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2014, Sep, 18]
> I0430 21:08:41.251446 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 4]
> I0430 21:08:41.443357 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2014, Sep, 17]
> I0430 21:08:41.618523 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2019, Dec, 3]
> I0430 21:08:42.110314 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2014, Sep, 16]
> I0430 21:08:42.218703 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2019, Dec, 2]
> I0430 21:08:42.327061 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2014, Sep, 15] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
