Sai Hemanth Gantasala created IMPALA-14051:
----------------------------------------------
Summary: Use batch insert HMS API to fire insert events from Impala
Key: IMPALA-14051
URL: https://issues.apache.org/jira/browse/IMPALA-14051
Project: IMPALA
Issue Type: Improvement
Components: Catalog
Reporter: Sai Hemanth Gantasala
Assignee: Sai Hemanth Gantasala
Firing insert statements from Impala can be improved by leveraging the batch
insert HMS API. Right now, Impala is using a single call for each partition, so
firing insert statements on a huge partitioned table (look at logs below) can
become a performance bottleneck.
{code:java}
I0430 21:08:39.927217 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2023,
Jan, 6]
I0430 21:08:40.134413 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2023,
Jan, 9]
I0430 21:08:40.284446 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2023,
Jan, 8]
I0430 21:08:40.409381 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2023,
Jan, 3]
I0430 21:08:40.534400 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2023,
Jan, 2]
I0430 21:08:40.659636 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2023,
Jan, 5]
I0430 21:08:40.884773 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2014,
Sep, 19]
I0430 21:08:41.076419 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2014,
Sep, 18]
I0430 21:08:41.251446 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2023,
Jan, 4]
I0430 21:08:41.443357 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2014,
Sep, 17]
I0430 21:08:41.618523 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2019,
Dec, 3]
I0430 21:08:42.110314 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2014,
Sep, 16]
I0430 21:08:42.218703 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2019,
Dec, 2]
I0430 21:08:42.327061 826 MetastoreShim.java:433]
ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition [2014,
Sep, 15] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
