[
https://issues.apache.org/jira/browse/IMPALA-14051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith updated IMPALA-14051:
-----------------------------------
Target Version: Impala 4.5.1
> Use batch insert HMS API to fire insert events from Impala
> ----------------------------------------------------------
>
> Key: IMPALA-14051
> URL: https://issues.apache.org/jira/browse/IMPALA-14051
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Sai Hemanth Gantasala
> Assignee: Sai Hemanth Gantasala
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> Firing insert statements from Impala can be improved by leveraging the batch
> insert HMS API. Right now, Impala is using a single call for each partition,
> so firing insert statements on a huge partitioned table (look at logs below)
> can become a performance bottleneck.
> {code:java}
> I0430 21:08:39.927217 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 6]
> I0430 21:08:40.134413 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 9]
> I0430 21:08:40.284446 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 8]
> I0430 21:08:40.409381 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 3]
> I0430 21:08:40.534400 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 2]
> I0430 21:08:40.659636 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 5]
> I0430 21:08:40.884773 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2014, Sep, 19]
> I0430 21:08:41.076419 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2014, Sep, 18]
> I0430 21:08:41.251446 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2023, Jan, 4]
> I0430 21:08:41.443357 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2014, Sep, 17]
> I0430 21:08:41.618523 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2019, Dec, 3]
> I0430 21:08:42.110314 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2014, Sep, 16]
> I0430 21:08:42.218703 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2019, Dec, 2]
> I0430 21:08:42.327061 826 MetastoreShim.java:433]
> ed40d2f54233490f:5d56c42b00000000] Firing write notification log request for
> table highpartition_5k_100_hive2.stocks_highpartition_1002 on partition
> [2014, Sep, 15] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
