[
https://issues.apache.org/jira/browse/FLINK-30556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luoyuxia updated FLINK-30556:
-----------------------------
Description:
Currently, when read hive source in batch mode, it'll first enumerate all split
for the hive table. But when the table is large, the split will be too many
which may well cause OOM. Some commuity users has also reported this problem.
We need to optimize the logic for enumerating splits for hive table source to
avoid OOM.
was:
Currently, when read hive source in batch mode, it'll first enumerate all split
for the hive table. But when the table is large, the split will be too many
which it may well cause OOM. Some commuity users has also reported this
problem.
We need to optimize the logic for enumerating splits for hive table source to
avoid OOM.
> Improve the logic for enumerating splits for Hive source to avoid potential
> OOM
> -------------------------------------------------------------------------------
>
> Key: FLINK-30556
> URL: https://issues.apache.org/jira/browse/FLINK-30556
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Hive
> Affects Versions: 1.16.0
> Reporter: luoyuxia
> Priority: Major
>
> Currently, when read hive source in batch mode, it'll first enumerate all
> split for the hive table. But when the table is large, the split will be too
> many which may well cause OOM. Some commuity users has also reported this
> problem.
> We need to optimize the logic for enumerating splits for hive table source to
> avoid OOM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
