Udit Mehrotra created HUDI-1230:
-----------------------------------
Summary: Spark-submit for MOR table creation via DataSource API
hangs
Key: HUDI-1230
URL: https://issues.apache.org/jira/browse/HUDI-1230
Project: Apache Hudi
Issue Type: Bug
Components: Compaction, Spark Integration
Reporter: Udit Mehrotra
Fix For: 0.6.0
After the addition of Async compaction functionality for streaming workloads
with Hudi 0.6.0, MOR table creation via Hudi datasource hangs at the end of the
job.
This appears to be because of a bug where at
[https://github.com/apache/hudi/blob/release-0.6.0/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L408]
we are not checking if the async trigger function has been passed, which is
passed only in case of streaming. As a result this becomes true for datasource
as well.
This leads to
[https://github.com/apache/hudi/blob/release-0.6.0/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L421]
not being triggered which does not stop the timeline server at the end, and as
a result the job hangs and never exits.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
