[
https://issues.apache.org/jira/browse/HUDI-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-2865:
----------------------------
Component/s: Spark Integration
> Enable timeline-server-based markers for Spark structured streaming
> -------------------------------------------------------------------
>
> Key: HUDI-2865
> URL: https://issues.apache.org/jira/browse/HUDI-2865
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Spark Integration
> Reporter: Ethan Guo
> Priority: Major
>
> For Spark structured streaming, after the end of first micro-batch, the write
> client is closed and hence triggers closure of timeline service. But
> subsequent micro-batches do succeed though.
> Currently, we explicitly override the marker type to be DIRECT in Spark
> structured streaming. We need to revisit the behavior of timeline service
> before we can enable timeline-server-based markers for Spark structured
> streaming.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
