[
https://issues.apache.org/jira/browse/HUDI-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-3367:
--------------------------------------
Description: We have support to set scheduling configs in spark for
deltastreamer. this was added so that we can allocate resources efficiently
between regular delta syncs and async table services like compaction. We don't
have that support in spark streaming writes. we might need the support since
spark streaming also supports async table services. (was: We have support to
set scheduling configs in spark for deltastreamer. this was added so that we
can allocate resources efficiently between regular delta syncs and async table
services like compaction. We don't have that support in spark data source
writer. we might need the support to keep it flexible for users. )
> Add spark scheduling configs support to spark streaming writes
> --------------------------------------------------------------
>
> Key: HUDI-3367
> URL: https://issues.apache.org/jira/browse/HUDI-3367
> Project: Apache Hudi
> Issue Type: Task
> Components: spark
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Blocker
> Fix For: 0.11.0
>
>
> We have support to set scheduling configs in spark for deltastreamer. this
> was added so that we can allocate resources efficiently between regular delta
> syncs and async table services like compaction. We don't have that support in
> spark streaming writes. we might need the support since spark streaming also
> supports async table services.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
