[
https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491460#comment-15491460
]
Jeff Nadler commented on SPARK-17510:
-------------------------------------
That would be incredible, thank you very much for looking into this.
If we were able to set maxRate independently I think we'd just use Direct
Streams for both topics, with no maxRate for the session stream, and our normal
maxRate for the events.
> Set Streaming MaxRate Independently For Multiple Streams
> --------------------------------------------------------
>
> Key: SPARK-17510
> URL: https://issues.apache.org/jira/browse/SPARK-17510
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 2.0.0
> Reporter: Jeff Nadler
>
> We use multiple DStreams coming from different Kafka topics in a Streaming
> application.
> Some settings like maxrate and backpressure enabled/disabled would be better
> passed as config to KafkaUtils.createStream and
> KafkaUtils.createDirectStream, instead of setting them in SparkConf.
> Being able to set a different maxrate for different streams is an important
> requirement for us; we currently work-around the problem by using one
> receiver-based stream and one direct stream.
> We would like to be able to turn on backpressure for only one of the streams
> as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
