Jeff Nadler created SPARK-17510:
-----------------------------------
Summary: Set Streaming MaxRate Independently For Multiple Streams
Key: SPARK-17510
URL: https://issues.apache.org/jira/browse/SPARK-17510
Project: Spark
Issue Type: Improvement
Components: Streaming
Affects Versions: 2.0.0
Reporter: Jeff Nadler
We use multiple DStreams coming from different Kafka topics in a Streaming
application.
Some settings like maxrate and backpressure enabled/disabled would be better
passed as config to KafkaUtils.createStream and KafkaUtils.createDirectStream,
instead of setting them in SparkConf.
Being able to set a different maxrate for different streams is an important
requirement for us; we currently work-around the problem by using one
receiver-based stream and one direct stream.
We would like to be able to turn on backpressure for only one of the streams as
well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
