[
https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491412#comment-15491412
]
Jeff Nadler commented on SPARK-17510:
-------------------------------------
Well... both streams use updateStateByKey. The session one is simple+fast,
the event one is complex+slow.
I have a branch now where I experiment with mapWithState. I don't expect huge
benefits because we need expiration, and I understand from the docs that this
negates a lot of the benefit. I'd still love to get some perf gains of course
:)
Don't have a Kafka 0.10 cluster right now but could stand one up pretty quick.
> Set Streaming MaxRate Independently For Multiple Streams
> --------------------------------------------------------
>
> Key: SPARK-17510
> URL: https://issues.apache.org/jira/browse/SPARK-17510
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 2.0.0
> Reporter: Jeff Nadler
>
> We use multiple DStreams coming from different Kafka topics in a Streaming
> application.
> Some settings like maxrate and backpressure enabled/disabled would be better
> passed as config to KafkaUtils.createStream and
> KafkaUtils.createDirectStream, instead of setting them in SparkConf.
> Being able to set a different maxrate for different streams is an important
> requirement for us; we currently work-around the problem by using one
> receiver-based stream and one direct stream.
> We would like to be able to turn on backpressure for only one of the streams
> as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
