[Spark 2.0 state store] Streaming wordcount using spark state store

Hi What is the right way of using spark2.0 state store feature in spark streaming??I referred test cases in this(https://github.com/apache/spark/pull/11645/files) pull request and implemented word count using state store.My source is kafka(1 topic, 10 partitions). My data pump is pushing number

Spark Streaming share state between two streams

HiCan we share spark streaming state between two DStreams??Basically I want to create state using first stream and enrich second stream using state.Example: I have modified StatefulNetworkWordCount example. I am creating state using first stream and enriching second stream with count of first st

Re: --executor-cores cannot change vcores in yarn?

If you are using capacity scheduler in yarn: By default yarn capacity scheduler uses DefaultResourceCalculator. DefaultResourceCalculator consider¹s only memory while allocating contains. You can use DominantResourceCalculator, it considers memory and cpu. In capacity-scheduler.xml set yarn.schedul

Access file name in map function

Hi In one of our usecase, filename contains timestamp and we have to append it in the record for aggregation. How can I access filename in map function? Thanks!