Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-29 Thread Cody Koeninger
t; >> > run this with better performance would be of great help. >>>> >> > >>>> >> > java.lang.AssertionError: assertion failed: Failed to get records >>>> for >>>> >> > test >>

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread swetha kasireddy
p[String, Object]( >>> >> > "bootstrap.servers" -> kafkaBrokers, >>> >> > "key.deserializer" -> classOf[StringDeserializer], >>> >> > "value.deserializer" -> classOf[StringDeserializer], >>&g

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread swetha kasireddy
er], >> >> > "value.deserializer" -> classOf[StringDeserializer], >> >> > "auto.offset.reset" -> "latest", >> >> > "heartbeat.interval.ms" -> Integer.valueOf(2), >> >> >

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread swetha kasireddy
auto.offset.reset" -> "latest", > >> > "heartbeat.interval.ms" -> Integer.valueOf(2), > >> > "session.timeout.ms" -> Integer.valueOf(6), > >> > "request.timeout.ms" -> Integer.valueOf(9), > >&

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread Cody Koeninger
teger.valueOf(2), >> > "session.timeout.ms" -> Integer.valueOf(6), >> > "request.timeout.ms" -> Integer.valueOf(9), >> > "enable.auto.commit" -> (false: java.lang.Boolean), >> > "spark.streaming.kafka.consumer.cache.enabled&quo

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread swetha kasireddy
gt; "request.timeout.ms" -> Integer.valueOf(9), >> > "enable.auto.commit" -> (false: java.lang.Boolean), >> > "spark.streaming.kafka.consumer.cache.enabled" -> "false", >> > "group.id" -> "test1

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread swetha kasireddy
roup.id" -> "test1" > > ) > > > > val hubbleStream = KafkaUtils.createDirectStream[String, String]( > > ssc, > > LocationStrategies.PreferConsistent, > > ConsumerStrategies.Subscribe[String, String](topicsSet, >

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread Cody Koeninger
est1" > ) > > val hubbleStream = KafkaUtils.createDirectStream[String, String]( > ssc, > LocationStrategies.PreferConsistent, > ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams) > ) > > > &g

Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread SRK
nabled" -> "false", "group.id" -> "test1" ) val hubbleStream = KafkaUtils.createDirectStream[String, String]( ssc, LocationStrategies.PreferConsistent, ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams)

How to make sure that Spark Kafka Direct Streaming job maintains the state upon code deployment?

2017-06-27 Thread SRK
-make-sure-that-Spark-Kafka-Direct-Streaming-job-maintains-the-state-upon-code-deployment-tp28799.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr

Re: Spark Kafka Direct Streaming

2015-07-07 Thread Tathagata Das
ic); > > Questions - > > Q1- Is my Kafka configuration correct or should it be changed? > > Q2- I also looked into the Checkpointing but in my usecase, Data > checkpointing is not required but meta checkpointing is required. Can I > achieve this, i.e. enabling meta checkpoin

Spark Kafka Direct Streaming

2015-07-07 Thread abi_pat
I achieve this, i.e. enabling meta checkpointing and not the data checkpointing? Thanks Abhishek Patel -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Kafka-Direct-Streaming-tp23685.html Sent from the Apache Spark User List mailing list archive at Nabb