Spark application fail wit numRecords error

Hi, I searched the error in kafka but i think at last, it is related with spark not kafka. Has anyone faced to an exception that is terminating program with error "numRecords must not be negative" while streaming ? Thanx in advance. Regards. Bu ileti hukuke

RE: Spark application fail wit numRecords error

-kafka-fail-with-requirement-failed-n https://forums.databricks.com/questions/11055/how-to-resolve-illegalargumentexception-requiremen.html From: Prem Sure [mailto:sparksure...@gmail.com] Sent: Wednesday, November 1, 2017 8:11 PM To: Serkan TAS Cc: user@spark.apache.org Subject: Re: Spark

Which kafka client to use with spark streaming

Hi, Working on spark 2.2.0 cluster and 1.0 kafka brokers. I was using the library "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0" and had lots of problems during streaming process then downgraded to "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.2.0"

RE: Which kafka client to use with spark streaming

Kafka Clients are blocking spark streaming jobs and after a time streaming job queue increases. -Original Message- From: Cody Koeninger [mailto:c...@koeninger.org] Sent: Tuesday, December 26, 2017 6:47 PM To: Diogo Munaro Vieira Cc: Serkan TAS ; user Subject: Re: Which kafka client to

Invalid Spark URL: spark://HeartbeatReceiver@hostname

While trying to execute python script with pycharm on Windows version am getting this error. Anyone has and ideaabout the error ? Spark version : 2.3.0 py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : org.apache.spark.SparkExcepti

Java API for statistics of spark job running on yarn

Hi all, I am facing and issue for long running spark job on yarn. If there occures some bottle neck on hdfs and/or kafka, active batch count increases immidiately. I am plannning to check the active batch count with java client and create alarms for the operations group. So, is it possible to