Re: Kinesis Receiver not respecting spark.streaming.receiver.maxRate

Hi Aravindh, spark.streaming.receiver.maxRate is per receiver. You should multiply with number of receivers with max rate. Regards, Sai On Mon, Sep 19, 2016 at 9:31 AM, Aravindh [via Apache Spark User List] < ml-node+s1001560n27754...@n3.nabble.com> wrote: > I am trying to throttle my spark c

Spark Job not failing

Hi , I have primary key on sql table iam trying to insert Dataframe into table using insertIntoJDBC. I could see failure instances in logs but still spark job is getting successful. Do you know how can we handle in code to make it fail? 16/09/19 18:52:51 INFO TaskSetManager: Starting task 0.

Re: very slow parquet file write

Hi, try this conf val sc = new SparkContext(conf) sc.hadoopConfiguration.setBoolean("parquet.enable.summary-metadata", false) Regards, Sai Ganesh On Thu, Sep 15, 2016 at 11:34 PM, gaurav24 [via Apache Spark User List] < ml-node+s1001560n27738...@n3.nabble.com> wrote: > Hi Rok, > > facing sim

Re: Guide Step by step Stark streaming

Hi, You can refer HDinsight link. It has working example for cloud based eventhub-sparkstreaming application https://azure.microsoft.com/en-us/documentation/articles/hdinsight-apache-spark-eventhub-streaming/ Regards, Sai On Thu, Sep 15, 2016 at 10:44 AM, kiaia [via Apache Spark User List]

Re: Is it possible to submit Spark Application remotely?

you can livy to submit the spark jobs to remotely http://gethue.com/how-to-use-the-livy-spark-rest-job-server-api-for-submitting-batch-jar-python-and-streaming-spark-jobs/ Regards, Sai Ganesh On Tue, Sep 6, 2016 at 1:24 PM, neil90 [via Apache Spark User List] < ml-node+s1001560n27668...@n3.nabbl

Spark Streaming WAL issue**: File exists and there is no append support!

Hello I am using Azure Blob storage for Wal persistence. Iam getting below warnings in driver logs. Is it it something related to Compatibility/throttling issues with storage? java.lang.IllegalStateException: File exists and there is no append support! at org.apache.spark.streaming.util.