Writing to mysql from pyspark spark structured streaming

2020-10-15 Thread Krishnanand Khambadkone
Hi,  I am trying to write to mysql from a spark structured streaming kafka source.  Using spark 2.4.0. I get this exception, AnalysisException: u"'write' can not be called on streaming Dataset/DataFrame

Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077

2015-03-02 Thread Krishnanand Khambadkone
Hi,  I am running spark on my mac.   It is reading from a kafka topic and then writes the data to a hbase table.  When I do a spark submit,  I get this error, Error connecting to master spark://localhost:7077 (akka.tcp://sparkMaster@localhost:7077), exiting. Cause was: akka.remote.InvalidAssociat

Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077

2015-03-02 Thread Krishnanand Khambadkone
be some more output following the above line. Can you post them ? Cheers On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone wrote: Hi,  I am running spark on my mac.   It is reading from a kafka topic and then writes the data to a hbase table.  When I do a spark submit,  I get this error

Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077

2015-03-02 Thread Krishnanand Khambadkone
e. Can you post them ? Cheers On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone wrote: Hi,  I am running spark on my mac.   It is reading from a kafka topic and then writes the data to a hbase table.  When I do a spark submit,  I get this error, Error connecting to master spark://localhost:

Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077

2015-03-02 Thread Krishnanand Khambadkone
r 2, 2015 at 2:06 PM, Krishnanand Khambadkone wrote: Hi,  I am running spark on my mac.   It is reading from a kafka topic and then writes the data to a hbase table.  When I do a spark submit,  I get this error, Error connecting to master spark://localhost:7077 (akka.tcp://sparkMaster@localh

Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077

2015-03-03 Thread Krishnanand Khambadkone
e) Can you turn on life cycle event logging to see if you would get some more clue ? Cheers On Mon, Mar 2, 2015 at 3:56 PM, Krishnanand Khambadkone wrote: I see these messages now, spark.master -> spark://krishs-mbp:7077 Classpath elements: Sending launch command to spark://krishs-mbp:7

java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils

2015-03-03 Thread Krishnanand Khambadkone
Hi,  When I submit my spark job, I see the following runtime exception in the log, Exception in thread "Thread-1" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils at SparkHdfs.run(SparkHdfs.java:56) Caused by: java.lang.ClassNotFoundException: org.apache.spark.