Re: Re: should one every make a spark streaming job in pyspark

2022-11-03 Thread Lingzhe Sun
teful-processing-structured-streaming.html Best Regards! ... Lingzhe Sun Hirain Technology / APIC From: Mich Talebzadeh Date: 2022-11-03 19:15 To: Joris Billen CC: User Subject: Re: should one every make a spark streaming j

Re: should one every make a spark streaming job in pyspark

2022-11-03 Thread Mich Talebzadeh
Well your mileage varies so to speak. - Spark itself is written in Scala. However, that does not imply you should stick with Scala. - I have used both for spark streaming and spark structured streaming, they both work fine - PySpark has become popular with the widespread use of iDa

should one every make a spark streaming job in pyspark

2022-11-02 Thread Joris Billen
Dear community, I had a general question about the use of scala VS pyspark for spark streaming. I believe spark streaming will work most efficiently when written in scala. I believe however that things can be implemented in pyspark. My question: 1)is it completely dumb to make a streaming job in