Re: Sounds like Structured streaming with foreach, can only run on one executor

2021-03-09 Thread Mich Talebzadeh
Thanks Sean, I am using PySpark. There seems to be some reports on foreach usage with local mode back on the 3rd March. For example, see "Spark structured streaming seems to work on local mode only" I believe the thread owner was reporting on* foreach *case not foreachBatch. cheers *Disclaime

Re: Sounds like Structured streaming with foreach, can only run on one executor

2021-03-09 Thread Sean Owen
That should not be the case. See https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch Maybe you are calling .foreach on some Scala object inadvertently. On Tue, Mar 9, 2021 at 4:41 PM Mich Talebzadeh wrote: > Hi, > > When I use *foreachB