Re: [pyspark 2.4] broadcasting DataFrame throws error

Hi, I think problem lies with driver memory. Broadcast in spark work by collecting all the data to driver and then driver broadcasting to all the executors. Different strategy could be employed for trasfer like bit torrent though. Please try increasing the driver memory. See if it works. Regards

Re: [DISCUSS] Spark cannot identify the problem executor

In spark 2.3 with blacklist enabled this is a common problem when executor A has some problem, for instance let’s say there’s some connection issue. Tasks on executor B, executor C will fail saying cannot read from executor A

unsubscribe

unsubscribe

Re: Structured Streaming Checkpoint Error

Hi Gabor, Makes sense, thanks a lot! On Thu, 17 Sep 2020 at 11:51, Gabor Somogyi wrote: > Hi, > > Structured Streaming is simply not working when checkpoint location is on > S3 due to it's read-after-write consistency. > Please choose an HDFS compliant filesystem and it will work like a charm.

Re: Structured Streaming Checkpoint Error

Hi, Structured Streaming is simply not working when checkpoint location is on S3 due to it's read-after-write consistency. Please choose an HDFS compliant filesystem and it will work like a charm. BR, G On Wed, Sep 16, 2020 at 4:12 PM German Schiavon wrote: > Hi! > > I have an Structured Stre

Re: Spark structured streaming: periodically refresh static data frame

As per the solution, if we are closing and starting the query, then what happens to the the state which is maintained in memory, will that be retained ? -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To u