Re: Spark 2.4 lifetime

I don't think there's an official EOL for Spark 2.4.x, but would expect another maintenance release in the first half of 2021 at least. I'd also guess it wouldn't be maintained by 2022. On Wed, Nov 11, 2020 at 12:24 AM Netanel Malka wrote: > Hi folks, > Do you know about how long Spark will cont

Slow insert into overwrite in spark in object store backed hive tables

Hi, We are using the InsertInto method of dataframe to write into an object store backed hive table in Google cloud. We have observed slowness in this approach. >From the internet, we got to know Writes to Hive tables in Spark happen in a two-phase manner. - Step 1 – DistributedWrite: Data is

spark UI storage tab

Hi , I have few questions as below 1. In the spark ui storage tab is displayed 'storage level',' size in memory' and size on disk, i am not sure it displays RDD ID 16 with memory usage 76 MB not sure why it is not getting 0 once a request for spark streaming is completed. I am caching some RDD ins