Hi Naga
Is it failing because of driver memory full or executor memory full ?
can you please try setting this property spark.cleaner.ttl ? . So that
older RDDs /metadata should also get clear automatically.
Can you please provide the complete error stacktrace and code snippet ?.
Regards
Prala
Hi,
I am trying to load 1.6 mb excel file which has 16 tabs. We converted excel
to csv and loaded 16 csv files to 8 tables. Job was running successful in
1st run in pyspark. When trying to run the same job 2 time, container
getting killed due to memory issues.
I am using unpersist and clearcache
