Re: Memory issue in pyspark for 1.6 mb file

Hi Naga Is it failing because of driver memory full or executor memory full ? can you please try setting this property spark.cleaner.ttl ? . So that older RDDs /metadata should also get clear automatically. Can you please provide the complete error stacktrace and code snippet ?. Regards Prala

Memory issue in pyspark for 1.6 mb file

Hi, I am trying to load 1.6 mb excel file which has 16 tabs. We converted excel to csv and loaded 16 csv files to 8 tables. Job was running successful in 1st run in pyspark. When trying to run the same job 2 time, container getting killed due to memory issues. I am using unpersist and clearcache

Re: Crowdsourced triage Scapegoat compiler plugin warnings

Looks like a whole lot of the results have been analyzed. I suspect there's more than enough to act on already. I think we should wait until after 2.2 is done. Anybody prefer how to proceed here -- just open a JIRA to take care of a batch of related types of issues and go for it? On Sat, Jun 17, 2

Re: Output Committers for S3

Hi, as @Venkata krishnan pointed out spark does not allow DFOC when append mode is enabled. in the following class in spark, there is a small check org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol if (isAppend) { // If we are appending data to an existing d

Re: Crowdsourced triage Scapegoat compiler plugin warnings

Gentle ping to dev for help. I hope this effort is not abandoned. On 25 May 2017 9:41 am, "Josh Rosen" wrote: I'm interested in using the Scapegoat Scala compiler plugin to find potential bugs and performance problems in Spark. Scapegoat has a useful built

Re: Output Committers for S3

I think Spark in itself doesn't allow DFOC when append mode is enabled. So DFOC works only for Insert overwrite queries/overwrite mode not for append mode. Regards Venkata krishnan On Fri, Jun 16, 2017 at 9:35 PM, sririshindra wrote: > Hi Ryan and Steve, > > Thanks very much for your reply. > >