Re: Powered By Spark: Can you please add our org?

Awesome! Thanks, Reynold! On Tue, Jul 8, 2014 at 4:00 PM, Reynold Xin wrote: > I added you to the list. Cheers. > > > > On Mon, Jul 7, 2014 at 6:19 PM, Alex Gaudio wrote: > >> Hi, >> >> Sailthru is also using Spark. Could you please add us

Powered By Spark: Can you please add our org?

Hi, Sailthru is also using Spark. Could you please add us to the Powered By Spark page when you have a chance? Organization Name: Sailthru URL: www.sailthru.com Short Description: Our data science platform uses Spark to build p

Re: how to set spark.executor.memory and heap size

Hi All, This is a bit late, but I found it helpful. Piggy-backing on Wang Hao's comment, spark will ignore the "spark.executor.memory" setting if you add it to SparkConf via: conf.set("spark.executor.memory", "1g") What you actually should do depends on how you run spark. I found some "offic

Re: Powered by Spark addition

Hi Matei, Sailthru is also using Spark. Could you please add us to the Powered By Spark page when you have a chance? Organization Name: Sailthru URL: www.sailthru.com Short Description: Our data science platform uses Spark to

Re: using Log4j to log INFO level messages on workers

Hi, I had the same problem with pyspark. Here's how I resolved it: What I've found in python (not sure about scala) is that if the function being serialized was written in the same python module as the main function, then logging fails. If the serialized function is in a separate module, then