Spark 2.0 on HDP

2016-10-27 Thread Deenar Toraskar
org.apache.spark.deploy.yarn.ExecutorLauncher I have verified that both spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions have the hdp.version set correctly. Anything else I am missing? Regards Deenar On 10 May 2016 at 13:48, Steve Loughran wrote: > > On 9 May 2016, at 21:24, Jesse F Chen wrote: >

Spark 2.0 Aggregator with complex types

2016-06-21 Thread Deenar Toraskar
. There is a similar JIRA, but that does not fix my issue https://issues.apache.org/jira/browse/SPARK-15704. Checked on last night's build. Regards Deenar import org.apache.spark.sql.functions._ import org.apache.spark.sql.TypedColumn import org.apache.spark.sql.catalyst.encoders.ExpressionEn

Re: Spark 1.6.1

2016-01-29 Thread deenar
eduction def sumArray(a: Seq[N], b: Seq[N]): Seq[N] = { (a, b) match { case (Nil, Nil) => Nil case (Nil, row) => row case (sum, Nil) => sum case (sum, row) => (a, b).zipped.map { case (a, b) => numeric.plus(a, b) } } } } Regards Deenar -- Vie