Inferring schema from GenericRowWithSchema

Hi, I have a requirement to create types dynamically in Spark and then instantiate those types from Spark SQL via a UDF. I tried doing the following: val addressType = StructType(List( new StructField("state", DataTypes.StringType), new StructField("zipcode", DataTypes.IntegerType) )) sqlCo

Re: Inferring schema from GenericRowWithSchema

be able to do that. ScalaReflection is based > on the TypeTag of the object, and thus the schema of any particular object > won't be available to it. > > Instead I think you want to use the register functions in UDFRegistration > that take a schema. Does that make sense? > > On Tue, M

Regression in Java RDD sortBy() in Spark 2.0

s, Andy. -- Andy Grove Chief Architect www.agildata.com

Re: Regression in Java RDD sortBy() in Spark 2.0

or is better for sure. Thanks, Andy. -- Andy Grove Chief Architect AgilData - Simple Streaming SQL that Scales www.agildata.com On Thu, Aug 4, 2016 at 10:25 PM, Andy Grove wrote: > Hi, > > I have some working Java code with Spark 1.6 that I am upgrading to Spark > 2.0 > >

Java and SparkSession

>From some brief experiments using Java with Spark 2.0 it looks like Java developers should stick to SparkContext and SQLContext rather than using the new SparkSession API? It would be great if someone could confirm if that is the intention or not. Thanks, Andy. -- Andy Grove Chief Archit

[Spark 2.0] Error during codegen for Java POJO

that I should file a JIRA for. Thanks, Andy. -- Andy Grove Chief Architect www.agildata.com

Re: Java and SparkSession

Ah, you still have to use the JavaSparkContext rather than using the sparkSession.sparkContext ... that makes sense. Thanks for your help. Thanks, Andy. -- Andy Grove Chief Architect www.agildata.com On Fri, Aug 5, 2016 at 12:03 PM, Everett Anderson wrote: > Hi, > > Can you

Re: [Spark 2.0] Error during codegen for Java POJO

I tracked this down in the end. It turns out the POJO was not actually defined as 'public' for some reason. It seems like this should be detected as an error prior to generating code though? Thanks, Andy. -- Andy Grove Chief Architect www.agildata.com On Fri, Aug 5, 2016 at 8:2

Re: Scala MatchError in Spark SQL

he-spark-rdd-vs-dataframe-vs-dataset/ Thanks, Andy. -- Andy Grove Chief Architect AgilData - Simple Streaming SQL that Scales www.agildata.com On Wed, Jan 20, 2016 at 7:07 AM, raghukiran wrote: > Hi, > > I created a custom UserDefinedType in Java as follows: > > SQLPoint = new User

Re: Scala MatchError in Spark SQL

I'm talking about implementing CustomerRecord as a scala case class, rather than as a Java class. Scala case classes implement the scala.Product trait, which Catalyst is looking for. Thanks, Andy. -- Andy Grove Chief Architect AgilData - Simple Streaming SQL that Scales www.agildata.com

Re: Scala MatchError in Spark SQL

Honestly, moving to Scala and using case classes is the path of least resistance in the long term. Thanks, Andy. -- Andy Grove Chief Architect AgilData - Simple Streaming SQL that Scales www.agildata.com On Wed, Jan 20, 2016 at 10:19 AM, Raghu Ganti wrote: > Thanks for your reply, A

Re: Scala MatchError in Spark SQL

aghu > > On Wed, Jan 20, 2016 at 12:25 PM, Andy Grove > wrote: > >> I'm talking about implementing CustomerRecord as a scala case class, >> rather than as a Java class. Scala case classes implement the scala.Product >> trait, which Catalyst is looking for. >

Re: Use case for RDD and Data Frame

This blog post should be helpful http://www.agildata.com/apache-spark-rdd-vs-dataframe-vs-dataset/ Thanks, Andy. -- Andy Grove Chief Architect AgilData - Simple Streaming SQL that Scales www.agildata.com On Tue, Feb 16, 2016 at 9:05 AM, Ashok Kumar wrote: > Gurus, > > What are