Re: How to patch sparkSQL on EC2?

ahhh never mind… I didn’t notice that a spark-assembly jar file gets produced after compiling the whole spark suite… So no more manual editing of the jar file of the AMI for now! Christos On Oct 10, 2014, at 12:15 AM, Christos Kozanitis wrote: > Hi > > I have written a few exten

How to patch sparkSQL on EC2?

Hi I have written a few extensions for sparkSQL (for version 1.1.0) and I am trying to deploy my new jar files (one for catalyst and one for sql/core) on ec2. My approach was to create a new spark/lib/spark-assembly-1.1.0-hadoop1.0.4.jar that merged the contents of the old one with the content

Re: SparkSQL extensions

and proposing a design before going too far. > > Michael > > > On Sat, Jul 26, 2014 at 3:32 AM, Christos Kozanitis < > kozani...@berkeley.edu> wrote: > >> Hello >> >> I was wondering is it easy for you guys to point me to what modules I >> need t

SparkSQL extensions

Hello I was wondering is it easy for you guys to point me to what modules I need to update if I had to add extra functionality to sparkSQL? I was thinking to implement a region-join operator and I guess I should add the implementation details under joins.scala but what else do I need to modify?

Re: SparkSQL operator priority

ub.com/apache/spark/pull/511/ > > On Jul 18, 2014, at 8:04 PM, Christos Kozanitis > wrote: > > Hello > > What is the order with which SparkSQL deserializes parquet fields? Is it > possible to modify it? > > I am using SparkSQL to query a parquet file that consists of a lot o

SparkSQL operator priority

Hello What is the order with which SparkSQL deserializes parquet fields? Is it possible to modify it? I am using SparkSQL to query a parquet file that consists of a lot of fields (around 30 or so). Let me call an example table MyTable and let's suppose the name of one of its fields is "position".