Re: Hive on Spark Vs Spark SQL

So does not benefit from Project Tungsten right? On Mon, Nov 16, 2015 at 12:07 PM, Reynold Xin wrote: > It's a completely different path. > > > On Sun, Nov 15, 2015 at 10:37 PM, kiran lonikar wrote: > >> I would like to know if Hive on Spark uses or shares the execut

Hive on Spark Vs Spark SQL

I would like to know if Hive on Spark uses or shares the execution code with Spark SQL or DataFrames? More specifically, does Hive on Spark benefit from the changes made to Spark SQL, project Tungsten? Or is it completely different execution path where it creates its own plan and executes on RDD?

Re: Code generation for GPU

ghran" wrote: > > > On 9 Sep 2015, at 20:18, lonikar wrote: > > > > I have seen a perf improvement of 5-10 times on expression evaluation > even > > on "ordinary" laptop GPUs. Thus, it will be a good demo along with some > > concrete proposals for

Re: Code generation for GPU

Thanks. Yes thats exactly what i would like to do: copy large amounts of data to GPU RAM, perform computation and get bulk rows back for map/filter or reduce result. It is true that non trivial operations benefit more. Even streaming data to GPU RAM and interleaving computation with data transfer w

Spark 1.5.0: setting up debug env

I have setup spark debug env on windows and mac, and thought its worth sharing given some of the issues I encountered and the instructions given here did not work for *eclipse* (possibly outd

Re: Spark 1.5: How to trigger expression execution through UnsafeRow/TungstenProject

thanks that worked -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-5-How-to-trigger-expression-execution-through-UnsafeRow-TungstenProject-tp14026p14053.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. -

Re: Spark 1.5.x: Java files in src/main/scala and vice versa

It does not cause any problem when building using maven. But when doing eclipse:eclipse, the generated .classpath files contained only . This caused all the .scala sources to be ignored and caused all kinds of eclipse build errors. It resolved only when I added prebuild jars in the java build path,

Spark 1.5.x: Java files in src/main/scala and vice versa

I found these files: spark-1.5.0/sql/catalyst/*src/main/scala*/org/apache/spark/sql/types/*SQLUserDefinedType.java* spark-1.5.0/core/src/main/java/org/apache/spark/api/java/function/package.scala and several java files in spark-1.5.0/core/src/main/scala/. Is this intentional or inadvertant? --

Spark 1.5: How to trigger expression execution through UnsafeRow/TungstenProject

The tungsten, cogegen etc options are enabled by default. But I am not able to get the execution through the UnsafeRow/TungstenProject. It still executes using InternalRow/Project. I see this in the SparkStrategies.scala: If unsafe mode is enabled and we support these data types in Unsafe, use the

Re: Code generation for GPU

I am already looking at the dataframes APIs and the implementation. In fact, the columnar representation https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnType.scala is what gave me the idea of my talk proposal. It is ideally suited for computat

Re: Code generation for GPU

Hi Reynold, Thanks for responding. I was waiting for this on the spark user group and my own email id since I had not posted this on spark dev. Just saw your reply. 1. I figured the various code generation classes have either *apply* or *eval* method depending on whether it computes something or