How to display column names in spark-sql output

Hi, When we run spark-sql, is there a way to get column names/headers with the result? -- Thanks, Ashwin

Re: Problem with pyspark on Docker talking to YARN cluster

Hi Eron, Thanks for your reply, but none of these options works for us. > > >1. use 'spark.driver.host' and 'spark.driver.port' setting to >stabilize the driver-side endpoint. (ref >) > > This unfortunately won't help

Problem with pyspark on Docker talking to YARN cluster

All, I was wondering if any of you have solved this problem : I have pyspark(ipython mode) running on docker talking to a yarn cluster(AM/executors are NOT running on docker). When I start pyspark in the docker container, it binds to port *49460.* Once the app is submitted to YARN, the app(AM) o

How to pass system properties in spark ?

Hi, I'm trying to use property substitution in my log4j.properties, so that I can choose where to write spark logs at runtime. The problem is that, system property passed to spark shell doesn't seem to getting propagated to log4j. *Here is log4j.properites(partial) with a parameter 'spark.log.path

Spark on Yarn : Map outputs lifetime ?

Hi, In spark on yarn and when running spark_shuffle as auxiliary service on node manager, does map spills of a stage gets cleaned up once the next stage completes OR is it preserved till the app completes(ie waits for all the stages to complete) ? -- Thanks, Ashwin

Re: Building spark targz

e but are you looking for the tar in assembly/target dir ? > > On Wed, Nov 12, 2014 at 3:14 PM, Ashwin Shankar > wrote: > >> Hi, >> I just cloned spark from the github and I'm trying to build to generate a >> tar ball. >> I'm doing : mvn -Pyarn -Pha

Building spark targz

Hi, I just cloned spark from the github and I'm trying to build to generate a tar ball. I'm doing : mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -DskipTests clean package Although the build is successful, I don't see the targz generated. Am I running the wrong command ? -- Thanks, Ashw

Re: Multitenancy in Spark - within/across spark context

et me try to answer to the best of my knowledge. > > On Wed, Oct 22, 2014 at 11:47 AM, Ashwin Shankar > wrote: > > Here are my questions : > > 1. Sharing spark context : How exactly multiple users can share the > cluster > > using same spark > > context

Multitenancy in Spark - within/across spark context

Hi Spark devs/users, One of the things we are investigating here at Netflix is if Spark would suit us for our ETL needs, and one of requirements is multi tenancy. I did read the official doc and the book, but I'm still not clear on certain t