Re: Assembly jar file name does not match profile selection

2014-12-26 Thread Kousuke Saruta
Hi Alessandro, It's fixed by SPARK-3787 and will be applied to 1.2.1 and 1.3.0. https://issues.apache.org/jira/browse/SPARK-3787 - Kousuke (2014/12/27 11:15), Alessandro Baretta wrote: I am building spark with sbt off of branch 1.2. I'm using the following command: sbt/sbt -Pyarn -Phadoop-2.

Unsupported Catalyst types in Parquet

2014-12-26 Thread Alessandro Baretta
Michael, I'm having trouble storing my SchemaRDDs in Parquet format with SparkSQL, due to my RDDs having having DateType and DecimalType fields. What would it take to add Parquet support for these Catalyst? Are there any other Catalyst types for which there is no Catalyst support? Alex

Re: Assembly jar file name does not match profile selection

2014-12-26 Thread Alessandro Baretta
Here's what I get: ./assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop2.6.0.jar Alex On Fri, Dec 26, 2014 at 8:41 PM, Ted Yu wrote: > Can you try this command ? > > sbt/sbt -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive assembly > > On Fri, Dec 26, 2014 at 6:15 PM, Alessandro

Re: SQLContext is Serializable, SparkContext is not

2014-12-26 Thread Cody Koeninger
The spark context reference is transient. On Fri, Dec 26, 2014 at 6:11 PM, Alessandro Baretta wrote: > How, O how can this be? Doesn't the SQLContext hold a reference to the > SparkContext? > > Alex >

Re: Assembly jar file name does not match profile selection

2014-12-26 Thread Ted Yu
Can you try this command ? sbt/sbt -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive assembly On Fri, Dec 26, 2014 at 6:15 PM, Alessandro Baretta wrote: > I am building spark with sbt off of branch 1.2. I'm using the following > command: > > sbt/sbt -Pyarn -Phadoop-2.3 assembly > > (http://spar

Assembly jar file name does not match profile selection

2014-12-26 Thread Alessandro Baretta
I am building spark with sbt off of branch 1.2. I'm using the following command: sbt/sbt -Pyarn -Phadoop-2.3 assembly (http://spark.apache.org/docs/latest/building-spark.html#building-with-sbt) Although the jar file I obtain does contain the proper version of the hadoop libraries (v. 2.4), the a

SQLContext is Serializable, SparkContext is not

2014-12-26 Thread Alessandro Baretta
How, O how can this be? Doesn't the SQLContext hold a reference to the SparkContext? Alex

SQL specification for reference during Spark SQL development

2014-12-26 Thread Nicholas Chammas
Do we have access to the SQL specification (say, SQL-92) for reference during Spark SQL development? I know it's not freely available on the web. Usually, you can only access drafts. I know that, generally, we look to other systems (especially Hive) when figuring out how something in Spark SQL sho

Re: Spark 1.2.0 Repl

2014-12-26 Thread Sean Owen
It was not intended to be a public API but there is a request to keep publishing it as a developer API: https://issues.apache.org/jira/browse/SPARK-4923 On Dec 26, 2014 2:09 PM, "Dirceu Semighini Filho" < dirceu.semigh...@gmail.com> wrote: > Hello, > Is there any reason in not publishing spark rep

Spark 1.2.0 Repl

2014-12-26 Thread Dirceu Semighini Filho
Hello, Is there any reason in not publishing spark repl in the version 1.2.0? In repl/pom.xml the deploy and publish are been skipped. Regards, Dirceu