Re: Support Hive 0.13 .1 in Spark SQL

Hey Cheng, Right now we aren't using stable API's to communicate with the Hive Metastore. We didn't want to drop support for Hive 0.12 so right now we are using a shim layer to support compiling for 0.12 and 0.13. This is very costly to maintain. If Hive has a stable meta-data API for talking to

Workaround for python's inability to unzip zip64 spark assembly jar

Hi All, We recently faced the known issue where pyspark does not work when the assembly jar contains more than 65K files. Our build and run time environment are both Java 7 but python fails to unzip the assembly jar as expected (https://issues.apache.org/jira/browse/SPARK-1911). All nodes in o

Re: HiveContext bug?

Well, looks like a huge coincidence, but this was just sent to github: https://github.com/apache/spark/pull/2967 On Mon, Oct 27, 2014 at 3:25 PM, Marcelo Vanzin wrote: > Hey guys, > > I've been using the "HiveFromSpark" example to test some changes and I > ran into an issue that manifests itself

HiveContext bug?

Hey guys, I've been using the "HiveFromSpark" example to test some changes and I ran into an issue that manifests itself as an NPE inside Hive code because some configuration object is null. Tracing back, it seems that `sessionState` being a lazy val in HiveContext is causing it. That variably is

Re: best IDE for scala + spark development?

I'll chime in as yet another user who is extremely happy with sbt and a text editor. (In my experience, running "ack" from the command line is usually just as easy and fast as using an IDE's find-in-project facility.) You can, of course, extend editors with Scala-specific IDE-like functionalit

Re: jenkins emergency restart now, was Re: jenkins downtime tomorrow morning ~6am-8am PDT

ok we're back up and building. i've retriggered the jobs i killed. On Mon, Oct 27, 2014 at 1:24 PM, shane knapp wrote: > so, i'm having a race condition between a plugin i installed putting > jenkins in to quiet mode and it failing to perform a backup from this past > weekend. i'll need to res

jenkins emergency restart now, was Re: jenkins downtime tomorrow morning ~6am-8am PDT

so, i'm having a race condition between a plugin i installed putting jenkins in to quiet mode and it failing to perform a backup from this past weekend. i'll need to restart the process and get it out of the constantly-in-to-quiet-mode cycle it's in now. this will be quick, and i'll restart the j

jenkins downtime tomorrow morning ~6am-8am PDT

i'll be bringing jenkins down tomorrow morning for some system maintenance and to get our backups kicked off. i do expect to have the system back up and running before 8am. please let me know ASAP if i need to reschedule this. thanks, shane

Re: best IDE for scala + spark development?

Also ctags works fine with vim for browsing scala classes Shivaram On Mon, Oct 27, 2014 at 8:49 AM, Koert Kuipers wrote: > editor of your choice + sbt console works + grep great. > > if only folks stopped using wildcard imports (it has little benefits in > terms of coding yet requires an IDE wit

Re: best IDE for scala + spark development?

editor of your choice + sbt console works + grep great. if only folks stopped using wildcard imports (it has little benefits in terms of coding yet requires an IDE with 1G+ of ram to track em down). On Mon, Oct 27, 2014 at 9:17 AM, andy petrella wrote: > I second the S[B]T combo! > > I tried AT

Re: best IDE for scala + spark development?

I second the S[B]T combo! I tried ATOM → lack of features and stability (atm) aℕdy ℙetrella about.me/noootsab [image: aℕdy ℙetrella on about.me] On Mon, Oct 27, 2014 at 2:15 PM, Dean Wampler wrote: > For what it's worth, I use Sublime Text + the SBT console for ever

Re: best IDE for scala + spark development?

For what it's worth, I use Sublime Text + the SBT console for everything. I can live without the extra IDE features. However, if you like an IDE, the Eclipse "Scala IDE" 4.0 RC1 is a big improvement over previous releases. For one thing, it can now supports projects using different versions of Sca

Re: Build with Hive 0.13.1 doesn't have datanucleus and parquet dependencies.

Ah I see. Thanks Hao! I'll wait for the fix. Jianshi On Mon, Oct 27, 2014 at 4:57 PM, Cheng, Hao wrote: > Hive-thriftserver module is not included while specifying the profile > hive-0.13.1. > > -Original Message- > From: Jianshi Huang [mailto:jianshi.hu...@gmail.com] > Sent: Monday, Oc

RE: Build with Hive 0.13.1 doesn't have datanucleus and parquet dependencies.

Hive-thriftserver module is not included while specifying the profile hive-0.13.1. -Original Message- From: Jianshi Huang [mailto:jianshi.hu...@gmail.com] Sent: Monday, October 27, 2014 4:48 PM To: dev@spark.apache.org Subject: Build with Hive 0.13.1 doesn't have datanucleus and parquet

Build with Hive 0.13.1 doesn't have datanucleus and parquet dependencies.

There's a change in build process lately for Hive 0.13 support and we should make it obvious. Based on the new pom.xml I tried to enable Hive 0.13.1 support by using option -Phive-0.13.1 However, it seems datanucleus and parquet dependencies are not available in the final build. Am I missing a