Re: SparkSQL errors in 1.4 rc when using with Hive 0.12 metastore

2015-05-24 Thread Cheolsoo Park
Thank you Hao for the confirmation! I filed two jiras as follows- https://issues.apache.org/jira/browse/SPARK-7850 (removing hive-0.12.0 profile from pom) https://issues.apache.org/jira/browse/SPARK-7851 (thrift error with hive metastore 0.12) On Sun, May 24, 2015 at 8:18 PM, Cheng, Hao wrote:

RE: SparkSQL errors in 1.4 rc when using with Hive 0.12 metastore

2015-05-24 Thread Cheng, Hao
Thanks for reporting this. We intend to support the multiple metastore versions in a single build(hive-0.13.1) by introducing the IsolatedClientLoader, but probably you’re hitting the bug, please file a jira issue for this. I will keep investigating on this also. Hao From: Mark Hamstra [mail

Re: [VOTE] Release Apache Spark 1.4.0 (RC2)

2015-05-24 Thread Krishna Sankar
+1 (non-binding, of course) 1. Compiled OSX 10.10 (Yosemite) OK Total time: 16:52 min mvn clean package -Pyarn -Dyarn.version=2.6.0 -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests 2. Tested pyspark, mlib - running as well as compare results with 1.3.1 2.1. statistics (min,max,mean,Pearson,Spe

Re: Kryo option changed

2015-05-24 Thread Ted Yu
The original PR from Liye didn't include test which exercises Kryo buffer size configured in mb which is below 2GB. In my PR, I added such a test and it passed on Jenkins: https://github.com/apache/spark/pull/6390 FYI On Sun, May 24, 2015 at 8:08 AM, Ted Yu wrote: > Please update to the follow

Re: Kryo option changed

2015-05-24 Thread Ted Yu
Please update to the following: commit c2f0821aad3b82dcd327e914c9b297e92526649d Author: Zhang, Liye Date: Fri May 8 09:10:58 2015 +0100 [SPARK-7392] [CORE] bugfix: Kryo buffer size cannot be larger than 2M On Sun, May 24, 2015 at 8:04 AM, Debasish Das wrote: > I am May 3rd commit: > > c

Re: Kryo option changed

2015-05-24 Thread Debasish Das
I am May 3rd commit: commit 49549d5a1a867c3ba25f5e4aec351d4102444bc0 Author: WangTaoTheTonic Date: Sun May 3 00:47:47 2015 +0100 [SPARK-7031] [THRIFTSERVER] let thrift server take SPARK_DAEMON_MEMORY and SPARK_DAEMON_JAVA_OPTS On Sat, May 23, 2015 at 7:54 PM, Josh Rosen wrote: > Whic

Re: Kryo option changed

2015-05-24 Thread Sean Owen
Ah right I misread this. I get it but I dont think the PR fixes this. Let me comment there. On May 24, 2015 3:56 PM, "Sean Owen" wrote: > Wait, isn't the error message just saying you can't set 8mb buffers? So it > is correctly parsing the args. I don't understand why this has to do with > parsin

Re: Kryo option changed

2015-05-24 Thread Sean Owen
Wait, isn't the error message just saying you can't set 8mb buffers? So it is correctly parsing the args. I don't understand why this has to do with parsing the value. That much works. On May 24, 2015 2:04 AM, "Debasish Das" wrote: > Hi, > > I am on last week's master but all the examples that se

Re: spark packages

2015-05-24 Thread Debasish Das
Yup netlib lgpl right now is activated through a profile...if we can reuse the same idea then csparse can also be added to spark with a lgpl flag. But again as Sean said its tricky. Better to keep it on spark packages for users to try. On May 24, 2015 1:36 AM, "Sean Owen" wrote: > I dont believe

Re: SparkSQL errors in 1.4 rc when using with Hive 0.12 metastore

2015-05-24 Thread Mark Hamstra
This discussion belongs on the dev list. Please post any replies there. On Sat, May 23, 2015 at 10:19 PM, Cheolsoo Park wrote: > Hi, > > I've been testing SparkSQL in 1.4 rc and found two issues. I wanted to > confirm whether these are bugs or not before opening a jira. > > *1)* I can no longer

Re: spark packages

2015-05-24 Thread Sean Owen
I dont believe we are talking about adding things to the Apache project, but incidentally LGPL is not OK in Apache projects either. On May 24, 2015 6:12 AM, "DB Tsai" wrote: > I thought LGPL is okay but GPL is not okay for Apache project. > > On Saturday, May 23, 2015, Patrick Wendell wrote: > >

Re: Spark Streaming - Design considerations/Knobs

2015-05-24 Thread Tathagata Das
Blocks are replicated immediately, before the driver launches any jobs using them. On Thu, May 21, 2015 at 2:05 AM, Hemant Bhanawat wrote: > Honestly, given the length of my email, I didn't expect a reply. :-) > Thanks for reading and replying. However, I have a follow-up question: > > I don't t

[ANNOUNCE] Nightly maven and package builds for Spark

2015-05-24 Thread Patrick Wendell
Hi All, This week I got around to setting up nightly builds for Spark on Jenkins. I'd like feedback on these and if it's going well I can merge the relevant automation scripts into Spark mainline and document it on the website. Right now I'm doing: 1. SNAPSHOT's of Spark master and release branch

[VOTE] Release Apache Spark 1.4.0 (RC2)

2015-05-24 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.0! The tag to be voted on is v1.4.0-rc2 (commit 03fb26a3): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=03fb26a3e50e00739cc815ba4e2e82d71d003168 The release files, including signatures, digests, etc. ca

Re: [VOTE] Release Apache Spark 1.4.0 (RC1)

2015-05-24 Thread Patrick Wendell
Hey jameszhouyi, Since SPARK-7119 is not a regression from earlier versions, we won't hold the release for it. However, please comment on the JIRA if it is affecting you... it will help us prioritize the bug. - Patrick On Fri, May 22, 2015 at 8:41 PM, jameszhouyi wrote: > We came across a Spark

[RESULT] [VOTE] Release Apache Spark 1.4.0 (RC1)

2015-05-24 Thread Patrick Wendell
This vote is cancelled in favor of RC2. On Tue, May 19, 2015 at 9:10 AM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.0! > > The tag to be voted on is v1.4.0-rc1 (commit 777a081): > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=c