Re: Error executing sql using shark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1

my spark can also work well with hadoop2.2.0 my shark can not work well with hadoop2.2.0 because protobuf version problem. in shark direcotry , i found two vesions of protobuf, and they all are loaded into classpath. [root@bigdata001 shark-0.9.0]# find . -name "proto*.jar" ./lib_managed/jars/or

Re: Error executing sql using shark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1

my spark can also work well with hadoop2.2.0 my shark can not work well with hadoop2.2.0 because protobuf version problem. in shark direcotry , i found two vesions of protobuf, and they all are loaded into classpath. [root@bigdata001 shark-0.9.0]# find . -name "proto*.jar" ./lib_managed/jars/or

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

Cool. It sounds like focusing on sbt-pom-reader would be a good thing for you guys then. There's a few... fun... issues around maven parent projects that are still running around with sbt-pom-reader that appear to be fundamental ivy-maven hate-based issues. IN any case, while I'm generally swampe

Re: Spark 0.9.1 release

Updates: 1. Fix for the ASM problemthat Kevin mentioned is already in Spark 0.9.1 RC2 2. Fix for pyspark's RDD.top() that Patrick mentioned has been pulled into branch 0.9. This will get int

Fwd: Apache projects based Business

Hi community, If there are things that I need to be aware of, please give me a heads up. -- Forwarded message -- From: Edward J. Yoon Date: Wed, Mar 26, 2014 at 11:13 AM Subject: Apache projects based Business To: legal-disc...@apache.org Cc: Commons Developers List Hi, I rec

Re: Error reading HDFS file using spark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1

@qingyang, spark 0.9.0 works for me perfectly when accessing (read/write) data on hdfs. BTW, if you look at pom.xml, you have to choose yarn profile to compile spark, so that it won't include protobuf 2.4.1 in your final jars. Here is the command line we use to compile spark with hadoop 2.2: mvn -

Re: Spark 0.9.1 release

Hey TD, This one we just merged into master this morning: https://spark-project.atlassian.net/browse/SPARK-1322 It should definitely go into the 0.9 branch because there was a bug in the semantics of top() which at this point is unreleased in Python. I didn't backport it yet because I figured yo

Re: ALS memory limits

Just a clarification, I am using Spark ALS explicit feedback on standalone cluster without deploying zookeeper master HA option yet... When in the standalone spark cluster, worker fails due to GC error, the worker dies as well and I have to restart the workerUnderstanding this issue will be us

Re: ALS memory limits

Thanks Sean. Looking into executor memory options now... I am at incubator_spark head. Does that has all the fixes or I need spark head ? I can deploy the spark head as well... I am not running implicit feedback yet...I remember memory enhancements were mainly for implicit right ? For ulimit let

Re: [VOTE] Release Apache Spark 0.9.1 (rc1)

This voting thread has been canceled to accommodate an important fix regarding Spark's use of the ASM library. Refer to https://spark-project.atlassian.net/browse/SPARK-782 for more details. TD On Tue, Mar 25, 2014 at 10:36 PM, Matei Zaharia wrote: > Actually I found one minor issue, which is t

Re: ALS memory limits

Much of this sounds related to the memory issue mentioned earlier in this thread. Are you using a build that has fixed that? That would be by far most important here. If the raw memory requirement is 8GB, the actual heap size necessary could be a lot larger -- object overhead, all the other stuff

Error reading HDFS file using spark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1

Egor, i encounter the same problem which you have asked in this thread: http://mail-archives.apache.org/mod_mbox/spark-user/201402.mbox/%3CCAMrx5DwJVJS0g_FE7_2qwMu4Xf0y5VfV=tlyauv2kh5v4k6...@mail.gmail.com%3E have you fixed this problem? i am using shark to read a table which i have created on h