Re: unable to extract tgz files downloaded from spark

>From which site did you download the tar ball ? Which package type did you choose (pre-built for which distro) ? Thanks On Wed, May 6, 2015 at 7:16 PM, Praveen Kumar Muthuswamy < muthusamy...@gmail.com> wrote: > Hi > I have been trying to install latest spark verison and downloaded the .tgz >

unable to extract tgz files downloaded from spark

Hi I have been trying to install latest spark verison and downloaded the .tgz files(ex spark-1.3.1.tgz). But, I could not extract them. It complains of invalid tar format. Has any seen this issue ? Thanks Praveen

Re: Recent Spark test failures

Thanks for doing this. Testing infra is one of the most important parts of a project, and this will make it easier to identify flaky tests. On Wed, May 6, 2015 at 5:41 PM, Andrew Or wrote: > Dear all, > > I'm sure you have all noticed that the Spark tests have been fairly > unstable recently. I

[build system] quick jenkins restart thursday morning (5-6-15) 7am PDT

we've had a spate of issues since the power outage, and now the github pull request builder is randomly deciding who can and can't trigger builds[1]. i think it's time for a quick restart of the master and workers, which i'll do early tomorrow morning. the outage should be very brief, and i'll le

Recent Spark test failures

Dear all, I'm sure you have all noticed that the Spark tests have been fairly unstable recently. I wanted to share a tool that I use to track which tests have been failing most often in order to prioritize fixing these flaky tests. Here is an output of the tool. This spreadsheet reports the top 1

Re: Pull request builder errors (taking Jenkins worker 3 offline)

ok, things are looking good... i'll definitely be keeping an eye on this worker, but it looks like the ivy cache somehow got poisoned and affected the builds. it's still not clear to me *how* this happened, but *what* happened is clear: the ivydata properties file for oro is-local=true for the j

Re: Spark/Mesos

Hi Sam, Spark cluster mode merge should also include documentation update that has details what it is, but in a nutshell it's basically supporting launching drivers that is managed in your cluster instead of launching it yourself via client mode. YARN and Standalone both supports cluster mode, so

Re: Spark/Mesos

So, to confirm - in this mode, when a Spark application/context runs a series of tasks, each task will launch a full SparkExecutor process? What is the cpu/mem cost of such Spark Executor process (resource sizing passed in the Mesos task launch request)?

Re: kryo version?

They are usually pretty responsive. We can ping chill to get them to do a release. On Wed, May 6, 2015 at 10:32 AM, Tom Graves wrote: > Hey folks, > I had a customer ask about updating the version of kryo to get fix: > https://github.com/EsotericSoftware/kryo/pull/164 which is in 2.23.Spark > c

kryo version?

Hey folks, I had a customer ask about updating the version of kryo to get fix:  https://github.com/EsotericSoftware/kryo/pull/164 which is in 2.23.Spark currently pull sin chill 0.5.0 which pulls in kryo 2.21.  I don't see a newer version of chill that has updated to kryo 2.23.   Anyone familiar

Re: Pull request builder errors (taking Jenkins worker 3 offline)

ok, i looked deeper and this is only happening on -03, and not linked specifically to the pull request builder: 3 NewSparkPullRequestBuilder 13 Spark-Master-SBT 4 Spark-1.4-SBT 49 SparkPullRequestBuilder also, it started at ~3pm on this past sunday... and nothing was done to

Re: jackson.databind exception in RDDOperationScope.jsonMapper.writeValueAsString(this)

Looks like mismatch of jackson version. Spark uses: 2.4.4 FYI On Wed, May 6, 2015 at 8:00 AM, A.M.Chan wrote: > Hey, guys. I meet this exception while testing SQL/Columns. > I didn't change the pom or the core project. > In the morning, it's fine to test my PR. > I don't know what happed. >

jackson.databind exception in RDDOperationScope.jsonMapper.writeValueAsString(this)

Hey, gays. I meet this exception while testing SQL/Columns. I didn't change the pom or the core project. In the morning, it's fine to test my PR. I don't know what happed. An exception or error caused a run to abort: com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.addField(Lcom/fas

Re: [discuss] ending support for Java 6?

That is correct. I plan to try it out and review it today. Tom On Wednesday, May 6, 2015 1:48 AM, Reynold Xin wrote: @tgraves can chime in, but I think this pr aims to fix it:  https://github.com/apache/spark/pull/5580 We should probably get that in for 1.4. On Tue, May 5, 2015 at 1

Re: Spark/Mesos

Hi Tim. Just a follow up, more related to your work on the rencently merged Spark Cluster Mode for Mesos. Can you elaborate how it works compared to the Standalone mode. and do you maintain the dyanamic allocation of mesos resources in the cluster mode unlike the coarse grained mode? On Tue, May 5

Re: Spark/Mesos

Thanks Tim, a few follow-up questions using the Mesos|Spark prefixing - > 2. In fine grained mode, what happens is that Spark scheduler > specifies a custom Mesos executor per slave, and each Mesos task is a > Spark executor that will be launched by the Mesos executor. It's hard > to determine

Re: Creating topology in spark streaming

Thanks alot Juan, That was a great post, One more thing if u can .Any there any demo/blog telling how to configure or create a topology of different types .. i mean how we can decide the pipelining model in spark as done in storm for https://storm.apache.org/documentation/images/topology.p

Re: Creating topology in spark streaming

Hi, You can use the method repartition from DStream (for the Scala API) or JavaDStream (for the Java API) defrepartition(numPartitions: Int): DStream [T] Return a new DStream with an increased or dec

Re: Creating topology in spark streaming

But main problem is how to increase the level of parallelism for any particular bolt logic . suppose i want this type of topology . https://storm.apache.org/documentation/images/topology.png How we can manage it . On Wed, May 6, 2015 at 1:36 PM, ayan guha wrote: > Every transformation on a

Creating topology in spark streaming

Please help guys, Even After going through all the examples given i have not understood how to pass the D-streams from one bolt/logic to other (without writing it on HDFS etc.) just like emit function in storm . Suppose i have topology with 3 bolts(say) *BOLT1(parse the tweets nd emit tweet u