Re: Updating

perfect, thanks Sean! On Thu, Apr 13, 2017 at 4:21 PM, Sean Owen wrote: > The site source is at https://github.com/apache/spark-website/ > > On Thu, Apr 13, 2017 at 4:20 PM Sam Elamin > wrote: > >> Hey all >> >> Who do I need to talk to in order to update the Useful developer tools >> page

Re: Updating

The site source is at https://github.com/apache/spark-website/ On Thu, Apr 13, 2017 at 4:20 PM Sam Elamin wrote: > Hey all > > Who do I need to talk to in order to update the Useful developer tools > page ? > > I want to update the build instructions

Re: Updating Parquet dep to 1.9

The stats problem is on the write side. Parquet compares byte buffers (used for UTF8 strings also) using byte-wise comparison, but got it wrong and compares the Java byte values, which are signed. UTF8 ordering is the same as byte-wise comparison, but only if the bytes are compared as unsigned valu

Re: Updating Parquet dep to 1.9

Sounds great. Regarding the min/max stats issue, is that an issue with the way the files are written or read? What's the Parquet project issue for that bug? What's the 1.9.1 release timeline look like? I will aim to have a PR in by the end of the week. I feel strongly that either this or https:

Re: Updating Parquet dep to 1.9

I can when I'm finished with a couple other issues if no one gets to it first. Michael, if you're interested in updating to 1.9.0 I'm happy to help review that PR. On Tue, Nov 1, 2016 at 1:03 PM, Reynold Xin wrote: > Ryan want to submit a pull request? > > > On Tue, Nov 1, 2016 at 9:05 AM, Ryan

Re: Updating Parquet dep to 1.9

Ryan want to submit a pull request? On Tue, Nov 1, 2016 at 9:05 AM, Ryan Blue wrote: > 1.9.0 includes some fixes intended specifically for Spark: > > * PARQUET-389: Evaluates push-down predicates for missing columns as > though they are null. This is to address Spark's work-around that requires

Re: Updating Parquet dep to 1.9

1.9.0 includes some fixes intended specifically for Spark: * PARQUET-389: Evaluates push-down predicates for missing columns as though they are null. This is to address Spark's work-around that requires reading and merging file schemas, even for metastore tables. * PARQUET-654: Adds an option to d

Re: Updating Parquet dep to 1.9

Yes this came up from a different direction: https://issues.apache.org/jira/browse/SPARK-18140 I think it's fine to pursue an upgrade to fix these several issues. The question is just how well it will play with other components, so bears some testing and evaluation of the changes from 1.8, but yes

Re: Updating Spark PR builder and 2.x test jobs to use Java 8 JDK

I finally figured out the problem: it seems that my *export JAVA_HOME=/path/to/java8/home* was somehow not affecting the javac executable that Zinc's SBT incremental compiler uses when it forks out to javac to handle Java source files. As a result, we were passing a -source 1.8 flag to the platform

Re: Updating Spark PR builder and 2.x test jobs to use Java 8 JDK

I've reverted the bulk of the conf changes while I investigate. I think that Zinc might be handling JAVA_HOME in a weird way and am SSH'ing to Jenkins to try to reproduce the problem in isolation. On Tue, Apr 5, 2016 at 4:14 PM Ted Yu wrote: > Josh: > You may have noticed the following error ( >

Re: Updating Spark PR builder and 2.x test jobs to use Java 8 JDK

Josh: You may have noticed the following error ( https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/566/console ): [error] javac: invalid source release: 1.8 [error] Usage: javac [error] use -help for a list of possible options On Tue, Apr 5, 2016 at 2:14 PM, Josh Ro

Re: Updating docs for running on Mesos

Perhaps linking to a Mesos page, which then can list the various package incantations. Cheers, Tim - Original Message - > From: "Matei Zaharia" > To: dev@spark.apache.org > Sent: Tuesday, May 13, 2014 2:59:42 AM > Subject: Re: Updating docs for running on Mesos &

Re: Updating docs for running on Mesos

Great work!. I just left some comments in the PR. In summary, it would be great to have more background on how Spark works on Mesos and how the different elements interact. That will (hopefully) help understanding the practicalities of the common assembly location (http/hdfs) and how the jobs are d

Re: Updating docs for running on Mesos

Andrew, Mesosphere has binary releases here: http://mesosphere.io/downloads/ (Anecdote: I actually burned a CPU building Mesos from source. No kidding - it was coming, as the laptop was crashing from time to time, but the mesos build was that one drop too much) kr, Gerard. On Tue, May 13, 201

Re: Updating docs for running on Mesos

Completely agree about preferring to link to the upstream project rather than a company's -- the only reason I'm using mesosphere's now is that I see no alternative from mesos.apache.org I included instructions for both using Mesosphere's packages and building from scratch in the PR: https://githu

Re: Updating docs for running on Mesos

I’ll ask the Mesos folks about this. Unfortunately it might be tough to link only to a company’s builds; but we can perhaps include them in addition to instructions for building Mesos from Apache. Matei On May 12, 2014, at 11:55 PM, Gerard Maas wrote: > Andrew, > > Mesosphere has binary rele

Re: Updating docs for running on Mesos

I have a draft of my proposed changes here: https://github.com/apache/spark/pull/756 https://issues.apache.org/jira/browse/SPARK-1818 Thanks! Andrew On Mon, May 12, 2014 at 9:57 PM, Andrew Ash wrote: > As far as I know, the upstream doesn't release binaries, only source code. > The downloads

Re: Updating docs for running on Mesos

As far as I know, the upstream doesn't release binaries, only source code. The downloads page for 0.18.0 only has a source tarball. Is there a binary release somewhere from Mesos that I'm missing? On Sun, May 11, 2014 at 2:16 PM, Patrick Wendell wrote: >

Re: Updating docs for running on Mesos

For trimming the Running Alongside Hadoop section I mostly think there should be a separate Spark+HDFS section and have the CDH+HDP page be merged into that one, but I supposed that's a separate docs change. On Sun, May 11, 2014 at 4:28 PM, Andy Konwinski wrote: > Thanks for suggesting this and

Re: Updating docs for running on Mesos

Andrew, Updating these docs would be great! I think this would be a welcome change. In terms of packaging, it would be good to mention the binaries produced by the upstream project as well, in addition to Mesosphere. - Patrick On Thu, May 8, 2014 at 12:51 AM, Andrew Ash wrote: > The docs for h

Re: Updating docs for running on Mesos

Thanks for suggesting this and volunteering to do it. On May 11, 2014 3:32 AM, "Andrew Ash" wrote: > > The docs for how to run Spark on Mesos have changed very little since > 0.6.0, but setting it up is much easier now than then. Does it make sense > to revamp with the below changes? > > > You n