Sure, I'll take a look. Planning to do full verification in a bit. On Wed, Apr 6, 2016 at 12:54 PM Ted Yu <yuzhih...@gmail.com> wrote:
> Josh: > Can you check spark-1.6.1-bin-hadoop2.4.tgz ? > > $ tar zxf spark-1.6.1-bin-hadoop2.4.tgz > > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error is not recoverable: exiting now > > $ ls -l !$ > ls -l spark-1.6.1-bin-hadoop2.4.tgz > -rw-r--r--. 1 hbase hadoop 323614720 Apr 5 19:25 > spark-1.6.1-bin-hadoop2.4.tgz > > Thanks > > On Wed, Apr 6, 2016 at 12:19 PM, Josh Rosen <joshro...@databricks.com> > wrote: > >> I downloaded the Spark 1.6.1 artifacts from the Apache mirror network and >> re-uploaded them to the spark-related-packages S3 bucket, so hopefully >> these packages should be fixed now. >> >> On Mon, Apr 4, 2016 at 3:37 PM Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >>> Thanks, that was the command. :thumbsup: >>> >>> On Mon, Apr 4, 2016 at 6:28 PM Jakob Odersky <ja...@odersky.com> wrote: >>> >>>> I just found out how the hash is calculated: >>>> >>>> gpg --print-md sha512 <spark-archive>.tgz >>>> >>>> you can use that to check if the resulting output matches the contents >>>> of <spark-archive>.tgz.sha >>>> >>>> On Mon, Apr 4, 2016 at 3:19 PM, Jakob Odersky <ja...@odersky.com> >>>> wrote: >>>> > The published hash is a SHA512. >>>> > >>>> > You can verify the integrity of the packages by running `sha512sum` on >>>> > the archive and comparing the computed hash with the published one. >>>> > Unfortunately however, I don't know what tool is used to generate the >>>> > hash and I can't reproduce the format, so I ended up manually >>>> > comparing the hashes. >>>> > >>>> > On Mon, Apr 4, 2016 at 2:39 PM, Nicholas Chammas >>>> > <nicholas.cham...@gmail.com> wrote: >>>> >> An additional note: The Spark packages being served off of >>>> CloudFront (i.e. >>>> >> the “direct download” option on spark.apache.org) are also corrupt. >>>> >> >>>> >> Btw what’s the correct way to verify the SHA of a Spark package? >>>> I’ve tried >>>> >> a few commands on working packages downloaded from Apache mirrors, >>>> but I >>>> >> can’t seem to reproduce the published SHA for >>>> spark-1.6.1-bin-hadoop2.6.tgz. >>>> >> >>>> >> >>>> >> On Mon, Apr 4, 2016 at 11:45 AM Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>> >>>> >>> Maybe temporarily take out the artifacts on S3 before the root >>>> cause is >>>> >>> found. >>>> >>> >>>> >>> On Thu, Mar 24, 2016 at 7:25 AM, Nicholas Chammas >>>> >>> <nicholas.cham...@gmail.com> wrote: >>>> >>>> >>>> >>>> Just checking in on this again as the builds on S3 are still >>>> broken. :/ >>>> >>>> >>>> >>>> Could it have something to do with us moving release-build.sh? >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Mar 21, 2016 at 1:43 PM Nicholas Chammas >>>> >>>> <nicholas.cham...@gmail.com> wrote: >>>> >>>>> >>>> >>>>> Is someone going to retry fixing these packages? It's still a >>>> problem. >>>> >>>>> >>>> >>>>> Also, it would be good to understand why this is happening. >>>> >>>>> >>>> >>>>> On Fri, Mar 18, 2016 at 6:49 PM Jakob Odersky <ja...@odersky.com> >>>> wrote: >>>> >>>>>> >>>> >>>>>> I just realized you're using a different download site. Sorry >>>> for the >>>> >>>>>> confusion, the link I get for a direct download of Spark 1.6.1 / >>>> >>>>>> Hadoop 2.6 is >>>> >>>>>> >>>> http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1-bin-hadoop2.6.tgz >>>> >>>>>> >>>> >>>>>> On Fri, Mar 18, 2016 at 3:20 PM, Nicholas Chammas >>>> >>>>>> <nicholas.cham...@gmail.com> wrote: >>>> >>>>>> > I just retried the Spark 1.6.1 / Hadoop 2.6 download and got a >>>> >>>>>> > corrupt ZIP >>>> >>>>>> > file. >>>> >>>>>> > >>>> >>>>>> > Jakob, are you sure the ZIP unpacks correctly for you? Is it >>>> the same >>>> >>>>>> > Spark >>>> >>>>>> > 1.6.1/Hadoop 2.6 package you had a success with? >>>> >>>>>> > >>>> >>>>>> > On Fri, Mar 18, 2016 at 6:11 PM Jakob Odersky < >>>> ja...@odersky.com> >>>> >>>>>> > wrote: >>>> >>>>>> >> >>>> >>>>>> >> I just experienced the issue, however retrying the download a >>>> second >>>> >>>>>> >> time worked. Could it be that there is some load >>>> balancer/cache in >>>> >>>>>> >> front of the archive and some nodes still serve the corrupt >>>> >>>>>> >> packages? >>>> >>>>>> >> >>>> >>>>>> >> On Fri, Mar 18, 2016 at 8:00 AM, Nicholas Chammas >>>> >>>>>> >> <nicholas.cham...@gmail.com> wrote: >>>> >>>>>> >> > I'm seeing the same. :( >>>> >>>>>> >> > >>>> >>>>>> >> > On Fri, Mar 18, 2016 at 10:57 AM Ted Yu < >>>> yuzhih...@gmail.com> >>>> >>>>>> >> > wrote: >>>> >>>>>> >> >> >>>> >>>>>> >> >> I tried again this morning : >>>> >>>>>> >> >> >>>> >>>>>> >> >> $ wget >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz >>>> >>>>>> >> >> --2016-03-18 07:55:30-- >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz >>>> >>>>>> >> >> Resolving s3.amazonaws.com... 54.231.19.163 >>>> >>>>>> >> >> ... >>>> >>>>>> >> >> $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz >>>> >>>>>> >> >> >>>> >>>>>> >> >> gzip: stdin: unexpected end of file >>>> >>>>>> >> >> tar: Unexpected EOF in archive >>>> >>>>>> >> >> tar: Unexpected EOF in archive >>>> >>>>>> >> >> tar: Error is not recoverable: exiting now >>>> >>>>>> >> >> >>>> >>>>>> >> >> On Thu, Mar 17, 2016 at 8:57 AM, Michael Armbrust >>>> >>>>>> >> >> <mich...@databricks.com> >>>> >>>>>> >> >> wrote: >>>> >>>>>> >> >>> >>>> >>>>>> >> >>> Patrick reuploaded the artifacts, so it should be fixed >>>> now. >>>> >>>>>> >> >>> >>>> >>>>>> >> >>> On Mar 16, 2016 5:48 PM, "Nicholas Chammas" >>>> >>>>>> >> >>> <nicholas.cham...@gmail.com> >>>> >>>>>> >> >>> wrote: >>>> >>>>>> >> >>>> >>>> >>>>>> >> >>>> Looks like the other packages may also be corrupt. I’m >>>> getting >>>> >>>>>> >> >>>> the >>>> >>>>>> >> >>>> same >>>> >>>>>> >> >>>> error for the Spark 1.6.1 / Hadoop 2.4 package. >>>> >>>>>> >> >>>> >>>> >>>>>> >> >>>> >>>> >>>>>> >> >>>> >>>> >>>>>> >> >>>> >>>> >>>>>> >> >>>> >>>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.4.tgz >>>> >>>>>> >> >>>> >>>> >>>>>> >> >>>> Nick >>>> >>>>>> >> >>>> >>>> >>>>>> >> >>>> >>>> >>>>>> >> >>>> On Wed, Mar 16, 2016 at 8:28 PM Ted Yu < >>>> yuzhih...@gmail.com> >>>> >>>>>> >> >>>> wrote: >>>> >>>>>> >> >>>>> >>>> >>>>>> >> >>>>> On Linux, I got: >>>> >>>>>> >> >>>>> >>>> >>>>>> >> >>>>> $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz >>>> >>>>>> >> >>>>> >>>> >>>>>> >> >>>>> gzip: stdin: unexpected end of file >>>> >>>>>> >> >>>>> tar: Unexpected EOF in archive >>>> >>>>>> >> >>>>> tar: Unexpected EOF in archive >>>> >>>>>> >> >>>>> tar: Error is not recoverable: exiting now >>>> >>>>>> >> >>>>> >>>> >>>>>> >> >>>>> On Wed, Mar 16, 2016 at 5:15 PM, Nicholas Chammas >>>> >>>>>> >> >>>>> <nicholas.cham...@gmail.com> wrote: >>>> >>>>>> >> >>>>>> >>>> >>>>>> >> >>>>>> >>>> >>>>>> >> >>>>>> >>>> >>>>>> >> >>>>>> >>>> >>>>>> >> >>>>>> >>>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz >>>> >>>>>> >> >>>>>> >>>> >>>>>> >> >>>>>> Does anyone else have trouble unzipping this? How did >>>> this >>>> >>>>>> >> >>>>>> happen? >>>> >>>>>> >> >>>>>> >>>> >>>>>> >> >>>>>> What I get is: >>>> >>>>>> >> >>>>>> >>>> >>>>>> >> >>>>>> $ gzip -t spark-1.6.1-bin-hadoop2.6.tgz >>>> >>>>>> >> >>>>>> gzip: spark-1.6.1-bin-hadoop2.6.tgz: unexpected end of >>>> file >>>> >>>>>> >> >>>>>> gzip: spark-1.6.1-bin-hadoop2.6.tgz: uncompress failed >>>> >>>>>> >> >>>>>> >>>> >>>>>> >> >>>>>> Seems like a strange type of problem to come across. >>>> >>>>>> >> >>>>>> >>>> >>>>>> >> >>>>>> Nick >>>> >>>>>> >> >>>>> >>>> >>>>>> >> >>>>> >>>> >>>>>> >> >> >>>> >>>>>> >> > >>>> >>> >>>> >>> >>>> >> >>>> >>> >
