That is a great observation. And I missed your previous email about the shaded
vs unshaded jars already getting fixed.
I guess we are good to go.
--------------------------------------------------------------------------------------------------------------------------
Looking at the RC. Went through my usual check-list. Here's my summary.
Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
-- Note: The format of the mds files changed a bit - not a biggie.
-- For e.g, in 3.0.0 and 2.x releases, it has lines of the form
"hadoop-3.0.0-src.tar.gz: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279 7BCED50B
B2600318 B7E0BE3A 74DFFF71"
-- But in 3.1.0 RC it is,
"/build/source/target/artifacts/hadoop-3.1.0.tar.gz: SHA256 = 670D2CED 595FA42D
9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
- [Check] Generating dist tarballs from source tar-ball
- [Check] Testing
-- Start NN, DN, RM, NM, JHS, Timeline Service
-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
-- Tested CLIs to print nodes, apps etc and also navigated UIs
+1 binding.
Thanks
+Vinod
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wheele...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
> wrote:
>
>> We vote on the source code. The binaries are convenience artifacts.
>>
>> This is what I would do - (a) Just replace both the maven jars as well as
>> the binaries to be consistent and correct. And then (b) Give a couple more
>> days for folks who tested on the binaries to reverify - I count one such
>> clear vote as of now.
>>
>> Thanks
>> +Vinod
>>
>>
>> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
>>
>> HI Arpit,
>>
>> I think it won't match if we do rebuild. It should be fine as far as
>> they're signed, correct? I don't see any policy doesn't allow this.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
>> wrote:
>>
>>> Thanks Wangda, I see the shaded jars now.
>>>
>>> Are the repo jars required to be the same as the binary release? They
>>> don’t match right now, probably they got rebuilt.
>>>
>>> +1 (binding), modulo that remaining question.
>>>
>>> * Verified signatures
>>> * Verified checksums for source and binary artefacts
>>> * Sanity checked jars on r.a.o.
>>> * Built from source
>>> * Deployed to 3 node secure cluster with NameNode HA
>>> * Verified HDFS web UIs
>>> * Tried out HDFS shell commands
>>> * Ran sample MapReduce jobs
>>>
>>> Thanks!
>>>
>>>
>>> ----------------------------------------------------------------------
>>> From: Wangda Tan <wheele...@gmail.com>
>>> Date: Monday, April 2, 2018 at 9:25 PM
>>> To: Arpit Agarwal <aagar...@hortonworks.com>
>>> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
>>> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
>>> hdfs-dev@hadoop.apache.org>, Hadoop Common <common-...@hadoop.apache.org>,
>>> "mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>,
>>> Vinod Kumar Vavilapalli <vino...@apache.org>
>>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>>
>>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>>> Just redeployed jars and staged. @Arpit, could you please check the updated
>>> Maven repo? https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1092
>>>
>>> Since the jars inside binary tarballs are correct (
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>>> need roll another RC, just update Maven repo should be sufficient.
>>>
>>> Best,
>>> Wangda
>>>
>>>
>>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com>
>>> wrote:
>>> Hi Arpit,
>>>
>>> Thanks for pointing out this.
>>>
>>> I just removed all .md5 files from artifacts. I found md5 checksums still
>>> exist in .mds files and I didn't remove them from .mds file because it is
>>> generated by create-release script and Apache guidance is "should not"
>>> instead of "must not". Please let me know if you think they need to be
>>> removed as well.
>>>
>>> - Wangda
>>>
>>>
>>>
>>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
>>> aagar...@hortonworks.com> wrote:
>>> Thanks for putting together this RC, Wangda.
>>>
>>> The guidance from Apache is to omit MD5s, specifically:
>>>> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>>
>>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>>
>>>
>>>
>>>
>>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com>
>>> wrote:
>>>
>>> Hi Gera,
>>>
>>> It's my bad, I thought only src/bin tarball is enough.
>>>
>>> I just uploaded all other things under artifact/ to
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>>
>>> Please let me know if you have any other comments.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com>
>>> wrote:
>>>
>>>
>>> Thanks, Wangda!
>>>
>>> There are many more artifacts in previous votes, e.g., see
>>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>>> site tarball is missing.
>>>
>>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org> wrote:
>>>
>>>
>>> Thanks Wangda for initiating the release.
>>>
>>> I tested this RC built from source file.
>>>
>>>
>>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>>> UI.
>>> - Below feature sanity is done
>>> - Application priority
>>> - Application timeout
>>> - Intra Queue preemption with priority based
>>> - DS based affinity tests to verify placement constraints.
>>> - Tested basic NodeLabel scenarios.
>>> - Added couple of labels to few of nodes and behavior is coming
>>> correct.
>>> - Verified old UI and new YARN UI for labels.
>>> - Submitted apps to labelled cluster and it works fine.
>>> - Also performed few cli commands related to nodelabel.
>>> - Test basic HA cases and seems correct.
>>> - Tested new YARN UI . All pages are getting loaded correctly.
>>>
>>>
>>> - Sunil
>>>
>>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheele...@gmail.com>
>>> wrote:
>>>
>>>
>>> Hi folks,
>>>
>>> Thanks to the many who helped with this release since Dec 2017 [1].
>>> We've
>>>
>>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>>
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>>
>>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>>
>>> The maven artifacts are available via http://repository.apache.org at
>>> https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1090/
>>>
>>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>>
>>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>>> include the first class GPU/FPGA support on YARN, Native services,
>>> Support
>>>
>>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>>> block replicas to be provided by an external storage system, etc.
>>>
>>> For 3.1.0 RC0 vote discussion, please see [3].
>>>
>>> We’d like to use this as a starting release for 3.1.x [1], depending on
>>> how
>>>
>>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>> as
>>>
>>> the stable release.
>>>
>>> We have done testing with a pseudo cluster:
>>> - Ran distributed job.
>>> - GPU scheduling/isolation.
>>> - Placement constraints (intra-application anti-affinity) by using
>>> distributed shell.
>>>
>>> My +1 to start.
>>>
>>> Best,
>>> Wangda/Vinod
>>>
>>> [1]
>>>
>>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
>>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>>
>>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>> BY
>>>
>>> fixVersion ASC
>>> [3]
>>>
>>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
>>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
