Vote summary:
+1
Xiaoqiao He (binding)
Steve (binding)
Ahmar (binding)
Wei-Chiu Chuang (binding)
Mukund (binding)
Syed Shameerur Rahman (non-binding)
Cheng Pan (non-binding)
Dongjoon Hyun (non-binding)
Sneha Vijayarajan (non-binding)
=0
Ayush Saxena
-1
None
With 5 binding +1 votes and 4 non-b
Thanks Ayush for finding this issue. Looks like there is a problem with the
ARM binaries. The reason for that could be that the create-release script
doesn't work in ARM docker container but works manually after logging
inside the container. So I was creating the tars manually. Seems like it
didn't
Thanx Mukund for sharing the details, I might have missed that
discussion. Do mention the purpose on the website once we upload the
lean jar, so that relevant people can use it.
I am holding my vote due to [3], that in general looks blocker to me,
If I haven't messed up. Would be great if someone
Hi Ayush,
"lean" tar is a small tar file which doesn't contain the AWS SDK because
the size of AWS SDK is itself 500 MB.
This can ease usage for non AWS users. Even AWS users can add this jar
explicitly if desired.
This is created using https://github.com/apache/hadoop-release-support
which is the
+1 (binding)
Ozone 2.0 test result
https://github.com/jojochuang/ozone/actions/runs/11388335221
Only one issue which is protobuf update. That is known and I will be fixing
that in the coming days.
HBase master branch (3.0) test result
https://github.com/apache/hbase/pull/6331
The one test failure
What exactly is this "lean tar"? I couldn’t find any mention of it in
[1], nor did I come across any thread establishing consensus on adding
it, as we did for the aarch64 tar. It’s also not clear to me from the
create-release script how it is getting generated. So, where is this
coming from? Additi
Adding my +1 (binding) after all the testing.
Ran a lot of validation workflows from hadoop-release-support all went
fine.
Verified checksum and signatures.
Build orc and reran the failing vectored io tests and it succeeded.
Build parquet and run the hadoop parquet module tests.
@Steve Loughran
+1 (non-binding)
Ran the ABFS driver tests and all the tests passed.
HNS-OAuth
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 646, Failures
Update my vote to +1(binding).
Addendum: Verified signature was correct for both src/binary/site tarball.
On Mon, Oct 14, 2024 at 5:26 PM Xiaoqiao He wrote:
> Thanks Mukund and Steve for driving this release.
>
> +0. Will +1 when signature check passed.
>
> [Y] LICENSE files exist and NOTICE i
thanks.
before the vote is done: can everyone with a pgp key sign the new keys and
push to github. Even after 24h my trust signature doesn't seem to be there
yet.
You've all just been running code he compiled on your machines, so your
trust must be fairly high unless you did it in an isolated cont
+1 (non-binding)
- build success on Ubuntu 22.04 / x86 / Java 8 with `mvn clean package -Pdist
-Dtar -DskipTests`.
- setup a HDFS/YARN cluster(1 master node + 3 worker nodes) and run some Spark
(3.5.3) jobs.
Thanks,
Cheng Pan
-
+1 binding
*Testing:*
*mac arm64:*
the hadoop-release-support validation workflow including building spark,
cloudstore, key validation etc. All good, except for gcs, which was
complaining about hadoop-client stuff. I'm not worrying about that as it is
a fussy little library. I did verify that t
* Verified S3A (hadoop-tools/hadoop-aws) integ tests with scale profile passing
against Amazon S3 in eu-west-1
From: Mukund Madhav Thakur
Sent: Thursday, October 10, 2024 12:17:34 PM
To: Hadoop Common
Subject: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.1 RC3
+1 (non-binding)
I ran Apache ORC (ORC-1782) and Apache Spark (SPARK-49550) unit tests.
It looks like there is no regression.
Thank you all for making Apache Hadoop 3.4.1 RC3.
Best Regards,
Dongjoon Hyun.
On 2024/10/15 14:08:00 Syed Shameerur Rahman wrote:
> Thanks Mukund. I am able to verify t
Thanks Mukund. I am able to verify the signature now,
+1 (non-binding)
* Verified sha512 checksum was correct for source tarball
* Verified signature was correct for source tarball (not verified trust)
* Built source code on Amazon-Linux 2023 and OpenJDK 8 in Amazon EC2 with
ARM64 machine)
* Ver
FYI - I created the RC3 a few days ago. So abandoning RC2. Hopefully we
won't find any issues there.
Thanks
Mukund
On Tue, Oct 8, 2024 at 11:46 PM Steve Loughran
wrote:
> On Sat, 5 Oct 2024 at 01:58, Wei-Chiu Chuang wrote:
>
> > Hey the 3.4.1 tarball is a whopping 929MB! The corresponding dock
I have updated the key in the KEYS file today morning so it should
work now.
On Mon, Oct 14, 2024 at 5:59 PM Mukund Madhav Thakur
wrote:
> Thanks for checking.
> Yes I have used a new key as it is my new laptop. I will update the key.
>
> *➜ **Downloads* gpg --verify hadoop-3.4.1.tar.gz.asc had
Thanks for checking.
Yes I have used a new key as it is my new laptop. I will update the key.
*➜ **Downloads* gpg --verify hadoop-3.4.1.tar.gz.asc hadoop-3.4.1.tar.gz
gpg: Signature made Wed Oct 9 22:40:30 2024 IST
gpg:using RSA key 53931DAA708291409958BD474D22BB7D32882201
gpg
I tested S3A unit test and integration with ARM64 machines. The tests are
passing but have concerns with signature
* Unable to verify signature
gpg --verify hadoop-3.4.1.tar.gz.asc hadoop-3.4.1.tar.gz gpg: Signature
made Wed Oct 9 17:10:24 2024 UTC gpg: using RSA key
53931DAA708291409958BD474D22B
Thanks Mukund and Steve for driving this release.
+0. Will +1 when signature check passed.
[Y] LICENSE files exist and NOTICE is included.
[Y] Rat check is ok. mvn clean apache-rat:check
[Y] Build the source code on Ubuntu and OpenJDK 11 by `mvn clean package
-DskipTests -Pnative -Pdist -Dtar`.
[
Hney, you did the -lean one too! nice
Anyway, yes I'll test. I will try not to find new problems
On Thu, 10 Oct 2024 at 12:18, Mukund Madhav Thakur
wrote:
> Apache Hadoop 3.4.1
>
>
> With help of Steve, I have put together a release candidate (RC3) for
> Hadoop 3.4.1.
>
>
> What we would like
Apache Hadoop 3.4.1
With help of Steve, I have put together a release candidate (RC3) for
Hadoop 3.4.1.
What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.
The RC is available at:
https://dist.apac
On Sat, 5 Oct 2024 at 01:58, Wei-Chiu Chuang wrote:
> Hey the 3.4.1 tarball is a whopping 929MB! The corresponding docker image
> is over 1.1GB. Not that long ago, 3.2.3 was less than 500MB 2 years ago.
> 3.3.6 was less than 700MB a year ago.
> That AWS SDK v2 jar itself is more than 500MB.
>
fo
This is a real problem with that aWS SDK, but it is that or field support
calls forever.
Let me see if I can revisit my idea of having the release support project
produce a "lean" edition where we delete that bundle.jar
On Sat, 5 Oct 2024 at 01:58, Wei-Chiu Chuang wrote:
> Hey the 3.4.1 tarball
Sent: Wednesday, September 25, 2024 9:13:48 PM
To: Hadoop Common
Subject: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.1 RC2
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
A
Hey the 3.4.1 tarball is a whopping 929MB! The corresponding docker image
is over 1.1GB. Not that long ago, 3.2.3 was less than 500MB 2 years ago.
3.3.6 was less than 700MB a year ago.
That AWS SDK v2 jar itself is more than 500MB.
One issue I found with Ozone is protobuf classpath.
this test is
TL;DR I think the 3.4.1 RC2 is okay from an Ozone's perspective. Most of
the issues were introduced in 3.4.0 and not too terrible.
I am planning to introduce Hadoop 3.4.0 for Ozone 2.0 so incompatibilities
are acceptable.
Thanks for the tips.
Yes I had to bump the hadoop-thirdparty version as wel
I hope the following information will be helpful.
>From the perspective of compilation, Hadoop on JDK17 currently doesn’t face
many obstacles.
The code on the trunk branch can already be compiled directly on JDK17.
I will verify the situation for hadoop-3.4.1 and then provide feedback.
If we wa
you using the hadoop thirdparty jar? there is a 1.3.0 release out
On Wed, 2 Oct 2024 at 17:01, Wei-Chiu Chuang wrote:
> HBase project is adding support for Hadoop 3.4.0, and I had to add a few
> changes on top of that to let HBase shading to pass (license issues due to
> transitive dependencies
HBase project is adding support for Hadoop 3.4.0, and I had to add a few
changes on top of that to let HBase shading to pass (license issues due to
transitive dependencies and so on). Those are quite common when updating to
a new Hadoop version.
But apart from that it builds and unit tests passed
Please do!
On Tue, 1 Oct 2024 at 20:54, Wei-Chiu Chuang wrote:
> Hi I'm late to the party, but I'd like to build and test this release with
> Ozone and HBase.
>
> On Tue, Oct 1, 2024 at 2:12 AM Mukund Madhav Thakur
> wrote:
>
> > Thanks @Dongjoon Hyun for trying out the RC
> and
> > finding ou
Hi I'm late to the party, but I'd like to build and test this release with
Ozone and HBase.
On Tue, Oct 1, 2024 at 2:12 AM Mukund Madhav Thakur
wrote:
> Thanks @Dongjoon Hyun for trying out the RC and
> finding out this bug. This has to be fixed.
> It would be great if others can give the RC a
Thanks @Dongjoon Hyun for trying out the RC and
finding out this bug. This has to be fixed.
It would be great if others can give the RC a try such that we know of any
issues earlier.
Thanks
Mukund
On Tue, Oct 1, 2024 at 2:21 AM Steve Loughran
wrote:
> ok, we will have to consider that a -1
>
>
>
>
> > Do you think we can have an option to handle the overlapping ranges in
> Hadoop layer instead of introducing a breaking change to the users at the
> maintenance release?
>
>
On this one. We decided that we won't support overlapping ranges in
Vectored IO as that will cause complexity in memo
ok, we will have to consider that a -1
Interestingly we haven't seen that on any of our internal QE, maybe none of
the requests weren't overlapping.
I was just looking towards an =0 because of
https://issues.apache.org/jira/browse/HADOOP-19295
*Unlike the v1 sdk, PUT/POST of data now shares the
Unfortunately, it turns out to be a regression in addition to a breaking change.
In short, HADOOP-19098 (or more) makes Hadoop 3.4.1 fails even when users give
disjoint ranges.
I filed a Hadoop JIRA issue and a PR. Please take a look at that.
- HADOOP-19291. `CombinedFileRange.merge` should not
Thank you for 3.4.1 RC2.
HADOOP-19098 (Vector IO: consistent specified rejection of overlapping ranges)
seems to be a hard breaking change at 3.4.1.
Do you think we can have an option to handle the overlapping ranges in Hadoop
layer instead of introducing a breaking change to the users at the m
Apache Hadoop 3.4.1
With help from Steve I have put together a release candidate (RC2) for
Hadoop 3.4.1.
What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.
The RC is available at:
https://dist.apa
Thank you for the information, Steve.
Dongjoon.
On 2024/09/25 09:40:25 Steve Loughran wrote:
> mukund was apparently building the next RC at the w/e before Anuj Modi @
> microsoft found a regression. There were some other problems they found
> related to scale testing (HADOOP-19279), so it's been
mukund was apparently building the next RC at the w/e before Anuj Modi @
microsoft found a regression. There were some other problems they found
related to scale testing (HADOOP-19279), so it's been some last-minute abfs
stabilisation.
Assuming we can get the RC out this week, and all is good, we
Hi, is there any schedule to resume Apache Hadoop 3.4.1 release?
Dongjoon.
On 2024/08/16 15:17:28 Steve Loughran wrote:
> Afraid I have to say -1 to this iteration, but I promise I'll help address
> the issues
>
> First, I've cherrypicked a few final changes from branch-3.4 in, including
> this
Afraid I have to say -1 to this iteration, but I promise I'll help address
the issues
First, I've cherrypicked a few final changes from branch-3.4 in, including
this major one
HADOOP-19153. hadoop-common exports logback as a transitive dependency
(#6999)
This broke parquet hadoop-test runs withou
Thanks Masatake for that. Just pushed the update to the branch-.3.4.
Also realized I have to update index.md.vm with new bult delete information
which I just did and review the open jiras targeting 3.4.1 and move them to
3.4.2 if required. Will review that with Steve and update soon.
On Thu, Aug 8
Hi Mukund,
Thanks for your effort for the new release.
This is off branch-3.4.
Could you bump the version of branch-3.4 from 3.4.1-SNAPSHOT to 3.4.2-SNAPSHOT
as described in the HowToRelease wiki page[1]?
I added 3.4.2 in JIRA administration page of HADOOP[2], HDFS, YARN and
MAPREDUCE.
Now
Apache Hadoop 3.4.1
I with help of Steve have put together a release candidate (RC1) for Hadoop
3.4.1.
What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.
The RC is available at:
https://dist.ap
45 matches
Mail list logo
