Thanks @Dongjoon Hyun <dongjoon.h...@gmail.com> for trying out the RC and finding out this bug. This has to be fixed. It would be great if others can give the RC a try such that we know of any issues earlier.
Thanks Mukund On Tue, Oct 1, 2024 at 2:21 AM Steve Loughran <ste...@cloudera.com.invalid> wrote: > ok, we will have to consider that a -1 > > Interestingly we haven't seen that on any of our internal QE, maybe none of > the requests weren't overlapping. > > I was just looking towards an =0 because of > > https://issues.apache.org/jira/browse/HADOOP-19295 > > *Unlike the v1 sdk, PUT/POST of data now shares the same timeout as all > other requests, and on a slow network connection requests time out. > Furthermore, large file uploads cn generate the same failure > condition because the competing block uploads reduce the bandwidth for the > others.* > > I'll describe more on the JIRA -the fix is straightforward, have a much > longer timeout, such as 15 minutes. It will mean that problems with other > calls will not timeout for the same time. > > Note that In previous releases that request timeout *did not* apply to the > big upload. This has been reverted. > > This is not a regression between 3.4.0; it had the same problem just nobody > has noticed. That's what comes from doing a lot of the testing within AWS > and other people doing the testing (me) not trying to upload files > 1GB. I > have now. > > Anyway, I do not consider that a -1 because it wasn't a regression and it's > straightforward to work around in a site configuration. > > Other than that, my findings were > -Pnative breaks enforcer on macos (build only; fix is upgrade enforcer > version) > > -native code probes on my ubuntu rasberry pi5 (don't laugh -this is the > most powerful computer I personally own) wan about a missing link in the > native checks. > I haven't yet set up openssl bindings for s3a and abfs to see if they > actually work. > > [hadoopq] 2024-09-27 19:52:16,544 WARN crypto.OpensslCipher: Failed to > load OpenSSL Cipher. > [hadoopq] java.lang.UnsatisfiedLinkError: EVP_CIPHER_CTX_block_size > [hadoopq] at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native > Method) > [hadoopq] at > org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90) > [hadoopq] at > org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker. > > You're one looks like it is. Pity -but thank you for the testing. Give it a > couple more days to see if people report any other issues. > > Mukund has been doing all the work on this; I'll see how much I can do > myself to share the joy. > > On Sun, 29 Sept 2024 at 06:24, Dongjoon Hyun <dongj...@apache.org> wrote: > > > Unfortunately, it turns out to be a regression in addition to a breaking > > change. > > > > In short, HADOOP-19098 (or more) makes Hadoop 3.4.1 fails even when users > > give disjoint ranges. > > > > I filed a Hadoop JIRA issue and a PR. Please take a look at that. > > > > - HADOOP-19291. `CombinedFileRange.merge` should not convert disjoint > > ranges into overlapped ones > > - https://github.com/apache/hadoop/pull/7079 > > > > I believe this is a Hadoop release blocker for both Apache ORC and Apache > > Parquet project perspective. > > > > Dongjoon. > > > > On 2024/09/29 03:16:18 Dongjoon Hyun wrote: > > > Thank you for 3.4.1 RC2. > > > > > > HADOOP-19098 (Vector IO: consistent specified rejection of overlapping > > ranges) seems to be a hard breaking change at 3.4.1. > > > > > > Do you think we can have an option to handle the overlapping ranges in > > Hadoop layer instead of introducing a breaking change to the users at the > > maintenance release? > > > > > > Dongjoon. > > > > > > On 2024/09/25 20:13:48 Mukund Madhav Thakur wrote: > > > > Apache Hadoop 3.4.1 > > > > > > > > > > > > With help from Steve I have put together a release candidate (RC2) > for > > > > Hadoop 3.4.1. > > > > > > > > > > > > What we would like is for anyone who can to verify the tarballs, > > especially > > > > > > > > anyone who can try the arm64 binaries as we want to include them too. > > > > > > > > > > > > The RC is available at: > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC2/ > > > > > > > > > > > > The git tag is release-3.4.1-RC2, commit > > > > b3a4b582eeb729a0f48eca77121dd5e2983b2004 > > > > > > > > > > > > The maven artifacts are staged at > > > > > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1426 > > > > > > > > > > > > You can find my public key at: > > > > > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > > > > > Change log > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC2/CHANGELOG.md > > > > > > > > > > > > Release notes > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC2/RELEASENOTES.md > > > > > > > > > > > > This is off branch-3.4. > > > > > > > > > > > > Key changes include > > > > > > > > > > > > * Bulk Delete API. > https://issues.apache.org/jira/browse/HADOOP-18679 > > > > > > > > * Fixes and enhancements in Vectored IO API. > > > > > > > > * Improvements in Hadoop Azure connector. > > > > > > > > * Fixes and improvements post upgrade to AWS V2 SDK in S3AConnector. > > > > > > > > * This release includes Arm64 binaries. Please can anyone with > > > > > > > > compatible systems validate these. > > > > > > > > > > > > Note, because the arm64 binaries are built separately on a different > > > > > > > > platform and JVM, their jar files may not match those of the x86 > > > > > > > > release -and therefore the maven artifacts. I don't think this is > > > > > > > > an issue (the ASF actually releases source tarballs, the binaries are > > > > > > > > there for help only, though with the maven repo that's a bit > blurred). > > > > > > > > > > > > The only way to be consistent would actually untar the x86.tar.gz, > > > > > > > > overwrite its binaries with the arm stuff, retar, sign and push out > > > > > > > > for the vote. Even automating that would be risky. > > > > > > > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Mukund > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > >
