Hi, any updates on the current 1.4.1 progress? Ratis 3.1.1 should be in Ozone now that HDDS-11504 <https://issues.apache.org/jira/browse/HDDS-11504> is resolved. I see there’s discussion of doing a Ratis 3.1.2 to fix RATIS-2149 <https://issues.apache.org/jira/browse/RATIS-2149> and RATIS-2172 <https://issues.apache.org/jira/browse/RATIS-2172>, but our 1.4.1 release has already been delayed for a while, so I think we should ship with Ratis 3.1.1 and do a 1.4.2 release with just the patch version of Ratis if necessary.
I see some new fixes targeting the release like HDDS-11223 <https://issues.apache.org/jira/browse/HDDS-11223> and HDDS-11136 <https://issues.apache.org/jira/browse/HDDS-11136>, which is good. What is the overall status update? Are we ready for the next release candidate? Ethan On Wed, Aug 21, 2024 at 12:33 PM Tsz Wo Sze <szets...@gmail.com> wrote: > > (2) Key put fails for large files (> 20GB) due to a memory leak in Ratis > 3.1.0 > ... > > Duong & Wei-chiu, > > Thanks for finding this problem! > > Agree that we should have a Ratis 3.1.1 release. > BTW, "Memory leak" usually means that memory was allocated but not > released; see https://en.wikipedia.org/wiki/Memory_leak . In this case, we > are not having such a problem. Our problem is unnecessarily using too much > memory. > > Tsz-Wo > > > On Tue, Aug 20, 2024 at 6:20 PM Duong Nguyen <du...@cloudera.com.invalid> > wrote: > > > I also filed https://issues.apache.org/jira/browse/RATIS-2141 to track > the > > memory leak issue. > > > > Thanks, > > Duong > > > > On Tue, Aug 20, 2024 at 6:17 PM Duong Nguyen <du...@cloudera.com> wrote: > > > > > Hi all, > > > > > > I just started a thread to discuss releasing Ratis 3.1.1 with the fixes > > of > > > the mentioned issues. > > > > > > Duong > > > > > > On Tue, Aug 20, 2024 at 5:30 PM Uma Maheswara Rao Gangumalla < > > > umaganguma...@gmail.com> wrote: > > > > > >> Hi Wei-Chiu, > > >> > > >> Thank you and Duong for the important update on RC1. > > >> > > >> @Duong would you be notifying this to Ratis community if they can > make a > > >> quick release with just above 2 fixes? > > >> > > >> Regards, > > >> Uma > > >> > > >> > > >> On Tue, Aug 20, 2024 at 4:51 PM Wei-Chiu Chuang <weic...@apache.org> > > >> wrote: > > >> > > >>> Hi thanks for the effort, > > >>> We are testing the latest Ozone master and Ratis 3.1.0 internally, > and > > >>> found a few critical issues. > > >>> > > >>> (1) RATIS-2132 <https://issues.apache.org/jira/browse/RATIS-2132> > > which > > >>> has > > >>> about 10% performance regression penalty. > > >>> (2) Key put fails for large files (> 20GB) due to a memory leak in > > Ratis > > >>> 3.1.0: it was a haft-done feature of RATIS-1931. DataNode could crash > > due > > >>> to out of memory. > > >>> > > >>> Both of them can only be fixed in Ratis. > > >>> I'd suggest to not use Ratis 3.1.0 in Ozone 1.4.1 release. > > >>> > > >>> If we can, I'd ask the Ratis community to release Ratis 3.1.1 with > the > > >>> above two fixes. > > >>> > > >>> cc: @Duong Nguyen <du...@cloudera.com> who helped root cause the two > > >>> issues. > > >>> > > >>> > > >>> > > >>> > > >>> On Tue, Aug 20, 2024 at 3:31 PM Siyao Meng <si...@apache.org> wrote: > > >>> > > >>> > +1 (binding) > > >>> > > > >>> > > > >>> > - Verified signatures > > >>> > - Verified checksums > > >>> > - Checked ./bin/ozone version output from binary tarball > > >>> > - Checked ./bin/ozone checknative output from binary tarball > > >>> > - rocks_tools_native lib check is missing, filed HDDS-11347 > > >>> > <https://issues.apache.org/jira/browse/HDDS-11347>, > > >>> non-blocking. > > >>> > - Checked source tarball content matched repo tag > > ozone-1.4.1-RC1 > > >>> > - Built from source (without native libs support) > > >>> > - Verified compose/ozone Docker dev cluster boots up correctly > > with > > >>> 3 > > >>> > Ozone datanodes. > > >>> > - Verified basic volume, bucket, key creation and deletion works > > in > > >>> > Docker dev cluster. > > >>> > - Volume recursive deletion prompt is incorrect, filed > > HDDS-11346 > > >>> > <https://issues.apache.org/jira/browse/HDDS-11346>, > > >>> non-blocking. > > >>> > > > >>> > > > >>> > -Siyao > > >>> > > > >>> > On Aug 19, 2024 at 6:39:08 AM, Ayush Saxena <ayush...@gmail.com> > > >>> wrote: > > >>> > > > >>> > > +1 (Binding), some minor stuff which we should fix in next > release > > >>> > > > > >>> > > * Built from source > > >>> > > * Verified Checksums > > >>> > > * Verified Signatures > > >>> > > * All source files have apache header > > >>> > > * No code diff b/w the git tag & the contents of src tar > > >>> > > (dependency-reduced-pom only in src tar, maybe that ain't > required > > >>> > > there) > > >>> > > * Verified the output of ozone version > > >>> > > * Ran some basic shell commands > > >>> > > * Checked the NOTICE file: The year is *wrong*, it says 2022, it > > >>> > > should be 2024 [1], should correct in next release > > >>> > > * The NOTICE file inside the packaged Jars is *wrong*, It > mentions > > >>> > > *Apache Hadoop* & Copyright since 2006, it should be Apache > Ozone, > > >>> > > should fix in the next release. > > >>> > > It currently prints: > > >>> > > ``` > > >>> > > Apache Hadoop > > >>> > > Copyright 2006 and onwards The Apache Software Foundation. > > >>> > > . > > >>> > > . > > >>> > > Hadoop Yarn Server Web Proxy uses the BouncyCastle Java > > >>> > > cryptography APIs written by the Legion of the Bouncy Castle Inc. > > >>> > > > > >>> > > ``` > > >>> > > Can try something like to validate: > > >>> > > jar xf share/ozone/lib/ozone-client-1.4.1.jar META-INF/NOTICE.txt > > >>> > > cat META-INF/NOTICE.txt > > >>> > > > > >>> > > Thanx Xi Chen for driving the release, Good Luck!!! > > >>> > > > > >>> > > -Ayush > > >>> > > > > >>> > > [1] > > >>> > > > https://github.com/apache/ozone/blob/ozone-1.4.1-RC1/NOTICE.txt#L1-L2 > > >>> > > > > >>> > > On Mon, 19 Aug 2024 at 11:20, Sammi Chen <sammic...@apache.org> > > >>> wrote: > > >>> > > > > >>> > > > > >>> > > +1 (binding) > > >>> > > > > >>> > > > > >>> > > * Verified the signature and checksums > > >>> > > > > >>> > > * Verified tag > > >>> > > > > >>> > > * Build from source > > >>> > > > > >>> > > * Run ozonesecure acceptance test > > >>> > > > > >>> > > * Start a cluster using bin package > > >>> > > > > >>> > > * Run freon rk command with data verification > > >>> > > > > >>> > > * Verified information displayed on Recon UI, for both empty > > cluster > > >>> and > > >>> > > > > >>> > > cluster with data > > >>> > > > > >>> > > Sammi > > >>> > > > > >>> > > > > >>> > > On Fri, 16 Aug 2024 at 13:13, mrchenx <mrch...@126.com> wrote: > > >>> > > > > >>> > > > > >>> > > > Dear Ozone Devs, As discussed in the last email, I am > calling > > >>> for a > > >>> > > > > >>> > > > vote on Apache Ozone 1.4.1 RC1. > > >>> > > > > >>> > > > We have released 1.4.0 on Jan 19th. Now there are 177 new > > >>> commits > > >>> > > > > >>> > > > already landed on 1.4.1 branch, Includes Ratis upgrade (upgrade > > to > > >>> > Ratis > > >>> > > > > >>> > > > 3.1.0), some bug fixes, as well as performance optimizations, > and > > >>> some > > >>> > > > > >>> > > > necessary dependencies. I am calling for a vote on Apache > > Ozone > > >>> > 1.4.1 > > >>> > > > > >>> > > > RC1. - The RC1 tag can be found on Github at: > > >>> > > > > >>> > > > - > > >>> https://github.com/apache/ozone/releases/tag/ozone-1.4.1-RC1 > > >>> > > > > >>> > > > - 177 Jiras were cherry-pick for ozone-1.4.1 > > >>> > > > > >>> > > > - > > >>> > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20HDDS%20AND%20fixVersion%20%3D%201.4.1 > > >>> > > > > >>> > > > - The source and binary tarballs can be found at: > > >>> > > > > >>> > > > - > > https://dist.apache.org/repos/dist/dev/ozone/1.4.1-rc1/ > > >>> > > > > >>> > > > - Maven artifacts are staged at: > > >>> > > > > >>> > > > - > > >>> > > > > >>> > > > > > >>> > https://repository.apache.org/content/repositories/orgapacheozone-1024 > > >>> > > > > >>> > > > - The public key used to sign the artifacts can be found at: > > >>> > > > > >>> > > > - > https://dist.apache.org/repos/dist/release/ozone/KEYS > > >>> > > > > >>> > > > - The fingerprint of the key used to sign the artifacts is: > > >>> > > > > >>> > > > - 0D8C19F5514E2786007936F758C87003FF9A1A38 > > >>> > > > > >>> > > > The vote will run for 7 days, ending on Aug 23th 2024 at > 13:10 > > >>> pm > > >>> > > UTC+8. > > >>> > > > > >>> > > > > > >>> > > > > >>> > > > Thanks > > >>> > > > > >>> > > > > > >>> > > > > >>> > > > Xi Chen > > >>> > > > > >>> > > > > >>> > > > > --------------------------------------------------------------------- > > >>> > > To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org > > >>> > > For additional commands, e-mail: dev-h...@ozone.apache.org > > >>> > > > > >>> > > > > >>> > > > >>> > > >> > > >
