[jira] [Resolved] (HDFS-17768) Observer namenode network delay causing empty block location for getBatchedListing

[ https://issues.apache.org/jira/browse/HDFS-17768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-17768. Fix Version/s: 3.5.0 Resolution: Fixed > Observer namenode network delay caus

[jira] [Resolved] (HDFS-15230) Sanity check should not assume key base name can be derived from version name

[ https://issues.apache.org/jira/browse/HDFS-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-15230. Resolution: Fixed Thanks [~szetszwo]! > Sanity check should not assume key base name

[jira] [Resolved] (HDFS-16644) java.io.IOException Invalid token in javax.security.sasl.qop

[ https://issues.apache.org/jira/browse/HDFS-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16644. Resolution: Fixed Cherrypicked the commit all the way to branch-3.2

[jira] [Resolved] (HDFS-17159) Can't decode Identifier HDFS tokens with only the hdfs client jar

[ https://issues.apache.org/jira/browse/HDFS-17159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-17159. Fix Version/s: 2.10.3 Resolution: Fixed I am not sure if we will ever have another

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

I am +1 too. * Verified signature. * Release notes and changelogs look good. On Fri, Mar 21, 2025 at 4:35 PM Chris Nauroth wrote: > +1 > > * Verified signatures. > * LICENSE-binary has the expected dependencies and versions. > * Site looks good too. > * Specifically confirmed inclusion of HADOO

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

> update or you need to refresh your keys. I will have a look at the keys > file tomorrow > > On Fri, 14 Mar 2025 at 21:46, Wei-Chiu Chuang wrote: > > > Looks like your GPG key expired? > > > > gpg: Signature made Fri Mar 14 08:56:00 2025 PDT > > gpg:

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Verified SHA-512 checksum, GPG signature -- all good except that the signing key expired. Confirmed git tag release-1.4.0-RC0 and branch name branch-1.4.0 both maps to commit hash 5595f1357eaad02e1cfb660bcce7fd34515197ff. Confirmed it is released as 1.4.0. Noticed HADOOP-19075

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Looks like your GPG key expired? gpg: Signature made Fri Mar 14 08:56:00 2025 PDT gpg:using RSA key 38237EE425050285077DB57AD22CF846DBB162A0 gpg: Good signature from "Steve Loughran (ASF code sign key - 2018) < ste...@apache.org>" [expired] gpg: aka "[jpeg image of

Re: Data not visible to new readers after SequenceFile.writer.hflush()

This is Hadoop's unit test for sequence file exercising hsync/hflush: https://github.com/apache/hadoop/blob/61df1b27a797efd094328c7d9141b9e157e01bf4/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestHSync.java#L151 On Mon, Jan 27, 2025 at 9:53 AM Wei

Re: Data not visible to new readers after SequenceFile.writer.hflush()

Do you have a unit test to reproduce? Note that for performance reasons, the actual HDFS hflush/hsync implementation does not update visible length at NameNode. The data is flushed to DataNode though. So the writer would need to record the client side visible length, and pass it to the reader, if

[jira] [Resolved] (HDFS-17080) Fix ec connection leak (GitHub PR#5807)

[ https://issues.apache.org/jira/browse/HDFS-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-17080. Fix Version/s: 3.5.0 Resolution: Fixed > Fix ec connection leak (GitHub PR#5

[jira] [Created] (HDFS-17645) Standby NameNode may fail to replay OP_ADD during transition to active

Wei-Chiu Chuang created HDFS-17645: -- Summary: Standby NameNode may fail to replay OP_ADD during transition to active Key: HDFS-17645 URL: https://issues.apache.org/jira/browse/HDFS-17645 Project

[jira] [Created] (HDFS-17635) MutableQuantiles.getQuantiles() should be made a static method

Wei-Chiu Chuang created HDFS-17635: -- Summary: MutableQuantiles.getQuantiles() should be made a static method Key: HDFS-17635 URL: https://issues.apache.org/jira/browse/HDFS-17635 Project: Hadoop

Re: HDFS-17611. Move all DistCp execution logic to execute()

shadedclient error looks concerning. Please do not merge until you build it locally successfully without -DskipShade option On Thu, Sep 5, 2024 at 1:21 PM Steve Loughran wrote: > before I merge, has anyone here see any problems from this PR? > https://github.com/apache/hadoop/pull/7025 >

[jira] [Resolved] (HDFS-17181) WebHDFS not considering whether a DN is good when called from outside the cluster

[ https://issues.apache.org/jira/browse/HDFS-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-17181. Fix Version/s: 3.5.0 Resolution: Fixed > WebHDFS not considering whether a DN

[jira] [Resolved] (HDFS-17024) Potential data race introduced by HDFS-15865

[ https://issues.apache.org/jira/browse/HDFS-17024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-17024. Fix Version/s: 3.4.0 Resolution: Fixed > Potential data race introduced by H

Join us at the Storage User Group Meetup!

Hi Please join us at the Storage Meetup at Cloudera's office next Wednesday. https://www.meetup.com/futureofdata-sanfrancisco/events/295917033/ We have HDFS developers from Uber join us to talk about optimizing HDFS for high density disks, and developers from Cloudera to talk about Apache Ozone a

HADOOP-18207 hadoop-logging module about to land

Hi, I am preparing to resolve HADOOP-18207 ( https://github.com/apache/hadoop/pull/5717). This change affects all modules. With this change, it will eliminate almost all the direct log4j usage. As always, landing such a big piece is tricky. I

Re: [DISCUSS][HDFS] Add rust binding for libhdfs

Inline On Sat, Jul 15, 2023 at 5:04 AM Ayush Saxena wrote: > Forwarding from dev@hadoop to relevant ML > > Original mail: > https://lists.apache.org/thread/r5rcmc7lwwvkysj0320myxltsyokp9kq > > -Ayush > > On 2023/07/15 09:18:42 Xuanwo wrote: > > Hello, everyone. > > > > I'm the maintainer of [hdf

[jira] [Created] (HDFS-17080) Fix ec connection leak (GitHub PR#5807)

Wei-Chiu Chuang created HDFS-17080: -- Summary: Fix ec connection leak (GitHub PR#5807) Key: HDFS-17080 URL: https://issues.apache.org/jira/browse/HDFS-17080 Project: Hadoop HDFS Issue Type

[ANNOUNCE] Apache Hadoop 3.3.6 release

On behalf of the Apache Hadoop Project Management Committee, I am pleased to announce the release of Apache Hadoop 3.3.6. It contains 117 bug fixes, improvements and enhancements since 3.3.5. Users of Apache Hadoop 3.3.5 and earlier should upgrade to this release. https://hadoop.apache.org/releas

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

> -Ayush > > > [1] https://github.com/ayushtkn/hadoop-ozone/actions/runs/5282707769 > [2] https://github.com/apache/tez/pull/285#issuecomment-1590962978 > > On Sat, 24 Jun 2023 at 09:43, Nilotpal Nandi > wrote: > >> +1 (Non-binding). >> Thanks a lot Wei-Chiu

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

Launched a pseudo distributed cluster using released binary packages, > > done some HDFS dir/file basic opeations. > > + Run grep, pi and wordcount MR tasks on the pseudo cluster. > > > > Bests, > > Sammi Chen > > > > 发件人:

Clean up old Hadoop release tarballs

https://dist.apache.org/repos/dist/release/hadoop/common/ has Hadoop release tarballs 3.3.1 ~ 3.3.5. I plan to remove the tarballs from 3.3.1 to 3.3.4 and leave only 3.3.5 (and the upcoming 3.3.6). Shout out if you have something depending on the old release tarballs (you shouldn't) Other release

DockerHub admin for Apache Hadoop

Ayush and I have acquired the DockerHub admin privilege for the Hadoop project in order to facilitate the release of Hadoop 3.3.6. Apache Infra allows only two seats per project. So if you need something, let Ayush and I know and we will make it happen for you. If you are a Docker guru and can't h

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

gt; > + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK > 8 > >> > with native profile enabled. > >> >+ launched pseudo distributed cluster including kms and httpfs with > >> > Kerberos and SSL enabled. > >> >+ created

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

Bumping this thread to the top. If you are verifying the release, please vote on this thread. RC0 and RC1 are exactly the same. The only material difference is the Changelog. Thanks!! On Sun, Jun 18, 2023 at 5:52 PM Wei-Chiu Chuang wrote: > I am inviting anyone to try and vote on this rele

[VOTE] Release Apache Hadoop 3.3.6 RC1

I am inviting anyone to try and vote on this release candidate. Note: This is exactly the same as RC0, except the CHANGELOG. The RC is available at: https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64) https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64) Git tag:

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

lease. The next RC will be prepared, > right? > If true, I would like to try and vote on the next RC. > Just notice that some JIRAs are not included and need to revert some PRs to > pass HBase verification which are mentioned above. > > Best Regards, > - He Xiaoqiao > &

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

ey for gs://fake-in-memory-test-b ucket/contract-test/testOpenFileApplyAsyncRead "fs.option.openfile.length" On Wed, Jun 14, 2023 at 5:01 PM Wei-Chiu Chuang wrote: > The hbase-filesystem tests passed after reverting HADOOP-18596 > <https://issues.apache.org/jira/b

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

It's branching off branch-3.3 On Thu, Jun 15, 2023 at 3:18 AM Steve Loughran wrote: > Which branch is -3.3.6 off? 3.3.5 or 3.3? > > I'm travelling for the next few days and unlikely to be able to test this; > will do my best > > On Wed, 14 Jun 2023 at 07:27, Wei-Ch

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

7;s not the end of the world. I think we can address it by adding an incompatible change flag and a release note. On Wed, Jun 14, 2023 at 3:55 PM Wei-Chiu Chuang wrote: > Cross referenced git history and jira. Changelog needs some update > > Not in the release > >1. HDFS-16858 &

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

59->AbstractContractDistCpTest.distCpUpdateDeepDirectoryStructure:334->AbstractContractDistCpTest.assertCounterInRange:294->Assert.assertTrue:41->Assert.fail:88 Files Copied value 2 above maximum 1 [INFO] [ERROR] Tests run: 240, Failures: 3, Errors: 0, Skipped: 58 Ozone test in progress. Wil

[VOTE] Release Apache Hadoop 3.3.6 RC0

I am inviting anyone to try and vote on this release candidate. Note: This is built off branch-3.3.6 plus PR#5741 (aws sdk update) and PR#5740 (LICENSE file update) The RC is available at: https://home.apache.org/~weichiu/hadoop-3.3.6-RC0-amd64/ (for amd64) https://home.apache.org/~weichiu/hadoop

Re: [DISCUSS] Hadoop 3.3.6 release planning

ace. While setSafeMode() is exposed through SafeMode interface. On Thu, Jun 8, 2023 at 11:11 AM Wei-Chiu Chuang wrote: > Thanks for comments > > Looking at jiras fixed in 3.3.6 and 3.3.9 (my bad, forgot that most > commits landing in branch-3.3 was 3.3.9), most are okay. We have about

Fwd: [jira] [Created] (HADOOP-18768) Integrating Apache Hadoop into OSS-Fuzz

Are there Hadoop committers who would like to help triage bug reports from OSS-Fuzz? -- Forwarded message - From: Henry Lin (Jira) Date: Mon, Jun 12, 2023 at 9:10 AM Subject: [jira] [Created] (HADOOP-18768) Integrating Apache Hadoop into OSS-Fuzz To: Henry Lin created HADOOP-18

Re: [DISCUSS] Mysql-connector-java is GPL licensed; Hadoop can't use it.

ginal jira. > > .. Owen > > > On Fri, Jun 9, 2023 at 1:58 AM Wei-Chiu Chuang > > wrote: > > > Hi community, > > > > While preparing for 3.3.6 RC, I realized the mysql-connector-java > > dependency added by HADOOP-18535 > > <https://issues.ap

[DISCUSS] Mysql-connector-java is GPL licensed; Hadoop can't use it.

Hi community, While preparing for 3.3.6 RC, I realized the mysql-connector-java dependency added by HADOOP-18535 is GPL licensed. Source: https://github.com/mysql/mysql-connector-j/blob/release/8.0/LICENSE See legal discussion at Apache LEGAL-

Re: [DISCUSS] Hadoop 3.3.6 release planning

iu for driving this release. > > > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed > version. > > > > > > > > so far only 8 jiras were resolved in the branch-3.3 line. > > > > > > > > > > > > If we should consider bo

[jira] [Created] (HDFS-17024) Potential data race introduced by HDFS-15865

Wei-Chiu Chuang created HDFS-17024: -- Summary: Potential data race introduced by HDFS-15865 Key: HDFS-17024 URL: https://issues.apache.org/jira/browse/HDFS-17024 Project: Hadoop HDFS Issue

Re: Call for Presentations, Community Over Code 2023

There's also a call for presentation for Community over Code Asia 2023 https://www.bagevent.com/event/cocasia-2023-EN Happening Aug 18-20. CfP due by 6/6 On Tue, May 9, 2023 at 8:39 PM Ayush Saxena wrote: > Forwarding from dev@hadoop to the dev ML which we use. > > The actual mail lies here: >

[DISCUSS] Hadoop 3.3.6 release planning

Hi community, I'd like to kick off the discussion around Hadoop 3.3.6 release plan. I'm being selfish but my intent for 3.3.6 is to have the new APIs in HADOOP-18671 added so we can have HBase to adopt this new API. Other than that, perhaps thi

Re: [DISCUSS] hadoop branch-3.3+ going to java11 only

My random thoughts. Probably bad takes: There are projects experimenting with JDK17 now. JDK11 active support will end in 6 months. If it's already hard to migrate from JDK8 why not retarget JDK17. On Tue, Mar 28, 2023 at 10:30 AM Ayush Saxena wrote: > I know Jersey upgrade as a blocker. Some f

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

I think moving up interfaces to FileSystem or some abstract FileSystem class has a few benefits: 1. Application can potentially be made FS-agnostic, with hasPathCapabilities() check. At least, make the code to compile. 2. We will be able to add a contract test to ensure behavior is expected. The

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

is. Yes, those of us who can build the > entire stack can do it, but that gradually adds more divergence from the > open source libraries, reduces the test coverage overall and only increases > maintenance costs over time. > > steve > > On Thu, 16 Mar 2023 at 20:56, Wei-Chiu

[DISCUSS] Move HDFS specific APIs to FileSystem abstration

Hi, Stephen and I are working on a project to make HBase to run on Ozone. HBase, born out of the Hadoop project, depends on a number of HDFS specific APIs, including recoverLease() and isInSafeMode(). The HBase community [1] strongly voiced that they don't want the project to have direct dependen

[jira] [Resolved] (HDFS-16947) RBF NamenodeHeartbeatService to report error for not being able to register namenode in state store

[ https://issues.apache.org/jira/browse/HDFS-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16947. Resolution: Fixed > RBF NamenodeHeartbeatService to report error for not being able

Re: Request to create an ASF Jira account

I've submitted a request to create the account. You should receive an email shortly. On Tue, Feb 14, 2023 at 1:40 PM ravindra dingankar < ravindradingan...@gmail.com> wrote: > Hi, > > I am part of LinkedIn's HDFS team, and would like to start contributing to > HDFS and be part of the mailing lis

[jira] [Resolved] (HDFS-16873) FileStatus compareTo does not specify ordering

[ https://issues.apache.org/jira/browse/HDFS-16873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16873. Fix Version/s: 3.4.0 Resolution: Fixed > FileStatus compareTo does not spec

[jira] [Resolved] (HDFS-16871) DiskBalancer process may throws IllegalArgumentException when the target DataNode has capital letter in hostname

[ https://issues.apache.org/jira/browse/HDFS-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16871. Fix Version/s: 3.4.0 Resolution: Fixed > DiskBalancer process may thr

[jira] [Resolved] (HDFS-16854) TestDFSIO to support non-default file system

[ https://issues.apache.org/jira/browse/HDFS-16854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16854. Resolution: Duplicate > TestDFSIO to support non-default file sys

[jira] [Resolved] (HDFS-16839) It should consider EC reconstruction work when we determine if a node is busy

[ https://issues.apache.org/jira/browse/HDFS-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16839. Resolution: Fixed > It should consider EC reconstruction work when we determine if a n

[jira] [Created] (HDFS-16854) TestDFSIO to support non-default file system

Wei-Chiu Chuang created HDFS-16854: -- Summary: TestDFSIO to support non-default file system Key: HDFS-16854 URL: https://issues.apache.org/jira/browse/HDFS-16854 Project: Hadoop HDFS Issue

Re: Code coverage report on github PRs

I believe most of them can be added by us using GitHub Workflow. There's a marketplace for these tools and most of them are free for open source projects. On Wed, Nov 23, 2022 at 11:43 AM Ayush Saxena wrote: > A simple Infra ticket I suppose should get it done for us, eg. > https://issues.apache

[jira] [Resolved] (HDFS-9536) OOM errors during parallel upgrade to Block-ID based layout

[ https://issues.apache.org/jira/browse/HDFS-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-9536. --- Resolution: Duplicate I believe this is no longer an issue after HDFS-15937 and HDFS-15610

[DISCUSS] Supporting partial file rewrite/compose

There were a number of discussions that happened during ApacheCon. In the spirit of the Apache Way, I am taking the conversation online, sharing with the larger community and also capturing requirements. Credits to Owen who started this discussion. There are a number of scenarios where users want

[DISCUSS] Hadoop 3.3.5 release planning

Bumping this up. Adding the [DISCUSS] text to make this message stand out of your inbox. I certainly missed this message and only realized 3.3.5 has more than just security updates. What was the issue with the ARM64 build? I was able to publish ARM64 build for 3.3.1 release without problems. On

Re: Hadoop BoF at ApacheCon NA'22

t; On Sat, Sep 24, 2022 at 8:52 PM 俊平堵 wrote: > > > > > Yes. It should be a short talk and I can only present remotely for this > > > time. That would be great if you help to coordinate. :) > > > > > > Thanks, > > > > > > Junping &g

Re: Hadoop BoF at ApacheCon NA'22

document/d/1_ha1BFeEyIkAJtl5tJ8Z4Vf5Md_0rN23RCrXkyNCY1c/edit?usp=sharing Please add more details here. On Thu, Sep 22, 2022 at 10:27 PM 俊平堵 wrote: > Thanks Wei-Chiu. I am happy to share the status of Hadoop Meetups in China > (2019-2022) if that is a suitable topic. :) > > Thanks, > > Junping

Hadoop BoF at ApacheCon NA'22

We've not had a physical event for a long long time and we're way overdue for one. I'm excited to announce we've reserved a room at the upcoming ApacheCon for Birds-of-Feather on October 4th from 17:50-18:30 CDT in Rhythms I. I was also told that participants can stay after that until the hotel pe

[jira] [Resolved] (HDFS-4043) Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name.

[ https://issues.apache.org/jira/browse/HDFS-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-4043. --- Resolution: Fixed > Namenode Kerberos Login does not use proper hostname for host qualif

[jira] [Created] (HDFS-16730) Update the doc that append to EC files is supported

Wei-Chiu Chuang created HDFS-16730: -- Summary: Update the doc that append to EC files is supported Key: HDFS-16730 URL: https://issues.apache.org/jira/browse/HDFS-16730 Project: Hadoop HDFS

[jira] [Created] (HDFS-16727) Consider reading chunk files using MappedByteBuffer

Wei-Chiu Chuang created HDFS-16727: -- Summary: Consider reading chunk files using MappedByteBuffer Key: HDFS-16727 URL: https://issues.apache.org/jira/browse/HDFS-16727 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-16619) Fix HttpHeaders.Values And HttpHeaders.Names Deprecated Import.

[ https://issues.apache.org/jira/browse/HDFS-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16619. Resolution: Fixed > Fix HttpHeaders.Values And HttpHeaders.Names Deprecated Imp

[jira] [Resolved] (HDFS-16595) Slow peer metrics - add median, mad and upper latency limits

[ https://issues.apache.org/jira/browse/HDFS-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16595. Fix Version/s: 3.4.0 Resolution: Fixed > Slow peer metrics - add median, mad

[jira] [Resolved] (HDFS-16583) DatanodeAdminDefaultMonitor can get stuck in an infinite loop

[ https://issues.apache.org/jira/browse/HDFS-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16583. Resolution: Fixed > DatanodeAdminDefaultMonitor can get stuck in an infinite l

[jira] [Resolved] (HDFS-16603) Improve DatanodeHttpServer With Netty recommended method

[ https://issues.apache.org/jira/browse/HDFS-16603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16603. Resolution: Fixed > Improve DatanodeHttpServer With Netty recommended met

[jira] [Resolved] (HDFS-16585) Add @VisibleForTesting in Dispatcher.java after HDFS-16268

[ https://issues.apache.org/jira/browse/HDFS-16585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16585. Fix Version/s: 3.4.0 3.2.4 3.3.4 Resolution

[jira] [Resolved] (HDFS-16583) DatanodeAdminDefaultMonitor can get stuck in an infinite loop

[ https://issues.apache.org/jira/browse/HDFS-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16583. Resolution: Fixed > DatanodeAdminDefaultMonitor can get stuck in an infinite l

[jira] [Created] (HDFS-16585) Add @VisibleForTesting in Dispatcher.java after HDFS-16268

Wei-Chiu Chuang created HDFS-16585: -- Summary: Add @VisibleForTesting in Dispatcher.java after HDFS-16268 Key: HDFS-16585 URL: https://issues.apache.org/jira/browse/HDFS-16585 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-16520) Improve EC pread: avoid potential reading whole block

[ https://issues.apache.org/jira/browse/HDFS-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16520. Resolution: Fixed Merged the PR and cherrypicked into branch-3.3. Thanks! > Improve

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

Running builds for all platforms for each and every PR seems too excessive. How about doing all platform builds in the nightly jobs? On Fri, May 6, 2022 at 8:02 AM Steve Loughran wrote: > I'm not enthusiastic here as it not only makes the builds slower, it > reduces the #of builds we can throug

[jira] [Resolved] (HDFS-16521) DFS API to retrieve slow datanodes

[ https://issues.apache.org/jira/browse/HDFS-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16521. Resolution: Fixed > DFS API to retrieve slow datano

Re: [DISCUSS] Hadoop on Windows

Great! Sorry I missed the earlier discussion thread. Is there a target version for this support? I assume the milestone is still in a dev branch? On Thu, Apr 28, 2022 at 8:26 AM Gautham Banasandra wrote: > Hi Hadoop devs, > > I would like to announce that we recently reached a new milestone - w

[jira] [Resolved] (HDFS-16551) Backport HADOOP-17588 to 3.3 and other active old branches.

[ https://issues.apache.org/jira/browse/HDFS-16551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16551. Fix Version/s: 2.10.2 3.2.4 Resolution: Fixed Done. Thanks

[jira] [Resolved] (HDFS-16389) Improve NNThroughputBenchmark test mkdirs

[ https://issues.apache.org/jira/browse/HDFS-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16389. Fix Version/s: 3.4.0 Resolution: Fixed > Improve NNThroughputBenchmark test mkd

[jira] [Resolved] (HDFS-16535) SlotReleaser should reuse the domain socket based on socket paths

[ https://issues.apache.org/jira/browse/HDFS-16535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16535. Fix Version/s: 3.3.3 3.4.0 Resolution: Fixed Merged. Thanks

Re: Institutions running kerberized Hadoop clusters

Last time I checked (~2 years ago), there were thousands of Kerberized clusters among Cloudera customers. The largest ones had a few thousand nodes. What are you looking for? On Wed, Apr 13, 2022 at 7:22 AM Santosh Marella wrote: > Hey folks, > > Just curious if we have a list of institutions

Re: [VOTE] Release Apache Hadoop 3.2.3 - RC0

aarch64 support is only introduced in/after 3.3.0 On Thu, Mar 17, 2022 at 2:27 PM Emil Ejbyfeldt wrote: > Hi, > > > There is no aarch64 artifact in the release candidate. Is this something > that is intended? > > Best, > Emil Ejbyfeldt > > On 14/03/2022 08:14, Masatake Iwasaki wrote: > > Hi all,

[jira] [Resolved] (HDFS-16502) Reconfigure Block Invalidate limit

[ https://issues.apache.org/jira/browse/HDFS-16502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16502. Fix Version/s: 3.4.0 3.3.3 Resolution: Fixed > Reconfigure Bl

[ANNOUNCE] New Hadoop PMC Sun Chao

On behalf of the Apache Hadoop PMC, I am pleased to announce that Sun Chao(sunchao) has accepted the PMC's invitation to become a PMC on the project. We appreciate all of Sun's generous contributions thus far and look forward to his continued involvement. Congratulations and welcome, Sun!

Re: [ANNOUNCE] Apache Hadoop 3.3.2 release

> > Many thanks to Viraj Jasani, Michael Stack, Masatake Iwasaki, Xiaoqiao He, > Mukund Madhav Thakur, Wei-Chiu Chuang, Steve Loughran, Akira Ajisaka and > other folks who helped for this release process. > > Best Regards, > Chao >

[jira] [Resolved] (HDFS-16422) Fix thread safety of EC decoding during concurrent preads

[ https://issues.apache.org/jira/browse/HDFS-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16422. Fix Version/s: 3.4.0 3.2.3 3.3.3 Resolution

[jira] [Resolved] (HDFS-16437) ReverseXML processor doesn't accept XML files without the SnapshotDiffSection.

[ https://issues.apache.org/jira/browse/HDFS-16437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16437. Fix Version/s: 3.4.0 Resolution: Fixed Merged. Thanks [~it_singer] for the

[jira] [Resolved] (HDFS-16423) balancer should not get blocks on stale storages

[ https://issues.apache.org/jira/browse/HDFS-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16423. Fix Version/s: 3.3.3 Resolution: Fixed > balancer should not get blocks on st

[jira] [Resolved] (HDFS-16403) Improve FUSE IO performance by supporting FUSE parameter max_background

[ https://issues.apache.org/jira/browse/HDFS-16403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16403. Resolution: Fixed Thank you [~cndaimin] for the great work and the excellent performance

[jira] [Reopened] (HDFS-16423) balancer should not get blocks on stale storages

[ https://issues.apache.org/jira/browse/HDFS-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-16423: Reopen to backport this to lower branches. > balancer should not get blocks on stale stora

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2

I'll find time to check out the RC bits. I just feel bad that the tarball is now more than 600MB in size. On Fri, Jan 21, 2022 at 2:23 AM Steve Loughran wrote: > *+1 binding.* > > reviewed binaries, source, artifacts in the staging maven repository in > downstream builds. all good. > > *## test

Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2

+1 I think it makes sense to use reload4j in maint releases. I have a draft PR doing this (https://github.com/apache/hadoop/pull/3906) log4j2 in Hadoop 3.4.0 makes sense to me. There could be incompatibilities introduced by log4j2, but I feel we should at least make it 3.4.0 a "preview" release, a

Re: Hadoop-3.2.3 Release Update

Is this still making progress? On Tue, Oct 5, 2021 at 8:45 PM Brahma Reddy Battula wrote: > Hi Akira, > > Thanks for your email!! > > I am evaluating the CVE’s which needs to go for this release.. > > Will update soon!! > > > On Tue, 5 Oct 2021 at 1:46 PM, Akira Ajisaka wrote: > > > Hi Brahma,

Re: Next Mandarin Hadoop Online Meetup Jan 6th.

/JaNm70lZQGCZdlFzh9ZbsfrR7MJ7Nazb2g6NCtYPqsRLWtyEhLfgwXOppzMR3csp.HqRJNGXUGSaPu1qw Access Passcode: 4g1ZF&%f On Mon, Jan 3, 2022 at 5:39 PM Wei-Chiu Chuang wrote: > Hello community, > > This week we'll going to have Tao Li (tomscut) speaking about the > experience of operating HDFS at BIGO. See you on Thursday! > > 题目：《HDF

Re: Next Mandarin Hadoop Online Meetup Jan 6th.

Just a gentle reminder this is happening now. On Mon, Jan 3, 2022 at 5:39 PM Wei-Chiu Chuang wrote: > Hello community, > > This week we'll going to have Tao Li (tomscut) speaking about the > experience of operating HDFS at BIGO. See you on Thursday! > > 题目：《HDFS在BIGO的实践》

Next Mandarin Hadoop Online Meetup Jan 6th.

Hello community, This week we'll going to have Tao Li (tomscut) speaking about the experience of operating HDFS at BIGO. See you on Thursday! 题目：《HDFS在BIGO的实践》 概要：HDFS作为大数据底层存储服务，在BIGO的发展中起到了非常重要的作用。随着业务的发展和数据的爆发式增长，HDFS单个集群的瓶颈愈发凸显，我们借助Router，将多个HDFS集群整合成一个Namespace，增强集群的扩展能力；改造Router，使其支持Alluxi

[jira] [Resolved] (HDFS-16317) Backport HDFS-14729 for branch-3.2

[ https://issues.apache.org/jira/browse/HDFS-16317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16317. Fix Version/s: 3.2.3 Resolution: Fixed Merged the commit into branch-3.2 and branch

Apache Hadoop and CVE-2021-44228 Log4JShell vulnerability

Hi, Given the widespread attention to the recent log4j vulnerability (CVE-2021-44228), I'd like to share an update from the Hadoop developer community regarding the incident. As you probably know, Apache Hadoop depends on the log4j library to keep log files. The highlighted vulnerability CVE-2021-

[jira] [Reopened] (HDFS-16384) Upgrade Netty to 4.1.72.Final

[ https://issues.apache.org/jira/browse/HDFS-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-16384: > Upgrade Netty to 4.1.72.Final > - > >

Trunk broken by HDFS-16384

My bad. There was a transitive dependency problem in the PR causing trunk to fail the build. The commit has since been reverted. Sorry for the inconvenience.

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC0

Thanks a lot for pushing it forward! A few things I noticed that we should incorporate: 1. the overview page of the doc is for the Hadoop 3.0 release. It would be best to base the doc on top of Hadoop 3.3.0 overview page. (it's a miss on my part... The overview page of 3.3.1 wasn't updated) 2. A

Re: Next Hadoop community online meetup (Mandarin)

's in English so the wider audience can read) Please find today's recording here: https://cloudera.zoom.us/rec/share/MOwRDPP50eL7gRfNazfjqAXdl_7HwAaPH460sONmmlx_tLQrftVjjjCBkVvE-9sf.GRb9Kp_16xOqgzNu Passcode: L^n30&%d On Thu, Dec 9, 2021 at 2:10 PM Wei-Chiu Chuang wrote: >

Re: Next Hadoop community online meetup (Mandarin)

We are having a little technical difficulties. For those who are late, we are using this link instead: https://cloudera.zoom.us/j/93901387554 On Thu, Dec 9, 2021 at 12:55 PM Takanobu Asanuma wrote: > Thanks for the information! > > 2021年12月9日(木) 13:52 Wei-Chiu Chuang : > >>

  1   2   3   4   5   6   7   8   >