Re: Missing full tarball for Hadoop 3.4.2

2025-09-23 Thread Xiaoqiao He
irectory) >>- We update Hive and Ozone to use the hadoop-3.4.2-lean.tar.gz for >>now. And from the next release, can go back to using hadoop-3.4.2.tar.gz. >> >> >> >> What do others think? >> >> >> >> *From: *slfan1989 >> *Date: *

[RESULT][VOTE] Propose to close stale PRs

2025-09-04 Thread Xiaoqiao He
ead: https://lists.apache.org/thread/7lrz00j0p4l850bfplf941otxlw6f0bw I will file a new JIRA to trace this topic. Best Regards, - He Xiaoqiao On Wed, Sep 3, 2025 at 6:23 PM Jian Zhang wrote: > +1 > > - Jian Zhang > > On 2025/09/01 11:57:27 Xiaoqiao He wrote: > > Hi All, > >

[VOTE] Propose to close stale PRs

2025-09-01 Thread Xiaoqiao He
Hi All, This is initiating a formal [VOTE] to propose to close the stale PR at Github which was updated over 100 days ago based on the discussion early[1]. Please tell me what your opinion is (+1/+0/-0/-1). Thanks all. Best Regards, - He Xiaoqiao [1] https://lists.apache.org/thread/n3xs0snlmrq6

Re: [DISCUSS] Propose to close stale PRs

2025-09-01 Thread Xiaoqiao He
Thanks all for your positive feedback, I will try to launch another official vote thread for a while. Thanks again. Best Regards, - He Xiaoqiao On Mon, Sep 1, 2025 at 3:19 AM Uma Maheswara Rao Gangumalla < umaganguma...@gmail.com> wrote: > +1 to close long pending PRs. > > I agree if the author

[DISCUSS] Propose to close stale PRs

2025-08-29 Thread Xiaoqiao He
Hi All, I try to launch this thread again, and propose to close stale PRs at Github which was updated over 100 days ago. There are 1180 open PRs at Github[1], which is unfriendly for reviewers or someone who wants to be involved to review here. I see that we have discussed many times, including

Re: [VOTE] Release Apache Hadoop 3.4.2

2025-08-25 Thread Xiaoqiao He
+1(binding). [Y] Verified sha512 checksum and signature. [Y] LICENSE files exist and NOTICE is included in source tarball. [Y] Build the source code on Ubuntu and OpenJDK 11 by `mvn clean package -DskipTests -Pnative -Pdist -Dtar`. [Y] Rat check is ok. `mvn clean apache-rat:check` [Y] Setup pseudo

Re: Question about Compatibility of HDFS Client Version 3.2.4 with HDFS Server 2.10.2

2025-08-18 Thread Xiaoqiao He
Alirezs Asgarian, Thanks for reaching out here. After checking protocol definition of 2.10.2 and 3.2.4 and IO package format, I think it could be feasible if we don't use the new RPC method like `getLocatedFileInfo`. Anyway, regression testing is necessary before deploying to the prod environment

Re: [VOTE] Release Apache Hadoop 3.4.2

2025-08-14 Thread Xiaoqiao He
Thanks Ahmar for your work on this. Will check it ASAP. Best Regards, - He Xiaoqiao On Thu, Aug 14, 2025 at 5:14 PM Ahmar Suhail wrote: > Hey all, > > Just a reminder about this. Would be great to get some votes in asap and > hopefully finalise 3.4.2. > > Thanks, > Ahmar > > On Thu, Aug 7, 2025

[jira] [Resolved] (HDFS-17811) EC: DFSStripedInputStream supports retrying just like DFSInputStream

2025-07-30 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17811. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed >

Re: [VOTE] Release Apache Hadoop 3.4.2

2025-05-28 Thread Xiaoqiao He
Thanks Ahmar for your work. Will try to verify and give feedback here. Best Regards, - He Xiaoqiao On Wed, May 28, 2025 at 8:56 PM Ahmar Suhail wrote: > > Hey all, > > The first release candidate for Hadoop 3.4.2 is now available for voting. > > There are a couple of things to note: > > 1/ No Ar

Re: [DISCUSS] Optimize the invoke retry of async repsonder when some nameservices are slow

2025-04-23 Thread Xiaoqiao He
+ hdfs-dev. Thanks Haobo for your proposal. As you mentioned above, this may be related to RBF and ARR features, right? IMO, it is necessary to improve responder performance, but I am a little confused about nameservice will slow down the whole system, The first glance is client or package size sh

[jira] [Resolved] (HDFS-17772) The JournaledEditsCache has an int overflow issue, causing the maximum capacity to always be Integer MAX_VALUE

2025-04-20 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17772. Fix Version/s: 3.5.0 (was: 3.4.2) Hadoop Flags: Reviewed

[jira] [Resolved] (HDFS-17760) Fix MoveToTrash throws ParentNotDirectoryException when there is a file inode with the same name in the trash

2025-04-10 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17760. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix MoveToTrash thr

[jira] [Resolved] (HDFS-17752) Host2DatanodeMap will not update when re-register a node with a different hostname

2025-04-10 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17752. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Host2DatanodeMap w

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-03-25 Thread Xiaoqiao He
hub.com:apache/hadoop-thirdparty.git > [git] * [new tag] release-1.4.0-RC1 -> release-1.4.0-RC1 > > > > On Mon, 24 Mar 2025 at 03:32, Xiaoqiao He wrote: >> >> Thanks Steve for driving this release. >> >> +0 and will give my +1 while tag to be fixed. >

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-03-23 Thread Xiaoqiao He
Thanks Steve for driving this release. +0 and will give my +1 while tag to be fixed. [Y] LICENSE files exist and NOTICE is included. [Y] Build the source code on Ubuntu and OpenJDK 11. [Y] Verified checksums and Signature. [Y] Skimmed over the contents of ReleaseNotes and ChangeLogs. [N] release

[jira] [Resolved] (HDFS-17744) [ARR] getEnclosingRoot RPC adapts to async rpc

2025-03-07 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17744. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > [ARR] getEnclosingR

[jira] [Resolved] (HDFS-17731) [ARR] Add unit test for async RouterAdminServer

2025-03-02 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17731. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Target Version/s: (was: 3.5.0

[jira] [Resolved] (HDFS-17745) TestRouterMountTable should reset defaultNSEnable to true after each test method.

2025-03-02 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17745. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > TestRouterMountTa

Re: [DISCUSS] Hadoop 3.4.2 release

2025-02-26 Thread Xiaoqiao He
+1. Thanks Ahmar. On Thu, Feb 27, 2025 at 2:31 AM Chris Nauroth wrote: > +1 > > Thank you, Ahmar! > > Chris Nauroth > > > On Wed, Feb 26, 2025 at 8:31 AM Ahmar Suhail wrote: > > > Hey all, > > > > Would like to start a discussion for a Hadoop 3.4.2 release. > > > > For S3A, we would like to rel

[RESULT][VOTE] Request to merge branch HDFS-17531 into trunk.

2025-02-19 Thread Xiaoqiao He
Addendum vote result: 4(binding) +1: Hui Fei, Ayush Saxena, Xiaoqiao He, Shilun Fan 3(non-binding) +1: Zhanghaobo, HuaLong Zhang, Jian Zhang Change to the formal title. Merged to trunk. Thanks all for your works. Best Regards, - He Xiaoqiao On Thu, Feb 20, 2025 at 11:42 AM jian zhang wrote

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

2025-02-14 Thread Xiaoqiao He
+1 Best Regards, - He Xiaoqiao On Fri, Feb 14, 2025 at 3:53 PM Ayush Saxena wrote: > +1 > > -Ayush > > > On 14 Feb 2025, at 1:20 PM, Hui Fei wrote: > > > > Did some test work referring to the documentation. > > > > - Compiled source codes, built a local cluster and the async feature > > w

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

2025-02-12 Thread Xiaoqiao He
slfan1989's PR: HADOOP-19415. [JDK17] Upgrade JUnit from 4 to 5 in > hadoop-common Part 1. (#7339). slfan1989 will fix it later. > > Best Regards, > - Jian Zhang > > Xiaoqiao He 于2025年2月11日周二 11:15写道: > > > Hi Jian Zhang, Thanks for your great work. Please fix

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

2025-02-10 Thread Xiaoqiao He
For the first opinion, I have create an umbrella JIRA >> https://issues.apache.org/jira/browse/HDFS-17716 >> and move non-core JIRA under it. >> >> Best Wishes >> Haobo Zhang >> >> Replied Message >> From Hui Fei >> Date 01/22/

[jira] [Resolved] (HDFS-17704) TestDecommission and TestDecommissionWithBackoffMonitor often run timeout.

2025-01-23 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17704. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > TestDecommission

[jira] [Resolved] (HDFS-17715) [ARR] Add documentation for asynchronous router.

2025-01-23 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17715. Hadoop Flags: Reviewed Resolution: Fixed > [ARR] Add documentation for asynchronous rou

[jira] [Resolved] (HDFS-17706) TestBlockTokenWithDFSStriped fails due to closed streams

2025-01-13 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17706. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed

[jira] [Resolved] (HDFS-17602) RBF: Fix mount point with SPACE order can not find the available namespace.

2025-01-07 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17602. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Fix mount po

Re: Discussion about NameNode Fine-grained locking

2025-01-05 Thread Xiaoqiao He
> From haiyang hu > Date 12/31/2024 23:08 > To Ayush Saxena > Cc Hui Fei , > ZanderXu , > Hdfs-dev , > , > Xiaoqiao He , > slfan1989 , > > Subject Re: Discussion about NameNode Fine-grained locking > Thanks for your hard work and push it forward

[jira] [Resolved] (HDFS-17683) Add metrics for acquiring dataset read/write lock

2025-01-05 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17683. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Add metrics

[jira] [Resolved] (HDFS-17496) DataNode supports more fine-grained dataset lock based on blockid

2025-01-02 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17496. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > DataNode supports m

[jira] [Resolved] (HDFS-17655) Cannot run HDFS balancer with BlockPlacementPolicyWithNodeGroup

2024-12-25 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17655. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Cannot run H

[jira] [Resolved] (HDFS-17648) Fix BalancerMetrics duplicate registration issue.

2024-12-08 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17648. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix BalancerMetr

Re: Compare metrics to detect performance issues in the latest versions

2024-11-10 Thread Xiaoqiao He
Thanks for your report. Change ML to hdfs-dev. Please first to choose common-dev, hdfs-dev or yarn-dev to discuss the issues. This ML is for security issues. Thanks. Best Regards, - He Xiaoqiao On Sun, Nov 10, 2024 at 2:26 AM 徐子康 wrote: > Hello! > > I am currently conducting a study to detect p

[jira] [Resolved] (HDFS-17607) Reduce the number of times conf is loaded when DataNode startUp

2024-11-04 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17607. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Reduce the number

[jira] [Resolved] (HDFS-17654) Fix bugs in TestRouterMountTable

2024-11-02 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17654. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix bugs

[jira] [Resolved] (HDFS-17646) Add Option to limit Balancer overUtilized nodes num in each iteration.

2024-11-02 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17646. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Add Option to li

[jira] [Resolved] (HDFS-17631) Fix RedundantEditLogInputStream.nextOp() state error when EditLogInputStream.skipUntil() throw IOException

2024-10-16 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17631. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed >

[jira] [Resolved] (HDFS-17626) Reduce lock contention at datanode startup

2024-09-28 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17626. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Reduce lock content

[jira] [Resolved] (HDFS-17624) Fix DFSNetworkTopology#chooseRandomWithStorageType() availableCount when excluded node is not in selected scope.

2024-09-27 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17624. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix DFSNetworkTopol

[INFO] branch HDFS-17531 rebase onto trunk

2024-09-12 Thread Xiaoqiao He
Hi All, Just rebase development feature branch HDFS-17531(RBF: Asynchronous router RPC) onto trunk, then keep branch HDFS-17531 closely with trunk. Thanks. Best Regards, - He Xiaoqiao

[jira] [Resolved] (HDFS-17573) Allow turn on both FSImage parallelization and compression

2024-08-25 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17573. Fix Version/s: (was: 3.4.1) Hadoop Flags: Reviewed Target Version/s: (was

[jira] [Resolved] (HDFS-16084) getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv() failed

2024-08-25 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-16084. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > getJNIEnv() retu

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

2024-08-22 Thread Xiaoqiao He
+1 (binding). Thanks Steve and Mukund for driving this release. Best Regards - He Xiaoqiao On Wed, Aug 21, 2024 at 11:51 PM Mukund Madhav Thakur wrote: > +1 ( binding) > > On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran > > wrote: > > > Mukund and I have built a release candidate (RC1) for Had

[jira] [Resolved] (HDFS-17544) [ARR] The router client rpc protocol PB supports asynchrony.

2024-08-02 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17544. Fix Version/s: HDFS-17531 Hadoop Flags: Reviewed Resolution: Fixed > [ARR] The rou

[jira] [Resolved] (HDFS-16690) Automatically format new unformatted JournalNodes using JournalNodeSyncer

2024-07-23 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-16690. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Automatically for

[jira] [Resolved] (HDFS-17566) Got wrong sorted block order when StorageType is considered.

2024-07-11 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17566. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Got wrong sorted bl

[jira] [Resolved] (HDFS-17564) EC: Fix the issue of inaccurate metrics when decommission mark busy DN

2024-07-05 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17564. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > EC: Fix the issue

Re: [DISCUSS] lots of yetus failures in hdfs

2024-06-07 Thread Xiaoqiao He
mzl61ow0sbs10p0hky17xxhsggbhc3g > [4] https://github.com/apache/hadoop/pull/6664#issuecomment-2082356393 > > On Fri, 7 Jun 2024 at 08:04, Xiaoqiao He wrote: > > > > Thanks Steve. Try to trigger CI manually and let's wait what it will say. > > BTW, the flaky tests seem not r

Re: [DISCUSS] lots of yetus failures in hdfs

2024-06-06 Thread Xiaoqiao He
Thanks Steve. Try to trigger CI manually and let's wait what it will say. BTW, the flaky tests seem not related to UT logic itself, but most of them throw OOM. Not sure if @Ayush Saxena knows how to re-config or tune the memory of Yetus? Best Regards, - He Xiaoqiao On Fri, Jun 7, 2024 at 3:59 AM

Re: [Discuss] RBF: Aynchronous router RPC.

2024-06-05 Thread Xiaoqiao He
Hi @zhangjian <1361320...@qq.com> , the dev branch HDFS-17531 is ready now. FYI. Best Regards, - He Xiaoqiao On Tue, Jun 4, 2024 at 10:08 PM zhangjian <1361320...@qq.com> wrote: > Hi, Xiaoqiao He: > Can you help create a dev branch? I don't have the permission to cre

Re: [Discuss] RBF: Aynchronous router RPC.

2024-05-29 Thread Xiaoqiao He
> On Fri, May 24, 2024 at 1:57 AM zhangjian <1361320...@qq.com> wrote: > >>>>> Hello everyone, currently, I have tested the performance of async > and sync router for a downstream ns: > >>>>> 1. The throughput, CPU, and thread performance of the a

Re: [DISCUSS] pruning surplus branches/tags from the main hadoop repo

2024-05-29 Thread Xiaoqiao He
Strong +1. One concerns, how we define 'unnecessary branches', which mean how to distinguish the branches someone accidentally created, I try to traverse some of them but didn't get one obvious rule. Thanks. Best Regards, - He Xiaoqiao On Wed, May 29, 2024 at 10:11 PM Ayush Saxena wrote: > +1

Re: [Discuss] RBF: Aynchronous router RPC.

2024-05-23 Thread Xiaoqiao He
gjin > > > > On Tue, May 21, 2024 at 5:13 AM zhangjian <1361320...@qq.com.invalid> > wrote: > > > >> Hi, xiaoqiao he, thank you for your reply. > >> > >> 1.Currently, the server and client protocols within router can be >

Re: [Discuss] RBF: Aynchronous router RPC.

2024-05-20 Thread Xiaoqiao He
Thanks for this great proposal! Some questions after reviewing the design doc (sorry didn't review PR carefully which is too large.) 1. This solution will involve RPC framework update, will it affect other modules and how to keep other modules off these changes. 2. Some RPC requests should be forw

[jira] [Resolved] (HDFS-17099) Fix Null Pointer Exception when stop namesystem in HDFS

2024-05-13 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17099. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix Null Poin

[jira] [Resolved] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection

2024-05-13 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17522. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > JournalNode

[jira] [Resolved] (HDFS-17500) Add missing operation name while authorizing some operations

2024-05-05 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17500. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Add missing operat

[jira] [Resolved] (HDFS-17456) Fix the incorrect dfsused statistics of datanode when appending a file.

2024-04-29 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17456. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix the incorr

Re: Discussion about NameNode Fine-grained locking

2024-04-28 Thread Xiaoqiao He
ts: >> > >> > 1. While ZanderXu and Hui Fei have deep expertise in HDFS and are familiar >> > with related development details, we still need more community member to >> > review the code to ensure that the relevant upgrades meet expectations. >> > >>

Re: Discussion about NameNode Fine-grained locking

2024-04-24 Thread Xiaoqiao He
cc private@h.a.o. On Wed, Apr 24, 2024 at 3:35 PM ZanderXu wrote: > > Here are some summaries about the first phase: > 1. There are no big changes in this phase > 2. This phase just uses FS lock and BM lock to replace the original global > lock > 3. It's useful to improve the performance, since s

[ANNOUNCE] New Hadoop Committer - Haiyang Hu

2024-04-21 Thread Xiaoqiao He
I am pleased to announce that Haiyang Hu has been elected as a committer on the Apache Hadoop project. We appreciate all of Haiyang's work, and look forward to her/his continued contributions. Congratulations and Welcome, Haiyang! Best Regards, - He Xiaoqiao (On behalf of the Apache Hadoop PMC)

[jira] [Resolved] (HDFS-17461) Fix spotbugs in PeerCache#getInternal

2024-04-12 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17461. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix spotbugs

[jira] [Resolved] (HDFS-17455) Fix Client throw IndexOutOfBoundsException in DFSInputStream#fetchBlockAt

2024-04-11 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17455. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix Client th

[jira] [Resolved] (HDFS-17368) HA: Standy should exit safemode when resources are from low available

2024-03-26 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17368. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > HA: Standy should e

[jira] [Resolved] (HDFS-17380) FsImageValidation: remove inaccessible nodes

2024-03-17 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17380. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > FsImageValidat

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

2024-03-13 Thread Xiaoqiao He
root root 2739952 Sep 19 13:09 >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1 >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40 >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3 >> >> Anyone got any insights on how I should set up this (debian-based) OS >> here? >>

[jira] [Resolved] (HDFS-17391) Adjust the checkpoint io buffer size to the chunk size

2024-03-12 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17391. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Adjust the checkpo

[jira] [Resolved] (HDFS-17422) Enhance the stability of the unit test TestDFSAdmin

2024-03-11 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17422. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Enhance the stabil

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

2024-03-11 Thread Xiaoqiao He
x27;ve > > verified they are good. > > > > Oh, and I've also got my raspberry p5 doing the download of the arm > > stuff for its checknative; not expecting problems. > > > > So: i've got some stuff still ongoing, but the core changes to packaging > &g

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

2024-03-11 Thread Xiaoqiao He
fixed in HADOOP-19057 > <https://issues.apache.org/jira/browse/HADOOP-19057>. > > -- > *From:* slfan1989 > *Sent:* Monday, March 4, 2024 10:07:05 PM > *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; < > priv...@hadoop.apache.org>

Re: Discussion about NameNode Fine-grained locking

2024-03-06 Thread Xiaoqiao He
ful. > > > > > > > > > > > > > > > > On Wed, Mar 6, 2024 at 10:51 AM Hui Fei > wrote: > > > > > > > > > Thanks for suggestions. > > > > > > > > > > Actually Started working on this improvement. And c

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

2024-03-06 Thread Xiaoqiao He
cc @Chao Sun @zhang...@apache.org @Ayush Saxena Would you mind helping to check the upstream systems(Spark, HBase, Hive) dependency with the new hadoop release version if necessary? Thanks. Best Regards, - He Xiaoqiao On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He wrote: > Shilun, Thanks

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

2024-03-06 Thread Xiaoqiao He
Shilun, Thanks for your great work. I am checking but not finished yet. Will confirm once done. Best Regards, - He Xiaoqiao On Thu, Mar 7, 2024 at 7:23 AM slfan1989 wrote: > @Xiaoqiao He @Ayush Saxena > @Steve > Loughran @Mukund Madhav Thakur > @Takanobu > Asanuma @M

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

2024-02-29 Thread Xiaoqiao He
files and run some test with it; this caught some >>> problems >>> > >> with exported slf4j and log4j2 artifacts getting into the >>> classpath. That >>> > >> is: hadoop-common pulling in log4j 1.2 and 2.x bindings. >>> > >> &g

Re: Discussion about NameNode Fine-grained locking

2024-02-26 Thread Xiaoqiao He
Thanks for this meaningful proposal. Some nit comments: a. Snapshot, Symbolic link and reserved feature are not mentioned at the design doc, should it be considered or different to this core design? b. For the benchmark result, what Read/Write request ratio? And do you meet any GC issues when reach

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

2024-02-05 Thread Xiaoqiao He
cc @PJ Fanning @Ayush Saxena @Steve Loughran @Takanobu Asanuma @Shuyan Zhang @inigo...@apache.org On Mon, Feb 5, 2024 at 11:30 AM Xiaoqiao He wrote: > +1(binding). > > I checked the following items: > - [X] Download links are valid. > - [X] Checksums and PGP signatures ar

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

2024-02-04 Thread Xiaoqiao He
10:29 PM slfan1989 wrote: > Hi folks, > > Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop > Thirdparty 1.2.0. > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC1 > > The RC tag is > ht

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

2024-02-01 Thread Xiaoqiao He
ds, > Shilun Fan. > > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He wrote: > > > Thanks Shilun for driving it and making it happen. > > > > +1(binding). > > > > [x] Checksums and PGP signatures are valid. > > [x] LICENSE files exist. > > [x] NOTICE

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

2024-01-30 Thread Xiaoqiao He
wrote: > Hi folks, > > Xiaoqiao He and I have put together a release candidate (RC0) for Hadoop > Thirdparty 1.2.0. > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0 > > The RC tag is > https://github.com/apache/

Re: [INFO] Hadoop-3.4 Release Update

2024-01-24 Thread Xiaoqiao He
Mon, 22 Jan 2024 at 11:50 AM, slfan1989 wrote: > > > >> Thank you very much for specifying the frozen time for the > >> Branch-3.4.0. @Xiaoqiao > >> He > >> > >> If there is a need to backport any PRs to branch-3.4/branch-3.4.0, > please &

Re: [INFO] Hadoop-3.4 Release Update

2024-01-21 Thread Xiaoqiao He
gt; > Best > > On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He > wrote: > >> Thanks all for your input. Connected to Shilun offline yesterday, and he >> will >> update or recover the fix version tag before 3.4.0-RC1. >> >> Best Regards, >> - He Xiao

Re: [INFO] Hadoop-3.4 Release Update

2024-01-21 Thread Xiaoqiao He
se versions have been released, > remove > > > the 3.4.0 entry from the fix version. > > > > > > 3. For JIRA issues with target version set to 3.4.0, fix version set to > > > 3.4.0, and status set to > > > RESOLVED, if Affects Ve

[INFO] Hadoop-3.4 Release Update

2024-01-18 Thread Xiaoqiao He
Hi All, Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is created[1]. Please set the proper fix version while committing jira, 3.5.0 should be priority. If there is any blocker/critical please also backport to branch-3.4 or sync to me or RM Shilun Fan. Shilun will start 3.4.0 RC1 vot

[jira] [Resolved] (HDFS-17182) DataSetLockManager.lockLeakCheck() is not thread-safe.

2024-01-03 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17182. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed

Re: [DISCUSS] Release Hadoop 3.4.0

2024-01-03 Thread Xiaoqiao He
> > I think the release discussion can be in public ML? Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML. Best Regards, - He Xiaoqiao On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena wrote: > +1 from me as well. > > We should definitely attempt to upgrade the thirdparty version for > 3.

Re: [VOTE] Hadoop 3.2.x EOL

2023-12-21 Thread Xiaoqiao He
Hi All, JIRA/WIKI/Website has been updated. Release line 3.2 is EOL now. Thanks all. Best Regards, - He Xiaoqiao On Tue, Dec 19, 2023 at 11:58 AM Xiaoqiao He wrote: > This vote passed with 10 +1. I'll update the JIRA, WIKI and website for a > while. > > Thanks all for y

Re: [DISCUSS][HDFS] Add rust binding for libhdfs

2023-12-20 Thread Xiaoqiao He
Thanks Xuanwo for your work. I believe it is valuable to enlarge hadoop ecosystem. I am also concerned that it will involve more hard work to release and version match, especially for one who is not familiar with C or Rust. Moreover, I am not aware the difference between `accept hdfs-sys as part o

Re: [VOTE] Hadoop 3.2.x EOL

2023-12-18 Thread Xiaoqiao He
; >> On Wed, Dec 6, 2023 at 8:39 AM Akira Ajisaka wrote: >> > >> > +1 >> > >> > >> > >> > On Wed, Dec 6, 2023 at 1:10 PM Xiaoqiao He wrote: >> > >> > > Dear Hadoop devs, >> > > >> > > Given the

[jira] [Resolved] (HDFS-17262) Transfer rate metric warning log is too verbose

2023-12-06 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17262. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Transfer rate met

[jira] [Resolved] (HDFS-17270) RBF: Fix ZKDelegationTokenSecretManagerImpl use closed zookeeper client to get token in some case

2023-12-06 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17270. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF:

[VOTE] Hadoop 3.2.x EOL

2023-12-05 Thread Xiaoqiao He
Dear Hadoop devs, Given the feedback from the discussion thread [1], I'd like to start an official thread for the community to vote on release line 3.2 EOL. It will include, a. An official announcement informs no further regular Hadoop 3.2.x releases. b. Issues which target 3.2.5 will not be fixe

Re: [DISCUSS] Make some release lines EOL

2023-12-05 Thread Xiaoqiao He
ease, as there's a clear story: we have a > > new 3.4.x line stabilising, if you want something already stable move > onto > > 3.3.x if you hadn't already > > > > > > > > > > > > On Mon, 4 Dec 2023 at 12:39, Xiaoqiao He wrote: > > &g

Re: [DISCUSS] Make some release lines EOL

2023-12-04 Thread Xiaoqiao He
> > -Ayush > > On Mon, 4 Dec 2023 at 18:09, Xiaoqiao He wrote: > > > > Hi folks, > > > > There are many discussions about which release lines should we still > > consider actively > > maintained in history. I want to launch this topic again, and

[jira] [Resolved] (HDFS-17250) EditLogTailer#triggerActiveLogRoll should handle thread Interrupted

2023-12-04 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17250. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > EditLogTai

[jira] [Resolved] (HDFS-17218) NameNode should process time out excess redundancy blocks

2023-12-04 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17218. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > NameNode sho

[DISCUSS] Make some release lines EOL

2023-12-04 Thread Xiaoqiao He
Hi folks, There are many discussions about which release lines should we still consider actively maintained in history. I want to launch this topic again, and try to get a consensus. >From download page[1] and active branches page[2], we have the following release lines: Hadoop 3.3 Release (relea

Re: [DISCUSS] Hadoop 3.4.0 release

2023-12-03 Thread Xiaoqiao He
Thanks Ahmar for starting this thread. Strong +1 to have 3.4.0 from trunk rather than from 3.3.*. We should keep release line clear, and it's not standard operation to cut a branch from another line. Let's move the trunk to 3.5.0-SNAPSHOT when we come to an agreement at this thread. About this rel

[jira] [Resolved] (HDFS-17063) Support to configure different capacity reserved for each disk of DataNode.

2023-11-16 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17063. Fix Version/s: 3.4.0 Resolution: Fixed > Support to configure different capacity reser

[jira] [Resolved] (HDFS-17172) Support FSNamesystemLock Parameters reconfigurable

2023-11-16 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17172. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Supp

  1   2   3   4   >