Re: [DISCUSS] Optimize the invoke retry of async repsonder when some nameservices are slow

+ hdfs-dev. Thanks Haobo for your proposal. As you mentioned above, this may be related to RBF and ARR features, right? IMO, it is necessary to improve responder performance, but I am a little confused about nameservice will slow down the whole system, The first glance is client or package size sh

[jira] [Resolved] (HDFS-17752) Host2DatanodeMap will not update when re-register a node with a different hostname

ill not update when re-register a node with a different > hostname > -- > > Key: HDFS-17752 > URL: https://issues.apache.org/jira/browse/HDFS-17752 >

Re: Fw: Data copy from HDFS to MinIO regularly

distcp can upload a directory tree of changed files, for cloud storage it looks for the different in file timestamps Otherwise, the HDFS namenode has an log4j audit logger: org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit This prints out all namenode filesystem operations in a structured

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

-1 Signatures look good, but I found a few spots that need an update for the new protobuf version: https://issues.apache.org/jira/browse/HADOOP-19510 https://github.com/apache/hadoop-thirdparty/pull/44 Chris Nauroth On Tue, Mar 18, 2025 at 8:47 AM Wei-Chiu Chuang wrote: > I just did > gpg -

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

thanks. will kick off a new build. Did anyone else see problems with the gpg versions? I destroyed the entire .gpgconf dir on the host i was building on; did manual verify after importing KEYS, and all seemed good. On Thu, 20 Mar 2025 at 04:37, Chris Nauroth wrote: > This is committed to tru

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

The time is up, the votes are in and they are +1 (binding) Ayush Saxena (binding) Chris Nauroth (binding) Steve Loughran (binding) Wei-Chiu Chuang (binding) Xiaoqiao He (binding) = 0: none -1: none Accordingly, the RC is the release; I will do the next steps in the process ASAP Steve On Tue,

Re: JDK 17 (or beyond) upgrade

Hello Vincent, There isn't a firm timeline yet, but the community is rallying around a 3.5.0 release with full support for JDK 17. In addition to the issue you linked, there is also this one: https://issues.apache.org/jira/browse/HADOOP-17177 Known work related to JDK 17 compatibility is going t

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Update: +1(binding). On Tue, Mar 25, 2025 at 7:10 PM Steve Loughran wrote: > > > pushed up. thanks for spotting that. Added another 3p. target to the > release-support build file. Really trying to automate all these steps > > 3p.git-tag-push: > [git] To github.com:apache/hadoop-thirdparty.

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

pushed up. thanks for spotting that. Added another 3p. target to the release-support build file. Really trying to automate all these steps 3p.git-tag-push: [git] To github.com:apache/hadoop-thirdparty.git [git] * [new tag] release-1.4.0-RC1 -> release-1.4.0-RC1 On Mon, 24 M

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Thanks Steve for driving this release. +0 and will give my +1 while tag to be fixed. [Y] LICENSE files exist and NOTICE is included. [Y] Build the source code on Ubuntu and OpenJDK 11. [Y] Verified checksums and Signature. [Y] Skimmed over the contents of ReleaseNotes and ChangeLogs. [N] release

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

+1 (Binding) * Built from source * Verified checksums * Verified Signature * Verified no diff b/w branch-1.4 & src tar [1] * Skimmed over the contents of ReleaseNotes & ChangeLog * Checked the NOTICE & LICENSE files Thanx Steve for driving the release. Good Luck!!! -Ayush [1] I think releas

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

+1 * Verified signatures. * LICENSE-binary has the expected dependencies and versions. * Site looks good too. * Specifically confirmed inclusion of HADOOP-19510, previously identified during RC0. Thanks for the release, Steve! Chris Nauroth On Fri, Mar 21, 2025 at 10:43 AM Steve Loughran wrot

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

I am +1 too. * Verified signature. * Release notes and changelogs look good. On Fri, Mar 21, 2025 at 4:35 PM Chris Nauroth wrote: > +1 > > * Verified signatures. > * LICENSE-binary has the expected dependencies and versions. > * Site looks good too. > * Specifically confirmed inclusion of HADOO

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

I did try everything, no problem with the keys or so. -Ayush On Thu, 20 Mar 2025 at 23:21, Steve Loughran wrote: > thanks. will kick off a new build. > > Did anyone else see problems with the gpg versions? > > I destroyed the entire .gpgconf dir on the host i was building on; did > manual verif

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

This is committed to trunk and branch-1.4.0 now, ready for another RC. Thanks to Aayush for the code review. Chris Nauroth On Wed, Mar 19, 2025 at 1:51 PM Chris Nauroth wrote: > -1 > > Signatures look good, but I found a few spots that need an update for the > new protobuf version: > > https:/

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

I just did gpg --verify .tar.gz.asc .tar.gz Ok after importing hadoop KEYS , this is the output which looks good to me: gpg: Signature made Fri Mar 14 08:56:01 2025 PDT gpg:using RSA key 38237EE425050285077DB57AD22CF846DBB162A0 gpg:

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Let me check that; getting signing set up on the transient vm was tricky as my yubikey one wasn't there so I used my other .. regarding the jersey -bigger piece off work to actually pick up, so not touching it. On Fri, 14 Mar 2025 at 21:46, Wei-Chiu Chuang wrote: > Looks like your GPG key expir

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

can you give me what command you ran to check the signature, and see if you can update all your keys from the central servers? My key thinks it is not expired, so maybe its the KEYS file i need to update or you need to refresh your keys. I will have a look at the keys file tomorrow On Fri, 14 Mar

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Verified SHA-512 checksum, GPG signature -- all good except that the signing key expired. Confirmed git tag release-1.4.0-RC0 and branch name branch-1.4.0 both maps to commit hash 5595f1357eaad02e1cfb660bcce7fd34515197ff. Confirmed it is released as 1.4.0. Noticed HADOOP-19075

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Looks like your GPG key expired? gpg: Signature made Fri Mar 14 08:56:00 2025 PDT gpg:using RSA key 38237EE425050285077DB57AD22CF846DBB162A0 gpg: Good signature from "Steve Loughran (ASF code sign key - 2018) < ste...@apache.org>" [expired] gpg: aka "[jpeg image of

Re: Can not run HDFS balancer cause metrics already exists

try to delete /system/balancer.id and search some error or warn logs in namenode. Replied Message | From | Sébastien Rebecchi | | Date | 3/9/2025 23:08 | | To | Zhanghaobo | | Cc | hadoop-user-maillist, hdfs-dev | | Subject | Re: Can not run HDFS balancer cause metrics already exists

Re: Can not run HDFS balancer cause metrics already exists

got it, you can use it as a service and see what will happen. Replied Message | From | Sébastien Rebecchi | | Date | 03/09/2025 22:22 | | To | Zhanghaobo | | Cc | u...@hadoop.apache.org、hdfs-dev@hadoop.apache.org | | Subject | Re: Can not run HDFS balancer cause metrics already exists

Re: Can not run HDFS balancer cause metrics already exists

Hi, @Sébastien Rebecchi Don't know more details about how you start balancer, did you use -asService? Replied Message | From | Sébastien Rebecchi | | Date | 3/9/2025 18:03 | | To | , | | Subject | Re: Can not run HDFS balancer cause metrics already exists | Hello

[jira] [Created] (HDFS-17752) Host2DatanodeMap will not update when re-register a node with a different hostname

WenjingLiu created HDFS-17752: - Summary: Host2DatanodeMap will not update when re-register a node with a different hostname Key: HDFS-17752 URL: https://issues.apache.org/jira/browse/HDFS-17752 Project

Re: [DISCUSS] Hadoop 3.4.2 release

+1, thanx Ahmar for driving it. I plan to c-pick YARN-11738 to branch-3.4, if folks are ok with it, that is need by Tez, or atleast makes life easier there -Ayush On Sun, 2 Mar 2025 at 01:30, Chao Sun wrote: > +1, thanks Ahmar! > > On Fri, Feb 28, 2025 at 7:21 AM larry mccay wrote: > > > +1 >

Re: [DISCUSS] Hadoop 3.4.2 release

+1, thanks Ahmar! On Fri, Feb 28, 2025 at 7:21 AM larry mccay wrote: > +1 > > Thanks, Ahmar! > > On Fri, Feb 28, 2025 at 7:37 AM slfan1989 wrote: > > > +1 > > > > I can provide assistance with the ARM release, as I originally planned to > > release Hadoop 3.5.0, which supports JDK17, and have b

Re: [DISCUSS] Hadoop 3.4.2 release

+1 Thanks, Ahmar! On Fri, Feb 28, 2025 at 7:37 AM slfan1989 wrote: > +1 > > I can provide assistance with the ARM release, as I originally planned to > release Hadoop 3.5.0, which supports JDK17, and have been conducting > related tests. > > My approach is a workaround, and I hope it helps. > >

Re: [DISCUSS] Hadoop 3.4.2 release

+1 I can provide assistance with the ARM release, as I originally planned to release Hadoop 3.5.0, which supports JDK17, and have been conducting related tests. My approach is a workaround, and I hope it helps. Best Regards, - Shilun Fan On Thu, Feb 27, 2025 at 9:08 PM Bhavik Patel wrote: > H

Re: [DISCUSS] Hadoop 3.4.2 release

Hello PMCs, Can we do a release for hadoop-thirdparty 1.4.0 or 1.3.1 with the fix for CVE-2024-7254(HADOOP-19289) and bump hadoop-thirdparty version in hadoop-3.4.2? On Thu, 27 Feb 2025, 5:15 pm Anuj Modi, wrote: > +1 > Thanks for initiative Ahmar. > > For ABFS we would like to highlight two ma

Re: [DISCUSS] Hadoop 3.4.2 release

+1 Thanks for initiative Ahmar. For ABFS we would like to highlight two major changes in the upcoming release. 1. Announcing Deprecation of WASB driver: Jira , Documentation

Re: [DISCUSS] Hadoop 3.4.2 release

+1. Thanks Ahmar. On Thu, Feb 27, 2025 at 2:31 AM Chris Nauroth wrote: > +1 > > Thank you, Ahmar! > > Chris Nauroth > > > On Wed, Feb 26, 2025 at 8:31 AM Ahmar Suhail wrote: > > > Hey all, > > > > Would like to start a discussion for a Hadoop 3.4.2 release. > > > > For S3A, we would like to rel

Re: [DISCUSS] Hadoop 3.4.2 release

+1 Thank you, Ahmar! Chris Nauroth On Wed, Feb 26, 2025 at 8:31 AM Ahmar Suhail wrote: > Hey all, > > Would like to start a discussion for a Hadoop 3.4.2 release. > > For S3A, we would like to release support for conditional writes, and > integration of a new analytics input stream. > > I hav

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

; common-...@hadoop.apache.org ; Hui Fei < > feihui.u...@gmail.com>; hfutzhan...@163.com > *Subject:* Re: [VOTE] Request to merge branch HDFS-17531 into trunk. > > +1 > > Best Regards, > - Shilun Fan > > On Fri, Feb 14, 2025 at 3:22 PM jian zhang wrote: > > > Hi, al

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

; Hui Fei ; hfutzhan...@163.com Subject: Re: [VOTE] Request to merge branch HDFS-17531 into trunk. +1 Best Regards, - Shilun Fan On Fri, Feb 14, 2025 at 3:22 PM jian zhang wrote: > Hi, all, the development of the asynchronous router functionality has been > completed. Thanks to a

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

feature. >> >> >> >> Best Regards, >> >> Jian Zhang >> >> >> >> Zhanghaobo 于2025年1月22日周三 18:22写道： >> >> >> >>> @Hui Fei Hi, Sir: >> >>> For the first opinion, I have create an umbrella JIRA >&

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

t point > does not exist, etc.), it directly puts the response into the response > queue; otherwise, it sends the request to the asynchronous handler thread > pool. > - Async Handler: Puts the request into the call queue > (connection.calls) of the connection thread and returns immediately w

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

>> >> >>> >> Best Regards, >>> >> Jian Zhang >>> >> >>> >> Zhanghaobo 于2025年1月22日周三 18:22写道： >>> >> >>> >>> @Hui Fei Hi, Sir: >>> >>> For the first opinion, I have create an umbr

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

+1 - Zhang Haobo Replied Message | From | Xiaoqiao He | | Date | 02/14/2025 19:38 | | To | Ayush Saxena | | Cc | Hui Fei , jian zhang , Hdfs-dev , , | | Subject | Re: [VOTE] Request to merge branch HDFS-17531 into trunk. | +1 Best Regards, - He Xiaoqiao On Fri, Feb 14, 2025

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

equests from the request queue for preliminary > >> processing. If there are exceptions in the request (such as the mount > point > >> does not exist, etc.), it directly puts the response into the response > >> queue; otherwise, it sends the request to the asynchr

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

response into the response >> queue; otherwise, it sends the request to the asynchronous handler thread >> pool. >> - Async Handler: Puts the request into the call queue >> (connection.calls) of the connection thread and returns immediately without >> blocking and waiting.

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

queue; otherwise, it sends the request to the asynchronous handler thread > pool. > - Async Handler: Puts the request into the call queue > (connection.calls) of the connection thread and returns immediately without > blocking and waiting. > - Async Responder: Is responsible for pr

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

t; Zhanghaobo 于2025年1月22日周三 18:22写道： >> >> >> >>> @Hui Fei Hi, Sir: >> >>> For the first opinion, I have create an umbrella JIRA >> >>> https://issues.apache.org/jira/browse/HDFS-17716 >> >>> and move non-core JIRA under i

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

16 > >>> and move non-core JIRA under it. > >>> > >>> Best Wishes > >>> Haobo Zhang > >>> > >>> Replied Message > >>> From Hui Fei > >>> Date 01/22/2025 17:37 > >>> To jian zhang > >>> Cc H

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

ve non-core JIRA under it. >>> >>> Best Wishes >>> Haobo Zhang >>> >>> Replied Message >>> From Hui Fei >>> Date 01/22/2025 17:37 >>> To jian zhang >>> Cc Hdfs-dev , >>> , >>>

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

For the first opinion, I have create an umbrella JIRA >> https://issues.apache.org/jira/browse/HDFS-17716 >> and move non-core JIRA under it. >> >> Best Wishes >> Haobo Zhang >> >> Replied Message >> From Hui Fei >> Date 01/22/

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

.org/jira/browse/HDFS-17716 > and move non-core JIRA under it. > > Best Wishes > Haobo Zhang > > Replied Message > From Hui Fei > Date 01/22/2025 17:37 > To jian zhang > Cc Hdfs-dev , > , > Xiaoqiao He , > > > Subject Re: [DISCUSS] Request

Re: [apache/hadoop] HDFS-17531. RBF: Aynchronous router RPC. (PR #7308)

Hi, slfan1989, Currently, this feature has been developed and all pipelines have passed. It is waiting for community voting before being merged into trunk. slfan1989 于2025年2月6日周四 16:55写道： > @KeeProMise @hfutatzhanghb > When is t

Re: Bug with ZStandardCodec in Hadoop,

Hello Sungwoo, It looks like HDFS-14099 was fixed in Apache Hadoop releases 3.2.3, 3.3.2 and 3.4.0. Users of Apache Hive 3.1.3 and earlier would be impacted. (Hive 3.1.3 used Hadoop 3.1.0.) However, Hive 4.0.0 used Hadoop 3.3.6, so I think we can consider this resolved by the latest Hive release.

Re: Bug with ZStandardCodec in Hadoop,

Thanks Sungwoo Park for sharing the details. I'm forwarding this to hdfs-dev@. I haven't had the chance to review the details in the ticket yet, but if you can reproduce the issue, I recommend creating an HDFS ticket and marking it as a blocker with the target versions set to the upcoming releases.

Re: Data not visible to new readers after SequenceFile.writer.hflush()

it's visible, but the namenode isn't updated so file.len() is still the same. if you go to the EOF and then try to read past it, you get the new data. I know, it's not "posix", but, well, view as eventual consistency on file length other thing to know: close() does an hflush, but not an hsync().

Re: Data not visible to new readers after SequenceFile.writer.hflush()

Hello Thank you for answering. As I understood hflush must make the data visible to new readers but this is not the case actually for performance reasons. Can it be considered a bug? When you say 'the writer would need to record the client side visible length', I don't see anything to do that fr

Re: Data not visible to new readers after SequenceFile.writer.hflush()

This is Hadoop's unit test for sequence file exercising hsync/hflush: https://github.com/apache/hadoop/blob/61df1b27a797efd094328c7d9141b9e157e01bf4/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestHSync.java#L151 On Mon, Jan 27, 2025 at 9:53 AM Wei-Chiu Ch

Re: Data not visible to new readers after SequenceFile.writer.hflush()

Do you have a unit test to reproduce? Note that for performance reasons, the actual HDFS hflush/hsync implementation does not update visible length at NameNode. The data is flushed to DataNode though. So the writer would need to record the client side visible length, and pass it to the reader, if

Re: Data not visible to new readers after SequenceFile.writer.hflush()

Hi Sébastien, I replied to your same question on the hadoop-user@ thread. Let's keep the discussion there unless we discover there is some kind of HDFS bug to discuss. Thank you! Chris Nauroth On Mon, Jan 27, 2025 at 6:31 AM Sébastien Rebecchi wrote: > Hello, > > I got this issue using hadoop

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

, , Xiaoqiao He , | | Subject | Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk. | Got your idea. Thank you! - How about removing unfinished tasks and placing them under a new task as subtasks, like ARR improvements? If this feature is completed but there are still some open tasks, it

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

ervices are >>>> not fully utilized, leading to unreasonable resource allocation. >>>> >>>> III. Design and Improvements of the Asynchronous Router >>>> >>>> The asynchronous router solves the above problems by redesigning >>>

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

process and introducing an asynchronous processing >>> mechanism. Its core improvements include: >>> >>> - Handler: Retrieves requests from the request queue for preliminary >>> processing. If there are exceptions in the request (such as the mount point >&

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

nt point >> does not exist, etc.), it directly puts the response into the response >> queue; otherwise, it sends the request to the asynchronous handler thread >> pool. >> - Async Handler: Puts the request into the call queue >> (connection.calls) of the connec

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

d > pool. > - Async Handler: Puts the request into the call queue > (connection.calls) of the connection thread and returns immediately without > blocking and waiting. > - Async Responder: Is responsible for processing the responses > received by the connection thread. I

Re: [VOTE] Merge HDFS-17384 to trunk

t; > To ZanderXu >> > Cc Hdfs-dev , >> > , >> > Ayush Saxena , >> > Xiaoqiao He , >> > slfan1989 , >> > Zhanghaobo , >> > , >> > >> > >> > Subject Re: [VOTE] Merge HDFS-17384 to trunk >> > +1 (binding) >> > Revie

Re: [VOTE] Merge HDFS-17384 to trunk

> Ayush Saxena , > > Xiaoqiao He , > > slfan1989 , > > Zhanghaobo , > > , > > > > > > Subject Re: [VOTE] Merge HDFS-17384 to trunk > > +1 (binding) > > Reviewed sub PRs > > It has been running for a long time on prod clusters > > > &g

Re: [VOTE] Merge HDFS-17384 to trunk

Fei > Date 01/13/2025 17:14 > To ZanderXu > Cc Hdfs-dev , > , > Ayush Saxena , > Xiaoqiao He , > slfan1989 , > Zhanghaobo , > , > > > Subject Re: [VOTE] Merge HDFS-17384 to trunk > +1 (binding) > Reviewed sub PRs > It has been running for a

Re: [VOTE] Merge HDFS-17384 to trunk

+1 (binding) It has been running stably for several months on prod clusters Replied Message | From | Hui Fei | | Date | 01/13/2025 17:14 | | To | ZanderXu | | Cc | Hdfs-dev , , Ayush Saxena , Xiaoqiao He , slfan1989 , Zhanghaobo , , | | Subject | Re: [VOTE] Merge HDFS-17384 to trunk

Re: [VOTE] Merge HDFS-17384 to trunk

+1 (binding) Reviewed sub PRs It has been running for a long time on prod clusters ZanderXu 于2025年1月13日周一 16:35写道： > Hi all, > > I would like to propose that HDFS-17384 to be merged to trunk: > >- PR Link: PR#6762 - HDFS-17384: [FGL] HDFS NameNode FGL Phase I >

Re: Discussion about NameNode Fine-grained locking

A0%E6%B5%A9%E5%8D%9A&uid=hfutzhanghb%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc804eb39b0e7885aa8801c3bb66e497d.jpg&items=%5B%22hfutzhanghb%40163.com%22%5D> >> >> Replied Message ---- >> From haiyang hu >> Date 12/31/2024 23:08 >> To Ay

Re: Discussion about NameNode Fine-grained locking

> From haiyang hu > Date 12/31/2024 23:08 > To Ayush Saxena > Cc Hui Fei , > ZanderXu , > Hdfs-dev , > , > Xiaoqiao He , > slfan1989 , > > Subject Re: Discussion about NameNode Fine-grained locking > Thanks for your hard work and push it forward

Re: Discussion about NameNode Fine-grained locking

Thanks for your hard work and push it forward. It looks good, +1 for merging phase 1 codes, hope we can work together to promote this major HDFS optimization, so that more companies can benefit from it. Thanks everyone~ Ayush Saxena 于2024年12月31日周二 20:33写道： > +1, > Thanx folks for your efforts o

Re: Discussion about NameNode Fine-grained locking

+1, Thanx folks for your efforts on this! I didn't have time to review everything thoroughly, but my initial pass suggests it looks good or atleast is safe to merge. If I find some spare time, I'll test it further and submit a ticket or so if I encounter any issues. Good Luck!!! -Ayush On Tue, 3

Re: Discussion about NameNode Fine-grained locking

Thanks Zander for bringing this discussion again and trying your best to push it forward. It's really a long time since last discussion. It’s indeed time, +1 for merging phase 1 codes based on the following points - The phase 1 feature has been running at scale within companies for a long time -

Re: Discussion about NameNode Fine-grained locking

Hi, everyone: Time to Merge FGL Phase I The PR for *FGL Phase I* is ready for merging! Please take a moment to review and cast your vote: https://github.com/apache/hadoop/pull/6762. The *FGL Phase I* has been running successfully in production for over six months at *Shopee* and *BOSS Zhipin*, wi

Re: Compare metrics to detect performance issues in the latest versions

Thanks for your report. Change ML to hdfs-dev. Please first to choose common-dev, hdfs-dev or yarn-dev to discuss the issues. This ML is for security issues. Thanks. Best Regards, - He Xiaoqiao On Sun, Nov 10, 2024 at 2:26 AM 徐子康 wrote: > Hello! > > I am currently conducting a study to detect p

Re: [External Sender] Re: JDK17 support for Hadoop

; Asanuma , Hadoop Common , > yarn-dev , mapreduce-dev < > mapreduce-...@hadoop.apache.org>, Hdfs-dev > *Subject: *[External Sender] Re: JDK17 support for Hadoop > > Thank you all for your interest in this topic! > > In my opinion, upgrading is very beneficial. Based on

Re: [External Sender] Re: JDK17 support for Hadoop

, Hadoop Common , yarn-dev , mapreduce-dev , Hdfs-dev Subject: [External Sender] Re: JDK17 support for Hadoop Thank you all for your interest in this topic! In my opinion, upgrading is very beneficial. Based on the roadmap for Spark 4.0, we can see that JDK8/JDK11 is no longer the default version

Re: [INFO] branch HDFS-17531 rebase onto trunk

Hi Xiaoqiao He Thank you for taking the time to do this work! Best Regards, - Jian Zhang Xiaoqiao He 于2024年9月12日周四 15:54写道： > Hi All, > > Just rebase development feature branch HDFS-17531(RBF: Asynchronous router > RPC) onto trunk, > then keep branch HDFS-17531 closely with trunk. Thanks. > > B

Re: HDFS-17611. Move all DistCp execution logic to execute()

shadedclient error looks concerning. Please do not merge until you build it locally successfully without -DskipShade option On Thu, Sep 5, 2024 at 1:21 PM Steve Loughran wrote: > before I merge, has anyone here see any problems from this PR? > https://github.com/apache/hadoop/pull/7025 >

Re: Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

+1 (binding) * Successfully built from source with `mvn clean install` * Verified signatures and checksums Thanks Steve and Mukund for driving this release. Hope the release goes smoothly. Shilun Fan. On Fri, Aug 23, 2024 at 11:41 PM Shilun Fan wrote: +1 (Binding) > > * Built from source > *

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

+1 (Binding) * Built from source * Verified Signature(Signed by Mukund) * Verified Checksums * No diff b/w the git tag & src tar * Verified the NOTICE & LICENSE files * Skimmed over the contents of the site jar * Built Hadoop trunk with 1.3.0 version * Checked the hadoop test results here [1], not

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

+1 (binding). Thanks Steve and Mukund for driving this release. Best Regards - He Xiaoqiao On Wed, Aug 21, 2024 at 11:51 PM Mukund Madhav Thakur wrote: > +1 ( binding) > > On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran > > wrote: > > > Mukund and I have built a release candidate (RC1) for Had

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

de-protobuf) on > project hadoop-shaded-protobuf_3_25: Error creating shaded jar: duplicate > entry: META-INF/LICENSE.txt -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

--- > > mvn install package -DskipTests > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (shade-protobuf) on > project hadoop-shaded-protobuf_3_25: Error creating shaded jar: duplicate > entry: META

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

project hadoop-shaded-protobuf_3_25: Error creating shaded jar: duplicate entry: META-INF/LICENSE.txt -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For m

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

+1 ( binding) On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran wrote: > Mukund and I have built a release candidate (RC1) for Hadoop-Thirdparty > 1.3.0. > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.3.0-RC1/ > > The git tag is release-1.3.0-RC1, co

Re: [DISCUSS] what needs to be done to switch to java17

> One thing I know is that it is necessary to upgrade JUnit 4 to JUnit 5 Do you know any specific reason or docs mentioning that? Just curious as to why we need to migrate to JUnit 5 for JDK-17, Looking here [1], it seems like it supports JDK-21 as well and even [2] mentions like it does support

Re: [DISCUSS] what needs to be done to switch to java17

Thanks, everyone, for starting this discussion. This sounds like a good plan to start with. As I was working on the JUnit 4 to 5 upgrade, I paused for a while as I got occupied with other stuff. But I would be happy to complete it as part of the Java switch process. On Tue, Jul 30, 2024 at 4:21 AM

Re: [DISCUSS] what needs to be done to switch to java17

Thank you very much for initiating this discussion! I am also very much looking forward to JDK 17. I have observed that using --add-opens= is quite common in other Apache projects. One thing I know is that it is necessary to upgrade JUnit 4 to JUnit 5. There are some JIRA issues currently being wor

Re: [DISCUSS] what needs to be done to switch to java17

Depends on what we mean by switching to JDK-17, Compile time support or Runtime support, We don't have compile time support for JDK-11 too, it is just runtime, We have a daily build as well for JDK-11, which has now some genuine failures now, need to check [1], as of now the version is hardcoded he

Re: [DISCUSS] what needs to be done to switch to java17

I think this is worth considering. I think it would require a minor release like 3.5.0 as opposed to considering it for future 3.4.x patch releases. I tend to build locally with Java 11, by default and I haven't hit major issues building Hadoop. There may be some gotcha somewhere but it is likely t

Re: [Discuss] RBF: Aynchronous router RPC.

66e497d.jpg&items=%5B%22hfutzhanghb%40163.com%22%5D> > Replied Message > From zhangjian<1361320...@qq.com> <mailto:1361320...@qq.com> > Date 06/19/2024 11:03 > ToZhanghaobo <mailto:hfutzhan...@163.com> > Subject Re: [Discuss] RBF: Ay

Re: [Discuss] RBF: Aynchronous router RPC.

...@163.com | Replied Message | From | zhangjian<1361320...@qq.com.INVALID> | | Date | 06/6/2024 11:49 | | To | Xiaoqiao He | | Cc | Hadoop Common , Hdfs-dev , | | Subject | Re: [Discuss] RBF: Aynchronous router RPC. | Hi, xiaoqiao He, thank you so much! Best Regards, - zhangjian

Re: [DISCUSS] lots of yetus failures in hdfs

f9bceddec/hadoop-project/pom.xml#L172 > >> [2] > https://github.com/apache/hadoop/blob/2ee0bf953492b66765d3d2c902407fbf9bceddec/dev-support/docker/Dockerfile#L77 > >> [3] https://lists.apache.org/thread/hmzl61ow0sbs10p0hky17xxhsggbhc3g > >> [4] https://github.com/apache

Re: [DISCUSS] lots of yetus failures in hdfs

ttps://lists.apache.org/thread/hmzl61ow0sbs10p0hky17xxhsggbhc3g >> [4] https://github.com/apache/hadoop/pull/6664#issuecomment-2082356393 >> >> On Fri, 7 Jun 2024 at 08:04, Xiaoqiao He wrote: >> > >> > Thanks Steve. Try to trigger CI manually and let's wait

Re: [DISCUSS] lots of yetus failures in hdfs

53492b66765d3d2c902407fbf9bceddec/dev-support/docker/Dockerfile#L77 >> [3] https://lists.apache.org/thread/hmzl61ow0sbs10p0hky17xxhsggbhc3g >> [4] https://github.com/apache/hadoop/pull/6664#issuecomment-2082356393 >> >> On Fri, 7 Jun 2024 at 08:04, Xiaoqiao He wrote: >> &

Re: [DISCUSS] lots of yetus failures in hdfs

mzl61ow0sbs10p0hky17xxhsggbhc3g > [4] https://github.com/apache/hadoop/pull/6664#issuecomment-2082356393 > > On Fri, 7 Jun 2024 at 08:04, Xiaoqiao He wrote: > > > > Thanks Steve. Try to trigger CI manually and let's wait what it will say. > > BTW, the flaky tests seem not r

Re: [DISCUSS] lots of yetus failures in hdfs

t; Thanks Steve. Try to trigger CI manually and let's wait what it will say. > BTW, the flaky tests seem not related to UT logic itself, but most of them > throw OOM. Not sure if @Ayush Saxena knows how to re-config or tune > the memory of Yetus? > > Best Regards, > - He Xiaoqi

Re: [DISCUSS] lots of yetus failures in hdfs

Thanks Steve. Try to trigger CI manually and let's wait what it will say. BTW, the flaky tests seem not related to UT logic itself, but most of them throw OOM. Not sure if @Ayush Saxena knows how to re-config or tune the memory of Yetus? Best Regards, - He Xiaoqiao On Fri, Jun 7, 2024

Re: [Discuss] RBF: Aynchronous router RPC.

Hi, xiaoqiao He, thank you so much! Best Regards, - zhangjian > 2024年6月6日 11:33，Xiaoqiao He 写道： > > Hi @zhangjian <1361320...@qq.com> , the dev branch HDFS-17531 is ready now. > FYI. > > Best Regards, > - He Xiaoqiao > > On Tue, Jun 4, 2024 at 10:08 PM zhangjian <1361320...@qq.com> wrote: > >

Re: [Discuss] RBF: Aynchronous router RPC.

Hi @zhangjian <1361320...@qq.com> , the dev branch HDFS-17531 is ready now. FYI. Best Regards, - He Xiaoqiao On Tue, Jun 4, 2024 at 10:08 PM zhangjian <1361320...@qq.com> wrote: > Hi, Xiaoqiao He: > Can you help create a dev branch? I don't have the permission to create > it. > > Thank you ver

Re: Discussion about NameNode Fine-grained locking

I plan to hold a meeting on 2024-06-06 from 3:00 PM - 4:00 PM to share the FGL's motivations and some concerns in detail in Chinese. The doc is : NameNode Fine-Grained Locking Based On Directory Tree (II)

Re: [Discuss] RBF: Aynchronous router RPC.

Hi, thank you all for supporting this proposal! Recently, I will split this huge PR to small PRs. Thank you again! - zhangjian > 2024年5月30日 11:57，Xiaoqiao He 写道： > > Great! It looks like there are no other nothing blockers. > > @zhangjian <1361320...@qq.com> If no other furthermore comments, w

Re: [DISCUSS] pruning surplus branches/tags from the main hadoop repo

On Thu, 30 May 2024 at 03:47, Xiaoqiao He wrote: > Strong +1. One concerns, how we define 'unnecessary branches', which > mean how to distinguish the branches someone accidentally created, I try > to traverse some of them but didn't get one obvious rule. Thanks. > If we take a snapshot into an

  1   2   3   4   5   6   7   8   9   10   >