Re: [VOTE] Release Apache Hadoop 3.3.1 - RC1

review the hadoop-thirdparty 1.1.1 RC0 and cast a vote. Thanks! On Wed, May 26, 2021 at 9:39 AM Wei-Chiu Chuang wrote: > > Found a few issues building downstreams > > Ozone: HADOOP-17730 <https://issues.apache.org/jira/browse/HADOOP-17730> -- > we may have to mak

Re: [VOTE] Release Apache Hadoop Thirdparty 1.1.1 RC0

ecked the documents >> - Compiled Hadoop trunk and branch-3.3 with Hadoop third-party 1.1.1. >> >> -Akira >> >> On Wed, May 26, 2021 at 5:29 PM Wei-Chiu Chuang >> wrote: >> > >> > Hi folks, >> > >> > I have put together a release

Re: [VOTE] Release Apache Hadoop Thirdparty 1.1.1 RC0

Compiled Hadoop trunk and branch-3.3 with Hadoop third-party 1.1.1. >> >> -Akira >> >> On Wed, May 26, 2021 at 5:29 PM Wei-Chiu Chuang >> wrote: >> > >> > Hi folks, >> > >> > I have put together a release candidate (RC0) for

[VOTE] Release Apache Hadoop 3.3.1 RC2

Hi community, This is the release candidate RC2 of Apache Hadoop 3.3.1 line. All blocker issues have been resolved [1] again. There are 4 issues resolved for RC2: * HADOOP-17666. Update LICENSE for 3.3.1 * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053) * Revert "HADOOP-17563. U

Re: [VOTE] Release Apache Hadoop 3.3.1 RC2

Cancelling RC2. @Ayush Saxena found an issue HADOOP-16878 <https://issues.apache.org/jira/browse/HADOOP-16878> which we need to revert. I'll work on the RC3 asap. On Tue, Jun 1, 2021 at 3:23 PM Wei-Chiu Chuang wrote: > Hi community, > > This is the release candidate RC2 of

Do I need a +1 to merge a backport PR?

I'm curious about the GitHub PR conventions we use today... say I want to backport a commit from trunk to branch-3.3, and there's a small code conflict so I push a PR against branch-3.3 using GitHub to go through the precommit check. Do I need explicit approval from another committer to merge the

Re: [VOTE] Hadoop 3.1.x EOL

+1 On Thu, Jun 3, 2021 at 2:14 PM Akira Ajisaka wrote: > Dear Hadoop developers, > > Given the feedback from the discussion thread [1], I'd like to start > an official vote > thread for the community to vote and start the 3.1 EOL process. > > What this entails: > > (1) an official announcement t

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

faster by 20% over Hadoop 3.2 library. Looks like we may need some more time to test. How about extending it by a week? On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang wrote: > Hi community, > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker > issu

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

e release is a > big release, but I get that it’s the current state of the project. > > > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang wrote: > > > > Hello, > > do we want to extend the release vote? I understand a big release like > this > > takes time to val

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

doesn't render right. >> Created https://issues.apache.org/jira/browse/HADOOP-17746. >> >> This is *not a blocker* >> >> I can prepare a fix and we can have it in so that if any other changes >> come >> in the page will look OK. >> >>

Heads up! Merging HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet

Hello, Just want to make sure everyone interested in HDFS dev is aware that HDFS-13671 (Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet) will soon to be merged. The folded tree set data structure introduced in Hadoop 3 has big performance regression when delete files. Since

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

> > > +1, >> > > Built from Source. >> > > Successful Native Build on Ubuntu 20.04 >> > > Verified Checksums >> > > Ran basic hdfs shell commands. >> > > Ran simple MR jobs. >> > > Browsed NN,DN,RM and NM UI. >> > > >> &g

Re: [VOTE] Hadoop 3.1.x EOL

Dropped 3.1.4 from website. Removed from https://dist.apache.org/repos/dist/release/hadoop/common/ On Thu, Jun 10, 2021 at 3:42 PM Akira Ajisaka wrote: > This vote has passed with 18 binding +1. I'll update the JIRA and the wiki. > > Thanks all for your participation. > > On Tue, Jun 8, 2021 at

[ANNOUNCE] Apache Hadoop 3.3.1 release

olks who continued helps for this release process. Best Regards, Wei-Chiu Chuang

LinkedIn’s journey of scaling the Hadoop Distributed File System

Our friends at LinkedIn published an awesome blog post on the HDFS scalability. Check it out here: https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr Thanks, Wei-Chiu

Three Hadoop talks in this year's ApacheCon Asia

For your information, While drafting the upcoming quarterly report, I found there are three talks that are directly related to Hadoop in this year's ApacheCon Asia. https://apachecon.com/acasia2021/tracks/bigdata.html Bigtop 3.0: Rerising community driven Hadoop distribution

[DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

I work on multiple projects and learned a bunch from those projects.There are nice add-ons that help with productivity. There are things we can do to help us manage the project better. 1. Add new issue types. We can add "Epic" jira type to organize a set of related jiras. This could be easier to m

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

Here's another one: We can update the JIRA workflow and add more state. For example, Cassandra has "Review in Progress" and "Need Reviewer" states. INFRA-22049 <https://issues.apache.org/jira/browse/INFRA-22049> On Tue, Jul 13, 2021 at 11:34 AM Wei-Chiu Chuan

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

ew GitHub webhooks? > - Just for discussion: is Integrating SonarQube worth the effort of > supporting the GitHub webhooks. SonarQube >is a double edge weapon and it needs dedicated man hours to weed out > the way through the reports. > > On Mon, Jul 12, 2021 at 11:18 PM Wei-

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

One more idea that I forgot to mention: We can have the Dependabot to automatically create a pull request whenever a vulnerability is discovered. On Wed, Jul 14, 2021 at 8:42 AM Wei-Chiu Chuang wrote: > So this is just up for discussion and welcome more brainstorming. I'm not > say

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

A, always use GitHub PR. It saves > > costs to configure and debug the precommit jobs. > > https://issues.apache.org/jira/browse/HADOOP-17798 > > 2. Improve the pull request template for the contributors > > https://issues.apache.org/jira/browse/HADOOP-17799 > > > >

Re: Hadoop-3.2.3 Release Update

Hi to raise the awareness, it looks like reverting the FoldedTreeSet HDFS-13671 breaks TestBlockManager in branch-3.2. Branch-3.3 is good. tracking jira: HDFS-16258 On Tue, Oct 5, 2021 at 8:45 P

ApacheCon@Home Big Data tracks recordings!

For those who missed the live Apache@Home Big Data tracks, the video recordings are being uploaded to the official ASF channel! Big Data: https://www.youtube.com/playlist?list=PLU2OcwpQkYCzXcumE9UxNirLF1IYLmARj Big Data Ozone: https://www.youtube.com/playlist?list=PLU2OcwpQkYCxtPdZ0nSowYLQMgkmoczM

Next Hadoop community online meetup (Mandarin)

Hello HDFS devs After a long hiatus, I'm happy to share that we'll have Yiyang from Shopee who is going to talk about their experience of using HDFS at Shopee. This is a Mandarin talk. 题目：《HDFS在Shopee的演进》 概要：HDFS作为最底层的存储架构，在Shopee的业务发展中起到了重要支撑作用。随着Shopee业务的扩展，HDFS存储服务也面临了越来越多的挑战。本次分享主要介绍在面临这些挑战时

Re: Next Hadoop community online meetup (Mandarin)

Hello, this talk will start in about an hour. (I'm sorry forgot to add this is 2PM Beijing time) On Thu, Dec 9, 2021 at 11:00 AM Takanobu Asanuma wrote: > What is the timezone? > > 2021年11月29日(月) 14:50 Wei-Chiu Chuang : > >> Hello HDFS devs >> >> After a l

Re: Next Hadoop community online meetup (Mandarin)

We are having a little technical difficulties. For those who are late, we are using this link instead: https://cloudera.zoom.us/j/93901387554 On Thu, Dec 9, 2021 at 12:55 PM Takanobu Asanuma wrote: > Thanks for the information! > > 2021年12月9日(木) 13:52 Wei-Chiu Chuang : > >>

Re: Next Hadoop community online meetup (Mandarin)

's in English so the wider audience can read) Please find today's recording here: https://cloudera.zoom.us/rec/share/MOwRDPP50eL7gRfNazfjqAXdl_7HwAaPH460sONmmlx_tLQrftVjjjCBkVvE-9sf.GRb9Kp_16xOqgzNu Passcode: L^n30&%d On Thu, Dec 9, 2021 at 2:10 PM Wei-Chiu Chuang wrote: >

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC0

Thanks a lot for pushing it forward! A few things I noticed that we should incorporate: 1. the overview page of the doc is for the Hadoop 3.0 release. It would be best to base the doc on top of Hadoop 3.3.0 overview page. (it's a miss on my part... The overview page of 3.3.1 wasn't updated) 2. A

Trunk broken by HDFS-16384

My bad. There was a transitive dependency problem in the PR causing trunk to fail the build. The commit has since been reverted. Sorry for the inconvenience.

Apache Hadoop and CVE-2021-44228 Log4JShell vulnerability

Hi, Given the widespread attention to the recent log4j vulnerability (CVE-2021-44228), I'd like to share an update from the Hadoop developer community regarding the incident. As you probably know, Apache Hadoop depends on the log4j library to keep log files. The highlighted vulnerability CVE-2021-

Next Mandarin Hadoop Online Meetup Jan 6th.

Hello community, This week we'll going to have Tao Li (tomscut) speaking about the experience of operating HDFS at BIGO. See you on Thursday! 题目：《HDFS在BIGO的实践》 概要：HDFS作为大数据底层存储服务，在BIGO的发展中起到了非常重要的作用。随着业务的发展和数据的爆发式增长，HDFS单个集群的瓶颈愈发凸显，我们借助Router，将多个HDFS集群整合成一个Namespace，增强集群的扩展能力；改造Router，使其支持Alluxi

Re: Next Mandarin Hadoop Online Meetup Jan 6th.

Just a gentle reminder this is happening now. On Mon, Jan 3, 2022 at 5:39 PM Wei-Chiu Chuang wrote: > Hello community, > > This week we'll going to have Tao Li (tomscut) speaking about the > experience of operating HDFS at BIGO. See you on Thursday! > > 题目：《HDFS在BIGO的实践》

Re: Next Mandarin Hadoop Online Meetup Jan 6th.

/JaNm70lZQGCZdlFzh9ZbsfrR7MJ7Nazb2g6NCtYPqsRLWtyEhLfgwXOppzMR3csp.HqRJNGXUGSaPu1qw Access Passcode: 4g1ZF&%f On Mon, Jan 3, 2022 at 5:39 PM Wei-Chiu Chuang wrote: > Hello community, > > This week we'll going to have Tao Li (tomscut) speaking about the > experience of operating HDFS at BIGO. See you on Thursday! > > 题目：《HDF

Re: Hadoop-3.2.3 Release Update

Is this still making progress? On Tue, Oct 5, 2021 at 8:45 PM Brahma Reddy Battula wrote: > Hi Akira, > > Thanks for your email!! > > I am evaluating the CVE’s which needs to go for this release.. > > Will update soon!! > > > On Tue, 5 Oct 2021 at 1:46 PM, Akira Ajisaka wrote: > > > Hi Brahma,

Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2

+1 I think it makes sense to use reload4j in maint releases. I have a draft PR doing this (https://github.com/apache/hadoop/pull/3906) log4j2 in Hadoop 3.4.0 makes sense to me. There could be incompatibilities introduced by log4j2, but I feel we should at least make it 3.4.0 a "preview" release, a

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2

I'll find time to check out the RC bits. I just feel bad that the tarball is now more than 600MB in size. On Fri, Jan 21, 2022 at 2:23 AM Steve Loughran wrote: > *+1 binding.* > > reviewed binaries, source, artifacts in the staging maven repository in > downstream builds. all good. > > *## test

Re: [ANNOUNCE] Apache Hadoop 3.3.2 release

> > Many thanks to Viraj Jasani, Michael Stack, Masatake Iwasaki, Xiaoqiao He, > Mukund Madhav Thakur, Wei-Chiu Chuang, Steve Loughran, Akira Ajisaka and > other folks who helped for this release process. > > Best Regards, > Chao >

[ANNOUNCE] New Hadoop PMC Sun Chao

On behalf of the Apache Hadoop PMC, I am pleased to announce that Sun Chao(sunchao) has accepted the PMC's invitation to become a PMC on the project. We appreciate all of Sun's generous contributions thus far and look forward to his continued involvement. Congratulations and welcome, Sun!

Re: [VOTE] Release Apache Hadoop 3.2.3 - RC0

aarch64 support is only introduced in/after 3.3.0 On Thu, Mar 17, 2022 at 2:27 PM Emil Ejbyfeldt wrote: > Hi, > > > There is no aarch64 artifact in the release candidate. Is this something > that is intended? > > Best, > Emil Ejbyfeldt > > On 14/03/2022 08:14, Masatake Iwasaki wrote: > > Hi all,

Re: Institutions running kerberized Hadoop clusters

Last time I checked (~2 years ago), there were thousands of Kerberized clusters among Cloudera customers. The largest ones had a few thousand nodes. What are you looking for? On Wed, Apr 13, 2022 at 7:22 AM Santosh Marella wrote: > Hey folks, > > Just curious if we have a list of institutions

Re: [DISCUSS] Hadoop on Windows

Great! Sorry I missed the earlier discussion thread. Is there a target version for this support? I assume the milestone is still in a dev branch? On Thu, Apr 28, 2022 at 8:26 AM Gautham Banasandra wrote: > Hi Hadoop devs, > > I would like to announce that we recently reached a new milestone - w

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

Running builds for all platforms for each and every PR seems too excessive. How about doing all platform builds in the nightly jobs? On Fri, May 6, 2022 at 8:02 AM Steve Loughran wrote: > I'm not enthusiastic here as it not only makes the builds slower, it > reduces the #of builds we can throug

Hadoop BoF at ApacheCon NA'22

We've not had a physical event for a long long time and we're way overdue for one. I'm excited to announce we've reserved a room at the upcoming ApacheCon for Birds-of-Feather on October 4th from 17:50-18:30 CDT in Rhythms I. I was also told that participants can stay after that until the hotel pe

Re: Hadoop BoF at ApacheCon NA'22

document/d/1_ha1BFeEyIkAJtl5tJ8Z4Vf5Md_0rN23RCrXkyNCY1c/edit?usp=sharing Please add more details here. On Thu, Sep 22, 2022 at 10:27 PM 俊平堵 wrote: > Thanks Wei-Chiu. I am happy to share the status of Hadoop Meetups in China > (2019-2022) if that is a suitable topic. :) > > Thanks, > > Junping

Re: Hadoop BoF at ApacheCon NA'22

t; On Sat, Sep 24, 2022 at 8:52 PM 俊平堵 wrote: > > > > > Yes. It should be a short talk and I can only present remotely for this > > > time. That would be great if you help to coordinate. :) > > > > > > Thanks, > > > > > > Junping &g

[DISCUSS] Hadoop 3.3.5 release planning

Bumping this up. Adding the [DISCUSS] text to make this message stand out of your inbox. I certainly missed this message and only realized 3.3.5 has more than just security updates. What was the issue with the ARM64 build? I was able to publish ARM64 build for 3.3.1 release without problems. On

[DISCUSS] Supporting partial file rewrite/compose

There were a number of discussions that happened during ApacheCon. In the spirit of the Apache Way, I am taking the conversation online, sharing with the larger community and also capturing requirements. Credits to Owen who started this discussion. There are a number of scenarios where users want

Re: Code coverage report on github PRs

I believe most of them can be added by us using GitHub Workflow. There's a marketplace for these tools and most of them are free for open source projects. On Wed, Nov 23, 2022 at 11:43 AM Ayush Saxena wrote: > A simple Infra ticket I suppose should get it done for us, eg. > https://issues.apache

Re: Request to create an ASF Jira account

I've submitted a request to create the account. You should receive an email shortly. On Tue, Feb 14, 2023 at 1:40 PM ravindra dingankar < ravindradingan...@gmail.com> wrote: > Hi, > > I am part of LinkedIn's HDFS team, and would like to start contributing to > HDFS and be part of the mailing lis

[DISCUSS] Move HDFS specific APIs to FileSystem abstration

Hi, Stephen and I are working on a project to make HBase to run on Ozone. HBase, born out of the Hadoop project, depends on a number of HDFS specific APIs, including recoverLease() and isInSafeMode(). The HBase community [1] strongly voiced that they don't want the project to have direct dependen

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

is. Yes, those of us who can build the > entire stack can do it, but that gradually adds more divergence from the > open source libraries, reduces the test coverage overall and only increases > maintenance costs over time. > > steve > > On Thu, 16 Mar 2023 at 20:56, Wei-Chiu

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

I think moving up interfaces to FileSystem or some abstract FileSystem class has a few benefits: 1. Application can potentially be made FS-agnostic, with hasPathCapabilities() check. At least, make the code to compile. 2. We will be able to add a contract test to ensure behavior is expected. The

Re: [DISCUSS] hadoop branch-3.3+ going to java11 only

My random thoughts. Probably bad takes: There are projects experimenting with JDK17 now. JDK11 active support will end in 6 months. If it's already hard to migrate from JDK8 why not retarget JDK17. On Tue, Mar 28, 2023 at 10:30 AM Ayush Saxena wrote: > I know Jersey upgrade as a blocker. Some f

[DISCUSS] Hadoop 3.3.6 release planning

Hi community, I'd like to kick off the discussion around Hadoop 3.3.6 release plan. I'm being selfish but my intent for 3.3.6 is to have the new APIs in HADOOP-18671 added so we can have HBase to adopt this new API. Other than that, perhaps thi

Re: Call for Presentations, Community Over Code 2023

There's also a call for presentation for Community over Code Asia 2023 https://www.bagevent.com/event/cocasia-2023-EN Happening Aug 18-20. CfP due by 6/6 On Tue, May 9, 2023 at 8:39 PM Ayush Saxena wrote: > Forwarding from dev@hadoop to the dev ML which we use. > > The actual mail lies here: >

Re: [DISCUSS] Hadoop 3.3.6 release planning

iu for driving this release. > > > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed > version. > > > > > > > > so far only 8 jiras were resolved in the branch-3.3 line. > > > > > > > > > > > > If we should consider bo

[DISCUSS] Mysql-connector-java is GPL licensed; Hadoop can't use it.

Hi community, While preparing for 3.3.6 RC, I realized the mysql-connector-java dependency added by HADOOP-18535 is GPL licensed. Source: https://github.com/mysql/mysql-connector-j/blob/release/8.0/LICENSE See legal discussion at Apache LEGAL-

Re: [DISCUSS] Mysql-connector-java is GPL licensed; Hadoop can't use it.

ginal jira. > > .. Owen > > > On Fri, Jun 9, 2023 at 1:58 AM Wei-Chiu Chuang > > wrote: > > > Hi community, > > > > While preparing for 3.3.6 RC, I realized the mysql-connector-java > > dependency added by HADOOP-18535 > > <https://issues.ap

Fwd: [jira] [Created] (HADOOP-18768) Integrating Apache Hadoop into OSS-Fuzz

Are there Hadoop committers who would like to help triage bug reports from OSS-Fuzz? -- Forwarded message - From: Henry Lin (Jira) Date: Mon, Jun 12, 2023 at 9:10 AM Subject: [jira] [Created] (HADOOP-18768) Integrating Apache Hadoop into OSS-Fuzz To: Henry Lin created HADOOP-18

Re: [DISCUSS] Hadoop 3.3.6 release planning

ace. While setSafeMode() is exposed through SafeMode interface. On Thu, Jun 8, 2023 at 11:11 AM Wei-Chiu Chuang wrote: > Thanks for comments > > Looking at jiras fixed in 3.3.6 and 3.3.9 (my bad, forgot that most > commits landing in branch-3.3 was 3.3.9), most are okay. We have about

[VOTE] Release Apache Hadoop 3.3.6 RC0

I am inviting anyone to try and vote on this release candidate. Note: This is built off branch-3.3.6 plus PR#5741 (aws sdk update) and PR#5740 (LICENSE file update) The RC is available at: https://home.apache.org/~weichiu/hadoop-3.3.6-RC0-amd64/ (for amd64) https://home.apache.org/~weichiu/hadoop

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

59->AbstractContractDistCpTest.distCpUpdateDeepDirectoryStructure:334->AbstractContractDistCpTest.assertCounterInRange:294->Assert.assertTrue:41->Assert.fail:88 Files Copied value 2 above maximum 1 [INFO] [ERROR] Tests run: 240, Failures: 3, Errors: 0, Skipped: 58 Ozone test in progress. Wil

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

7;s not the end of the world. I think we can address it by adding an incompatible change flag and a release note. On Wed, Jun 14, 2023 at 3:55 PM Wei-Chiu Chuang wrote: > Cross referenced git history and jira. Changelog needs some update > > Not in the release > >1. HDFS-16858 &

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

It's branching off branch-3.3 On Thu, Jun 15, 2023 at 3:18 AM Steve Loughran wrote: > Which branch is -3.3.6 off? 3.3.5 or 3.3? > > I'm travelling for the next few days and unlikely to be able to test this; > will do my best > > On Wed, 14 Jun 2023 at 07:27, Wei-Ch

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

ey for gs://fake-in-memory-test-b ucket/contract-test/testOpenFileApplyAsyncRead "fs.option.openfile.length" On Wed, Jun 14, 2023 at 5:01 PM Wei-Chiu Chuang wrote: > The hbase-filesystem tests passed after reverting HADOOP-18596 > <https://issues.apache.org/jira/b

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

lease. The next RC will be prepared, > right? > If true, I would like to try and vote on the next RC. > Just notice that some JIRAs are not included and need to revert some PRs to > pass HBase verification which are mentioned above. > > Best Regards, > - He Xiaoqiao > &

[VOTE] Release Apache Hadoop 3.3.6 RC1

I am inviting anyone to try and vote on this release candidate. Note: This is exactly the same as RC0, except the CHANGELOG. The RC is available at: https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64) https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64) Git tag:

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

Bumping this thread to the top. If you are verifying the release, please vote on this thread. RC0 and RC1 are exactly the same. The only material difference is the Changelog. Thanks!! On Sun, Jun 18, 2023 at 5:52 PM Wei-Chiu Chuang wrote: > I am inviting anyone to try and vote on this rele

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

gt; > + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK > 8 > >> > with native profile enabled. > >> >+ launched pseudo distributed cluster including kms and httpfs with > >> > Kerberos and SSL enabled. > >> >+ created

DockerHub admin for Apache Hadoop

Ayush and I have acquired the DockerHub admin privilege for the Hadoop project in order to facilitate the release of Hadoop 3.3.6. Apache Infra allows only two seats per project. So if you need something, let Ayush and I know and we will make it happen for you. If you are a Docker guru and can't h

Clean up old Hadoop release tarballs

https://dist.apache.org/repos/dist/release/hadoop/common/ has Hadoop release tarballs 3.3.1 ~ 3.3.5. I plan to remove the tarballs from 3.3.1 to 3.3.4 and leave only 3.3.5 (and the upcoming 3.3.6). Shout out if you have something depending on the old release tarballs (you shouldn't) Other release

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

Launched a pseudo distributed cluster using released binary packages, > > done some HDFS dir/file basic opeations. > > + Run grep, pi and wordcount MR tasks on the pseudo cluster. > > > > Bests, > > Sammi Chen > > > > 发件人:

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

> -Ayush > > > [1] https://github.com/ayushtkn/hadoop-ozone/actions/runs/5282707769 > [2] https://github.com/apache/tez/pull/285#issuecomment-1590962978 > > On Sat, 24 Jun 2023 at 09:43, Nilotpal Nandi > wrote: > >> +1 (Non-binding). >> Thanks a lot Wei-Chiu

[ANNOUNCE] Apache Hadoop 3.3.6 release

On behalf of the Apache Hadoop Project Management Committee, I am pleased to announce the release of Apache Hadoop 3.3.6. It contains 117 bug fixes, improvements and enhancements since 3.3.5. Users of Apache Hadoop 3.3.5 and earlier should upgrade to this release. https://hadoop.apache.org/releas

Re: [DISCUSS][HDFS] Add rust binding for libhdfs

Inline On Sat, Jul 15, 2023 at 5:04 AM Ayush Saxena wrote: > Forwarding from dev@hadoop to relevant ML > > Original mail: > https://lists.apache.org/thread/r5rcmc7lwwvkysj0320myxltsyokp9kq > > -Ayush > > On 2023/07/15 09:18:42 Xuanwo wrote: > > Hello, everyone. > > > > I'm the maintainer of [hdf

HADOOP-18207 hadoop-logging module about to land

Hi, I am preparing to resolve HADOOP-18207 ( https://github.com/apache/hadoop/pull/5717). This change affects all modules. With this change, it will eliminate almost all the direct log4j usage. As always, landing such a big piece is tricky. I

Re: How should we do about dependency update?

dependencies where practical. I don't know how often a CVE > comes along for things like our logging API dependency, for example. > But downstream folks should definitely not rely on dependencies we use > for internal service, so I'm surprised that a version change for Jetty >

Reminder: APAC Hadoop storage community sync

PDT 10pm Wednesday = tonight, CST 1pm Thursday = today. Feel free to join Zoom and chat. Join Zoom Meeting https://cloudera.zoom.us/j/880548968 Past sessions: https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit Also heads-up, On November 20/21, Feilong from Inte

HDFS sync

Hi folks, I don't want to add meetings unnecessarily. But I would like to see the oncall engineer of the week to join and summarize the support cases (CDH engineering escalations, HDP EARs). Ideally any one with a case open should join too, but let's start with this and see how it goes. Yes I am u

[DISCUSS] Enable github security notifications to all Hadoop committers

Hi, I raised INFRA-19327 to enable github security notification. How do people feel if we enable this notification to all committers? I already have hundreds of incoming emails from various Hadoop alias each day so I don't mind adding a few more.

Fwd: Cloud Recording - HDFS sync is now available

This today's HDFS sync recording. Siyao's Dynamometer presentation starts at around 18th minute. -- Forwarded message - From: Zoom Date: Thu, Oct 24, 2019 at 12:19 PM Subject: Cloud Recording - HDFS sync is now available To: Hi Wei-Chiu Chuang, Your cloud record

Re: [DISCUSS] Remove Ozone and Submarine from Hadoop repo

+1 filed HADOOP-16670 for stripping the Submarine code. On Thu, Oct 24, 2019 at 12:14 PM Subru Krishnan wrote: > +1. > > Thanks, > Subru > > On Thu, Oct 24, 2019 at 12:51 AM 张铎(Duo Zhang) > wrote: > > > +1 > > > > Akira Ajisaka 于2019年10月24日周

Re: Reminder: APAC Hadoop storage community sync

e are internal teams that rely on the access time stats. We discussed this and mentioned Uber stated the same issue in the January Hadoop meetup. Didi’s Hadoop team is interested in Ozone after all clusters are upgraded to Hadoop 3. On Wed, Oct 23, 2019 at 3:55 PM Wei-Chiu Chuang wrote: >

This week's Hadoop storage community online sync

Hello, I am super stoked to have Yiqun Lin with us this Wednesday morning Oct 30 US Pacific 10am/CET (Budapest) 6pm/ IST (Banglore) 10:30pm/ CST (Beijing) Oct 31 1am / JST (Tokyo) 2am to talk about “HDFS Cluster Optimization in eBay” — Yiqun happens to be in the bay area this week and this is the s

Re: This week's Hadoop storage community online sync

Gentle reminder. Yiqun will present in 30 minutes! On Mon, Oct 28, 2019 at 7:41 PM Wei-Chiu Chuang wrote: > Hello, I am super stoked to have Yiqun Lin with us this Wednesday morning > Oct 30 US Pacific 10am/CET (Budapest) 6pm/ IST (Banglore) 10:30pm/ CST > (Beijing) Oct 31 1am / JST (T

Re: This week's Hadoop storage community online sync

so not looking into RBF now. Looking into upgrading to Hadoop 2.9 or Hadoop 3 Ozone tested by Pinduoduo and JD in China. Talked about recent upstream Hadoop development: Namenode fine grained locking, OpenTracing, JDK11 On Wed, Oct 30, 2019 at 9:30 AM Wei-Chiu Chuang wrote: > Gentle remind

Topic for this Wednesday's APAC Hadoop storage community sync: Ozone

Hi! There have been a number of questions especially from the Chinese community regarding Ozone lately. I am happy to have @Xiaoyu Yao who graciously agreed to talk to us the current status of Ozone this Wednesday even (US) / Thursday (APAC). Time: US Pacific: Wednesday (Nov 6) 10pm Beijing: Th

Notes from Hadoop storage community online sync

Thanks @Xiaoyu Yao for giving us a great status update on Ozone! We had a pretty large group yesterday. Here's my notes for your reference: https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing 11/6/2019 ~20 contributors joined the discussion.Weichiu, X

Next Wednesday (Nov 13) Hadoop storage online community sync

Hi, I am happy to invite Zhenyu to join us to talk about the recent proposal of supporting ARM/aarch64 for Hadoop. November 13 (Wednesday) US Pacific Time 10am / November 13 (Wednesday) Bangalore 11:30pm) / November 14 (Thursday) Beijing 2am. Previous meeting notes: https://docs.google.com/docum

Re: Next Wednesday (Nov 13) Hadoop storage online community sync

Just a reminder. This online sync is starting in 3 minutes. On Fri, Nov 8, 2019 at 3:58 PM Wei-Chiu Chuang wrote: > Hi, > > I am happy to invite Zhenyu to join us to talk about the recent proposal > of supporting ARM/aarch64 for Hadoop. > > November 13 (Wednesday) US

Today's Hadoop storage online community sync

Clm-4-pR1UxEBRrrTiUkX3CxstfKY/edit?usp=sharing Steve L.: People need to care about nightly failures. Zhenyu: add ARM test build in nightly or periodically, and then precommit builds. On Fri, Nov 8, 2019 at 3:58 PM Wei-Chiu Chuang wrote: > Hi, > > I am happy to invite Zhenyu to join us

This Week's APAC Hadoop storage community sync

Hi! I am happy to have Feilong from Intel to join us and talk about the Storage Class Memory support for HDFS ( HDFS-13762 ). This is a new feature that will land in the next Hadoop minor release (Hadoop 3.3.0) Date/Time: Wednesday Nov 20 10pm Pac

Re: This Week's APAC Hadoop storage community sync

st [image: Screen Shot 2019-11-18 at 11.37.02 PM.png] On Mon, Nov 18, 2019 at 8:23 PM Wei-Chiu Chuang wrote: > Hi! > > I am happy to have Feilong from Intel to join us and talk about the > Storage Class Memory support for HDFS ( HDFS-13762 > <https://issues.apache.org/jira/brow

[DISCUSS] Move HDDS JIRA notification to ozone-dev@/ozone-issues@

Hi, Looking at the Notification settings of HDDS project, it is still sending notification emails to hdfs-dev@/hdfs-issues@. Can we move them to ozone-dev@ and ozone-issues@? Weichiu - To unsubscribe, e-mail: hdfs-dev-unsubscr..

Re: [DISCUSS] Move HDDS JIRA notification to ozone-dev@/ozone-issues@

On Tue, Nov 19, 2019 at 7:04 AM Wei-Chiu Chuang wrote: > Hi, > > Looking at the Notification settings of HDDS project, it is still sending > notification emails to hdfs-dev@/hdfs-issues@. Can we move them to > ozone-dev@ and ozone-issues@? > > Weichiu >

Re: This Week's APAC Hadoop storage community sync

Friendly reminder: this event is happening in an hour! On Mon, Nov 18, 2019 at 11:38 PM Wei-Chiu Chuang wrote: > I'm really bad at converting time across different time zones. > > Here's the correction: > Pacific Time: Wed Nov 20 10pm > Beijing Time: Thu Nov 21 2pm &g

Re: This Week's APAC Hadoop storage community sync

ached block files. Hierarchical cache storage. - Alibaba evaluated HBase BucketCache with Pmem. There’s a benchmark report somewhere that Rakesh On Wed, Nov 20, 2019 at 9:13 PM Wei-Chiu Chuang wrote: > Friendly reminder: this event is happening in an hour! > > On Mon, Nov 18,

Re: [DISCUSS] Enable github security notifications to all Hadoop committers

t; from various Hadoop alias each day so I don't mind adding a few more. >> >> Completely agreed. >> >> -Akira >> >> On Thu, Oct 24, 2019 at 3:48 PM Bharat Viswanadham >> wrote: >> >> > +1 (binding). I am interested in receiving th

APAC Mandarin community sync happening now

Join Zoom Meeting https://cloudera.zoom.us/j/880548968

Re: APAC Mandarin community sync happening now

d about EC performance. Haiyang stated they tested EC performance with Java codec (w/o Intel ISA-L library) so performance suffered. Weichiu: use the latest Intel ISA-L library gives the best performance. On Wed, Dec 4, 2019 at 9:59 PM Wei-Chiu Chuang wrote: > Join Zoom Meeting > > https:/

  1   2   3   4   5   6   7   8   >