Re: [DISCUSS] Hadoop 3.3.1 release

te reply.. > > > > > >> >> > > > > > >> >> I will come up with a plan.. Please let me know if anybody > has > > > some > > > > > >> >> features/improvements/bugs that need to be included. >

Re: [DISCUSS] Hadoop 3.3.1 release

the release, please tag the jira with the label 'release-blocker'. On Tue, Apr 20, 2021 at 2:44 AM Billie Rinaldi wrote: > I was thinking of backporting HADOOP-16948 for 3.3.1. > > Billie > > On Mon, Apr 19, 2021 at 1:33 AM Wei-Chiu Chuang > wrote: > > > H

Re: [ANNOUNCE] Apache Hadoop Ozone 1.1.0 release

+common-dev@hadoop Prashant, I am inclined to revert the Ozone announcement on the Hadoop website. Sorry, but it looks like it removed a few older Hadoop release announcements (Hadoop 3.3.0 release) For the Hadoop community, I think it would be a good idea to add a news announcement section to de

Re: [ANNOUNCE] Apache Hadoop Ozone 1.1.0 release

I reverted the change in the Hadoop website. Will go ahead adding the announcement of Ozone becoming a TLP to the Hadoop website later. On Wed, Apr 21, 2021 at 2:48 PM Wei-Chiu Chuang wrote: > +common-dev@hadoop > Prashant, > > I am inclined to revert the Ozone announcement o

[DISCUSS] hadoop-thirdparty 1.1.0 release

Hi community, In preparation of the Hadoop 3.3.1 release, I am starting a thread to discuss its prerequisite: the release of hadoop-thirdparty 1.1.0. My plan: update guava to 30.1.1 (latest). I have the PR ready to merge. Do we want to update protobuf and jaeger? Anything else? I suppose we won

Re: [DISCUSS] hadoop-thirdparty 1.1.0 release

version anyway. The version used is too old. On Fri, Apr 23, 2021 at 10:55 AM Wei-Chiu Chuang wrote: > Hi community, > > In preparation of the Hadoop 3.3.1 release, I am starting a thread to > discuss its prerequisite: the release of hadoop-thirdparty 1.1.0. > > My plan: >

Sorry for breaking the build

If you are seeing precommit failures, like this: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-annotations: Compilation failure [ERROR] javac: invalid flag: -Xmaxwarns= [ERROR] Usage: javac [ERROR] use -help for

Re: [DISCUSS] hadoop-thirdparty 1.1.0 release

apshot, the hadoop code would pick > that up, and watch out everything is safe and clean before release. Unless > you have a better way to verify or already verified!!! > > -Ayush > > > On 23-Apr-2021, at 3:16 PM, Wei-Chiu Chuang > wrote: > > > > Another suggestion:

Re: [DISCUSS] hadoop-thirdparty 1.1.0 release

02 PM Ayush Saxena wrote: > Yep, you have to do it manually > > -Ayush > > On 26-Apr-2021, at 3:23 PM, Wei-Chiu Chuang wrote: > >  > Does anyone know how we publish hadoop-thirdparty SNAPSHOT artifacts? > > The main Hadoop arifacts are published by this job > ht

Hadoop Jenkins account

If you're a committer and you're interested in getting involved in the build/release work, feel free to let me know so I can grant you access to the Jenkins box. We can definitely make use of extra help. More details: https://cwiki.apache.org/confluence/display/INFRA/Jenkins#Jenkins-HowdoIgetanacc

Re: Hadoop Jenkins account

e interested >> -Ayush >> >> > On 26-Apr-2021, at 4:56 PM, Wei-Chiu Chuang wrote: >> > >> > If you're a committer and you're interested in getting involved in the >> > build/release work, feel free to let me know so I can grant you acces

Re: [DISCUSS] hadoop-thirdparty 1.1.0 release

I'll start preparing the release vote for third-party 1.1.0. Thanks all for help! On Mon, Apr 26, 2021 at 7:20 PM Wei-Chiu Chuang wrote: > Thanks. I created a Jenkins job to upload the SNAPSHOT to Apache nexus > repository. > > https://ci-hadoop.apache.org/view/Hadoop/job/

Fwd: Error while running github feature from .asf.yaml in hadoop!

I received this message but it's impossible to tell what went wrong. My guess is this PR HADOOP-17623 https://github.com/apache/hadoop-site/commit/d3f5b0bc70c2c4fdb542c4a0bdfc2ed9c11edfa6 caused the error though. Any idea what we can do to fix this? I can easily revert the change if needed. -

Re: [DISCUSS] hadoop-thirdparty 1.1.0 release

with this change so please help review. Thanks! On Tue, Apr 27, 2021 at 2:59 PM Wei-Chiu Chuang wrote: > I'll start preparing the release vote for third-party 1.1.0. Thanks all > for help! > > On Mon, Apr 26, 2021 at 7:20 PM Wei-Chiu Chuang > wrote: > >> Thanks

Re: [DISCUSS] Hadoop 3.3.1 release

review the remaining jiras and push out those that don't have much progress today. Thanks all! Weichiu On Tue, Apr 20, 2021 at 12:57 PM Wei-Chiu Chuang wrote: > Billie and Viraj, > > As of today we are at 657 resolved jiras in 3.3.1 > https://issues.apache.org/jira/issues/?filter=-1&am

[VOTE] hadoop-thirdparty 1.1.0-RC0

Hello my fellow Hadoop developers, I am putting together the first release candidate (RC0) for Hadoop-thirdparty 1.1.0. This is going to be consumed by the upcoming Hadoop 3.3.1 release. The RC is available at: https://people.apache.org/~weichiu/hadoop-thirdparty-1.1.0-RC0/ The RC tag in github i

Updated Hadoop fix versions

Hi, I found hundreds of discrepancies between Apache JIRA and git history, and updated the jira fix versions according to git history. Some of the noticeable updates: (1) HADOOP-16730 ABFS: Support for Shared Access Signatures (SAS) --> originally a new feature in 3.3.1, is actually in 3.3.0. (

Hadoop LZ4 codec questions

Hi I'm trying to understand the LZ4 codec usage in Hadoop. Liang-Chi replaced the LZ4 codec with the lz4-java HADOOP-17292. The intent is so that we can use the native library that is bundled in the jar, no need to install lz4 native libraries on the host machine. However, there's another LZ4 cod

Re: [VOTE] hadoop-thirdparty 1.1.0-RC0

all -DskipTests > > - mvn clean install -DskipTests -Psrc > > * Built Hadoop trunk againt hadoop-thirdparty 1.1.0: ok > > > > > > On Thu, May 13, 2021 at 5:25 PM Wei-Chiu Chuang > > wrote: > > > > > Hello my fellow Hadoop developers, > > > &g

Re: [DISCUSS] Hadoop 3.3.1 release

5790) so you guys can quickly check out. The first RC will probably not have the aarch64 binary. I don't have much experience with that. Cheers, Wei-Chiu On Thu, May 13, 2021 at 9:04 AM Wei-Chiu Chuang wrote: > Hello it's me again. > > While working on the hadoop-thirdparty

Missing options and profiles in the create-release script

Hello, I'm trying to understand Hadoop's create-release script. Looks like it's missing a number of options and profiles that the regular precommit script does: https://github.com/apache/hadoop/blob/trunk/dev-support/bin/create-release#L133 https://github.com/apache/hadoop/blob/trunk/dev-support

Hadoop Branching: branch-3.3 is now 3.3.2-SNAPSHOT

Hi I created a branch-3.3.1. The branch-3.3. now tracks 3.3.2-SNAPSHOT. Please use Fix version 3.3.2 when cherrypicking your commits to branch-3.3. Regards, Wei-Chiu

Publishing Apache Hadoop 3.3.1-RC0 Preview bits

Hi community, This is the preview bits of the first release candidate of Apache Hadoop 3.3.1 line. It takes tremendous amount of effort to produce a Hadoop RC and we still have a number of unresolved issues [1]. Meanwhile, I'd like to publish preview bits so you guys can check out earlier than lat

Re: [DISCUSS] which release lines should we still consider actively maintained?

Sean, For reasons I don't understand, I never received emails from your new address in the mailing list. Only Akira's response. I was just able to start a thread like this. I am +1 to EOL 3.1.5. Reason? Spark is already on Hadoop 3.2. Hive and Tez are actively working to support Hadoop 3.3. HBas

[VOTE] Release Apache Hadoop 3.3.1 - RC1

Hi community, This is the release candidate RC1 of Apache Hadoop 3.3.1 line. All blocker issues have been resolved. It contains 697 fixed jira issues [2] since 3.3.0 which include a lot of features and improvements(read the full set of release notes). Below feature additions are the highlights o

Re: [VOTE] Release Apache Hadoop 3.3.1 - RC1

ertionError: expected null, but was:<1000> at org.apache.hadoop.hbase.TestHBaseConfiguration.testDeprecatedConfigurations(TestHBaseConfiguration.java:150) On Mon, May 24, 2021 at 10:36 PM Wei-Chiu Chuang wrote: > Hi community, > > This is the release candidate RC1 of Apach

[VOTE] Release Apache Hadoop Thirdparty 1.1.1 RC0

Hi folks, I have put together a release candidate (RC0) for Hadoop Thirdparty 1.1.1 which will be consumed by Hadoop 3.3.1 RC2. The RC is available at: https://people.apache.org/~weichiu/hadoop-thirdparty-1.1.1-RC0/ The RC tag in svn is here:https://github.com/apache/hadoop-thirdparty/releases

Re: [VOTE] Release Apache Hadoop 3.3.1 - RC1

review the hadoop-thirdparty 1.1.1 RC0 and cast a vote. Thanks! On Wed, May 26, 2021 at 9:39 AM Wei-Chiu Chuang wrote: > > Found a few issues building downstreams > > Ozone: HADOOP-17730 <https://issues.apache.org/jira/browse/HADOOP-17730> -- > we may have to mak

Re: [VOTE] Release Apache Hadoop Thirdparty 1.1.1 RC0

ecked the documents >> - Compiled Hadoop trunk and branch-3.3 with Hadoop third-party 1.1.1. >> >> -Akira >> >> On Wed, May 26, 2021 at 5:29 PM Wei-Chiu Chuang >> wrote: >> > >> > Hi folks, >> > >> > I have put together a release

Re: [VOTE] Release Apache Hadoop Thirdparty 1.1.1 RC0

Compiled Hadoop trunk and branch-3.3 with Hadoop third-party 1.1.1. >> >> -Akira >> >> On Wed, May 26, 2021 at 5:29 PM Wei-Chiu Chuang >> wrote: >> > >> > Hi folks, >> > >> > I have put together a release candidate (RC0) for

[VOTE] Release Apache Hadoop 3.3.1 RC2

Hi community, This is the release candidate RC2 of Apache Hadoop 3.3.1 line. All blocker issues have been resolved [1] again. There are 4 issues resolved for RC2: * HADOOP-17666. Update LICENSE for 3.3.1 * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053) * Revert "HADOOP-17563. U

Re: [VOTE] Release Apache Hadoop 3.3.1 RC2

Cancelling RC2. @Ayush Saxena found an issue HADOOP-16878 <https://issues.apache.org/jira/browse/HADOOP-16878> which we need to revert. I'll work on the RC3 asap. On Tue, Jun 1, 2021 at 3:23 PM Wei-Chiu Chuang wrote: > Hi community, > > This is the release candidate RC2 of

[VOTE] Release Apache Hadoop 3.3.1 RC3

Hi community, This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker issues have been resolved [1] again. There are 2 additional issues resolved for RC3: * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878 * Revert "HADOOP-16878. FileUtil.copy() t

Do I need a +1 to merge a backport PR?

I'm curious about the GitHub PR conventions we use today... say I want to backport a commit from trunk to branch-3.3, and there's a small code conflict so I push a PR against branch-3.3 using GitHub to go through the precommit check. Do I need explicit approval from another committer to merge the

Re: apply for jira permission

Added you to the contributor1 list of HADOOP. Do you also plan to contribute to YARN or HDFS? We will have to do that separately. On Wed, Jun 2, 2021 at 7:31 PM Jerry Zhao wrote: > Hi, > > I want to contribute to Apache hadoop. Would you please give me the > contributor permission? My JIRA ID is

Re: [VOTE] Hadoop 3.1.x EOL

+1 On Thu, Jun 3, 2021 at 2:14 PM Akira Ajisaka wrote: > Dear Hadoop developers, > > Given the feedback from the discussion thread [1], I'd like to start > an official vote > thread for the community to vote and start the 3.1 EOL process. > > What this entails: > > (1) an official announcement t

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

faster by 20% over Hadoop 3.2 library. Looks like we may need some more time to test. How about extending it by a week? On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang wrote: > Hi community, > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker > issu

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

e release is a > big release, but I get that it’s the current state of the project. > > > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang wrote: > > > > Hello, > > do we want to extend the release vote? I understand a big release like > this > > takes time to val

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

doesn't render right. >> Created https://issues.apache.org/jira/browse/HADOOP-17746. >> >> This is *not a blocker* >> >> I can prepare a fix and we can have it in so that if any other changes >> come >> in the page will look OK. >> >>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

> > > +1, >> > > Built from Source. >> > > Successful Native Build on Ubuntu 20.04 >> > > Verified Checksums >> > > Ran basic hdfs shell commands. >> > > Ran simple MR jobs. >> > > Browsed NN,DN,RM and NM UI. >> > > >> &g

Re: [VOTE] Hadoop 3.1.x EOL

Dropped 3.1.4 from website. Removed from https://dist.apache.org/repos/dist/release/hadoop/common/ On Thu, Jun 10, 2021 at 3:42 PM Akira Ajisaka wrote: > This vote has passed with 18 binding +1. I'll update the JIRA and the wiki. > > Thanks all for your participation. > > On Tue, Jun 8, 2021 at

[ANNOUNCE] Apache Hadoop 3.3.1 release

olks who continued helps for this release process. Best Regards, Wei-Chiu Chuang

Three Hadoop talks in this year's ApacheCon Asia

For your information, While drafting the upcoming quarterly report, I found there are three talks that are directly related to Hadoop in this year's ApacheCon Asia. https://apachecon.com/acasia2021/tracks/bigdata.html Bigtop 3.0: Rerising community driven Hadoop distribution

[DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

I work on multiple projects and learned a bunch from those projects.There are nice add-ons that help with productivity. There are things we can do to help us manage the project better. 1. Add new issue types. We can add "Epic" jira type to organize a set of related jiras. This could be easier to m

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

Here's another one: We can update the JIRA workflow and add more state. For example, Cassandra has "Review in Progress" and "Need Reviewer" states. INFRA-22049 <https://issues.apache.org/jira/browse/INFRA-22049> On Tue, Jul 13, 2021 at 11:34 AM Wei-Chiu Chuan

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

ew GitHub webhooks? > - Just for discussion: is Integrating SonarQube worth the effort of > supporting the GitHub webhooks. SonarQube >is a double edge weapon and it needs dedicated man hours to weed out > the way through the reports. > > On Mon, Jul 12, 2021 at 11:18 PM Wei-

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

One more idea that I forgot to mention: We can have the Dependabot to automatically create a pull request whenever a vulnerability is discovered. On Wed, Jul 14, 2021 at 8:42 AM Wei-Chiu Chuang wrote: > So this is just up for discussion and welcome more brainstorming. I'm not > say

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

A, always use GitHub PR. It saves > > costs to configure and debug the precommit jobs. > > https://issues.apache.org/jira/browse/HADOOP-17798 > > 2. Improve the pull request template for the contributors > > https://issues.apache.org/jira/browse/HADOOP-17799 > > > >

Re: Hadoop-3.2.3 Release Update

Hi to raise the awareness, it looks like reverting the FoldedTreeSet HDFS-13671 breaks TestBlockManager in branch-3.2. Branch-3.3 is good. tracking jira: HDFS-16258 On Tue, Oct 5, 2021 at 8:45 P

ApacheCon@Home Big Data tracks recordings!

For those who missed the live Apache@Home Big Data tracks, the video recordings are being uploaded to the official ASF channel! Big Data: https://www.youtube.com/playlist?list=PLU2OcwpQkYCzXcumE9UxNirLF1IYLmARj Big Data Ozone: https://www.youtube.com/playlist?list=PLU2OcwpQkYCxtPdZ0nSowYLQMgkmoczM

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC0

Thanks a lot for pushing it forward! A few things I noticed that we should incorporate: 1. the overview page of the doc is for the Hadoop 3.0 release. It would be best to base the doc on top of Hadoop 3.3.0 overview page. (it's a miss on my part... The overview page of 3.3.1 wasn't updated) 2. A

Trunk broken by HDFS-16384

My bad. There was a transitive dependency problem in the PR causing trunk to fail the build. The commit has since been reverted. Sorry for the inconvenience.

Apache Hadoop and CVE-2021-44228 Log4JShell vulnerability

Hi, Given the widespread attention to the recent log4j vulnerability (CVE-2021-44228), I'd like to share an update from the Hadoop developer community regarding the incident. As you probably know, Apache Hadoop depends on the log4j library to keep log files. The highlighted vulnerability CVE-2021-

Re: Hadoop-3.2.3 Release Update

Is this still making progress? On Tue, Oct 5, 2021 at 8:45 PM Brahma Reddy Battula wrote: > Hi Akira, > > Thanks for your email!! > > I am evaluating the CVE’s which needs to go for this release.. > > Will update soon!! > > > On Tue, 5 Oct 2021 at 1:46 PM, Akira Ajisaka wrote: > > > Hi Brahma,

Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2

+1 I think it makes sense to use reload4j in maint releases. I have a draft PR doing this (https://github.com/apache/hadoop/pull/3906) log4j2 in Hadoop 3.4.0 makes sense to me. There could be incompatibilities introduced by log4j2, but I feel we should at least make it 3.4.0 a "preview" release, a

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2

I'll find time to check out the RC bits. I just feel bad that the tarball is now more than 600MB in size. On Fri, Jan 21, 2022 at 2:23 AM Steve Loughran wrote: > *+1 binding.* > > reviewed binaries, source, artifacts in the staging maven repository in > downstream builds. all good. > > *## test

Re: [ANNOUNCE] Apache Hadoop 3.3.2 release

> > Many thanks to Viraj Jasani, Michael Stack, Masatake Iwasaki, Xiaoqiao He, > Mukund Madhav Thakur, Wei-Chiu Chuang, Steve Loughran, Akira Ajisaka and > other folks who helped for this release process. > > Best Regards, > Chao >

[ANNOUNCE] New Hadoop PMC Sun Chao

On behalf of the Apache Hadoop PMC, I am pleased to announce that Sun Chao(sunchao) has accepted the PMC's invitation to become a PMC on the project. We appreciate all of Sun's generous contributions thus far and look forward to his continued involvement. Congratulations and welcome, Sun!

Re: [VOTE] Release Apache Hadoop 3.2.3 - RC0

aarch64 support is only introduced in/after 3.3.0 On Thu, Mar 17, 2022 at 2:27 PM Emil Ejbyfeldt wrote: > Hi, > > > There is no aarch64 artifact in the release candidate. Is this something > that is intended? > > Best, > Emil Ejbyfeldt > > On 14/03/2022 08:14, Masatake Iwasaki wrote: > > Hi all,

Fwd: [NOTICE] Dependabot Updates enabled for all projects

-- Forwarded message - From: Chris Lambertus Date: Tue, Apr 5, 2022 at 12:38 PM Subject: [NOTICE] Dependabot Updates enabled for all projects To: Hi folks, Infra is pleased to announce that GitHub’s Dependabot service has been approved for use by ASF Legal and Infra, and is no

Re: Institutions running kerberized Hadoop clusters

Last time I checked (~2 years ago), there were thousands of Kerberized clusters among Cloudera customers. The largest ones had a few thousand nodes. What are you looking for? On Wed, Apr 13, 2022 at 7:22 AM Santosh Marella wrote: > Hey folks, > > Just curious if we have a list of institutions

Re: [DISCUSS] Hadoop on Windows

Great! Sorry I missed the earlier discussion thread. Is there a target version for this support? I assume the milestone is still in a dev branch? On Thu, Apr 28, 2022 at 8:26 AM Gautham Banasandra wrote: > Hi Hadoop devs, > > I would like to announce that we recently reached a new milestone - w

Re: Publishing hadoop-lzo maven package in com.hadoop namespace

Please be careful. Sorry but hadoop-lzo is GPL licensed. Internally at my employer we don't use/ship hadoop-lzo because of the licensing concerns. I suppose since the code is GPL we as an Apache project want to stay away from it? On Thu, May 5, 2022 at 11:17 AM Ayush Saxena wrote: > Didn’t read

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

Running builds for all platforms for each and every PR seems too excessive. How about doing all platform builds in the nightly jobs? On Fri, May 6, 2022 at 8:02 AM Steve Loughran wrote: > I'm not enthusiastic here as it not only makes the builds slower, it > reduces the #of builds we can throug

Re: Where is aarch64 tgz for 3.3.2 and 3.3.3

I think only the tarball needs to be uploaded. The Maven repository doesn't (the jars would be the same?) At least that's what I did before. On Thu, Jun 16, 2022 at 6:04 PM Ayush Saxena wrote: > > > > I'm assuming that the arm docker image can do the build, either on my > > macbook or more likel

Re: style rules

I have seen violated that rule before and wondered if it's because final variables are essentially constants in other languages so all caps make sense? On Mon, Jul 11, 2022 at 10:35 AM Owen O'Malley wrote: > I keep stubbing my toe on the unfortunate Hadoop style rule that says final > variables

Re: Assignment to JIRA tickets

What is your JIRA user id? I can help you by adding you to the contributor list where you'll be able to assign the tickets yourself. On Thu, Aug 4, 2022 at 1:43 PM Steve Vaughan wrote: > I'd like to be assignable to tickets that I open. I've been creating JIRA > tickets for issues I'm actively

Hadoop BoF at ApacheCon NA'22

We've not had a physical event for a long long time and we're way overdue for one. I'm excited to announce we've reserved a room at the upcoming ApacheCon for Birds-of-Feather on October 4th from 17:50-18:30 CDT in Rhythms I. I was also told that participants can stay after that until the hotel pe

Re: Hadoop BoF at ApacheCon NA'22

document/d/1_ha1BFeEyIkAJtl5tJ8Z4Vf5Md_0rN23RCrXkyNCY1c/edit?usp=sharing Please add more details here. On Thu, Sep 22, 2022 at 10:27 PM 俊平堵 wrote: > Thanks Wei-Chiu. I am happy to share the status of Hadoop Meetups in China > (2019-2022) if that is a suitable topic. :) > > Thanks, > > Junping

Re: Hadoop BoF at ApacheCon NA'22

t; On Sat, Sep 24, 2022 at 8:52 PM 俊平堵 wrote: > > > > > Yes. It should be a short talk and I can only present remotely for this > > > time. That would be great if you help to coordinate. :) > > > > > > Thanks, > > > > > > Junping &g

[DISCUSS] Hadoop 3.3.5 release planning

Bumping this up. Adding the [DISCUSS] text to make this message stand out of your inbox. I certainly missed this message and only realized 3.3.5 has more than just security updates. What was the issue with the ARM64 build? I was able to publish ARM64 build for 3.3.1 release without problems. On

[DISCUSS] Supporting partial file rewrite/compose

There were a number of discussions that happened during ApacheCon. In the spirit of the Apache Way, I am taking the conversation online, sharing with the larger community and also capturing requirements. Credits to Owen who started this discussion. There are a number of scenarios where users want

Re: way forward for Winutils excision from `FileSystem`

> > > > * Bare Naked Local File System v0.1.0 doesn't (yet) support symlinks > or the sticky bit. > ok to not support symlinks. The symlinks of HDFS are not being maintained and I am not aware of anything relying on it. So I assume people don't need it. Sticky bit would be useful, I guess.

Re: Code coverage report on github PRs

I believe most of them can be added by us using GitHub Workflow. There's a marketplace for these tools and most of them are free for open source projects. On Wed, Nov 23, 2022 at 11:43 AM Ayush Saxena wrote: > A simple Infra ticket I suppose should get it done for us, eg. > https://issues.apache

[DISCUSS] Move HDFS specific APIs to FileSystem abstration

Hi, Stephen and I are working on a project to make HBase to run on Ozone. HBase, born out of the Hadoop project, depends on a number of HDFS specific APIs, including recoverLease() and isInSafeMode(). The HBase community [1] strongly voiced that they don't want the project to have direct dependen

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

is. Yes, those of us who can build the > entire stack can do it, but that gradually adds more divergence from the > open source libraries, reduces the test coverage overall and only increases > maintenance costs over time. > > steve > > On Thu, 16 Mar 2023 at 20:56, Wei-Chiu

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

I think moving up interfaces to FileSystem or some abstract FileSystem class has a few benefits: 1. Application can potentially be made FS-agnostic, with hasPathCapabilities() check. At least, make the code to compile. 2. We will be able to add a contract test to ensure behavior is expected. The

Re: [DISCUSS] hadoop branch-3.3+ going to java11 only

My random thoughts. Probably bad takes: There are projects experimenting with JDK17 now. JDK11 active support will end in 6 months. If it's already hard to migrate from JDK8 why not retarget JDK17. On Tue, Mar 28, 2023 at 10:30 AM Ayush Saxena wrote: > I know Jersey upgrade as a blocker. Some f

Replacing Commons-httpclient and bumping httpclient version

o the latest 4.5 for security) has a patch that bumps the version to 4.5.1. But I’d like to ask the community whether we should do it or not, and the implication of bump the latest version. Best regards, Wei-Chiu Chuang A very happy Clouderan

Re: Replacing Commons-httpclient and bumping httpclient version

case, we can easily bump its version to address the security vulnerabilities. This refers to HADOOP-12767 <https://issues.apache.org/jira/browse/HADOOP-12767> (update apache httpclient version to the latest 4.5 for security) Thanks again, Wei-Chiu Chuang A very happy Clouderan > On Feb

Re: Apache Hadoop 2.9.1 Release Plan

Sorry Sammi I was late to this thread. Please considering incorporating HDFS-11915. Sync rbw dir on the first hsync() to avoid file lost on power failure. I thought it was already in 2.9.1 but turns out it didn't land. The cherry pick to branch-2.9 is conflict free. On Mon, Apr 2, 2018 at 4:34 AM,

Re: Apache Hadoop 2.9.1 Release Plan

you help to cherry-pick it to both branch-2.9 and branch-2.9.1? > > > Thanks, > Sammi > > -Original Message- > From: Wei-Chiu Chuang [mailto:weic...@cloudera.com] > Sent: Thursday, April 5, 2018 1:30 AM > To: Chen, Sammi > Cc: hdfs-dev ; mapreduce-...@hadoop.apach

Re: Apache Hadoop 2.9.1 Release Plan

Done. Both HDFS-11915 and HDFS-13364 are in branch-2.9 and branch-2.9.1. Thank you Sammi for driving 2.9.1 release On Fri, Apr 6, 2018 at 10:14 AM, Wei-Chiu Chuang wrote: > >1. > Looks like: >2. HDFS-13347 <https://issues.apache.org/jira/browse/HDF

Re: Apache Hadoop 3.0.3 Release plan

Thanks Yongjun for driving 3.0.3 release! IMHO, could we consider adding YARN-7190 into the list? I understand that it is listed as an incompatible change, however, because of this bug, HBase considers the entire Hadoop 3.0.x line not production re

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

I'm sorry I come to this thread late. Anu commented on INFRA-16727 saying he reverted the commit. Do we still need the vote? Thanks On Thu, Jul 5, 2018 at 2:47 PM Rohith Sharma K S wrote: > +1 > > On 5 July 2018 at 14:37, Subru Krishnan wrote: > > > Folks, > > > > There was a merge commit acci

Discussion on HTrace replacement

Greetings, As you have probably heard of, HTrace is no more. We as a community need to find a solution to replace HTrace. There's currently a great discussion on a good replacement (HADOOP-15566 ) and I urge you to participate in the discussion i

[DISCUSS] Deprecate fuse-dfs

Hi fellow Hadoop developers, I want to start this thread to raise the awareness of the quality of fuse-dfs. It appears that this sub-component is not being developed and maintained, and appears not many are using it. In the past two years, there has been only one bug fixed (HDFS-13322). https:/

Re: [DISCUSS] Deprecate fuse-dfs

com/Microsoft/hdfs-mount . Should we ignore fuse-dfs and > use or implement the other? > > Regards, > Shingo > > On 2018/10/01 20:10:07, Wei-Chiu Chuang wrote: > > Hi fellow Hadoop developers,> > > > > I want to start this thread to raise the awareness of the qu

Re: Hadoop 3.2 Release Plan proposal

Thanks Sunil G for driving the release, I filed HADOOP-15866 for a compat fix. If any one has cycle please review it, as I think it is needed for 3.2.0. On Thu, Oct 18, 2018 at 4:43 AM Sunil G wrote: > Hi Folks, > > As we previously communicat

[DISCUSS] Hadoop RPC encryption performance improvements

Folks, I would like to invite all to discuss the various Hadoop RPC encryption performance improvements. As you probably know, Hadoop RPC encryption currently relies on Java SASL, and have _really_ bad performance (in terms of number of RPCs per second, around 15~20% of the one without SASL) Ther

Re: [DISCUSS] Hadoop RPC encryption performance improvements

Ping. Any one? Cloudera is interested in moving forward with the RPC encryption improvements, but I just like to get a consensus which approach to go with. Otherwise I'll pick HADOOP-10768 since it's ready for commit, and I've spent time on testing it. On Thu, Oct 25, 2018 at 11

Re: [DISCUSS] Hadoop RPC encryption performance improvements

tch (sans TLS) up this week if that > > helps. > > > > > > The two cited jiras had some critical flaws. Skimming my comments, > both > > > use blocking IO (obvious nonstarter). HADOOP-10768 is a hand rolled > > > TLS-like encryption which I don't fee

Re: [DISCUSS] Hadoop RPC encryption performance improvements

l flaws. Skimming my comments, both > use blocking IO (obvious nonstarter). HADOOP-10768 is a hand rolled > TLS-like encryption which I don't feel is something the community can or > should maintain from a security standpoint. > > Daryn > > On Wed, Oct 31, 2018 at 8:4

Re: Can someone add me to the JIRA admin list

Quick questions here, Do committers get added to admin role on all Hadoop projects (i.e. HADOOP, HDFS, YARN, MAPREDUCE and HDDS) or just a subset of them? I am admin on HADOOP/HDFS and for the most part that works fine for me, because I primarily only work on HDFS and Hadoop Common space. And that'

Re: Can someone add me to the JIRA admin list

mitters if we can't > trust someone to not try to mess things up in one of the trackers we > have bigger problems. > On Thu, Dec 13, 2018 at 9:49 AM Wei-Chiu Chuang > wrote: > > > > Quick questions here, > > Do committers get added to admin role on all Hadoop projects (

Changing RPC SASL options without full cluster restart?

Hi fellow Hadoop developers, Do you know a way to change RPC SASL options without full cluster restart? (that is, rolling restart)? For example, enabling RPC encryption? Currently if you try to do rolling restart after enabling RPC encryption, applications such as HBase would fail to connect to Na

Re: [DISCUSS] Hadoop RPC encryption performance improvements

ncept soon. On Mon, Feb 18, 2019 at 1:37 PM Jonathan Eagles wrote: > Filed https://issues.apache.org/jira/browse/HADOOP-16119 to > implemented KMS on Hadoop RPC. It seemed there were interested parties > involved in the development of this feature. > > On Fri, Nov 2, 2018 at 4:35

Re: Update guava to 27.0-jre in hadoop-project

+1 I watched Gabor working on this and this is a very comprehensive, which also includes testing in downstreamers (HBase and Hive). So very good work. Thanks! On Wed, Apr 3, 2019 at 3:41 AM Steve Loughran wrote: > I am taking silence as happiness here. > > +1 to the patch > > On Tue, Apr 2, 20

Re: CredentialProvider API

A similar issue is https://issues.apache.org/jira/browse/HADOOP-15412 You can't put the keystore for KMS in HDFS, and I filed https://issues.apache.org/jira/browse/HADOOP-15413 to document it. Karthik, would you consider use HADOOP-15413 to document scenarios where it is not supported? Additional

Re: [DISCUSS] Release 3.0.4 or branch-3.0 EOL

Thanks for initiating the discussion, Akira. I've given similar thoughts on the possible EOL of Hadoop branch-3.0 line. IMHO, I'd love to see HDFS-13596 fixed to make it possible to rolling upgrade from Hadoop 2 to Hadoop 3.0, and then roll anothe

  1   2   3   4   5   6   7   8   >