Re: [E] [VOTE] Hadoop 3.1.x EOL

2021-06-07 Thread Kihwal Lee
+1 (binding) On Thu, Jun 3, 2021 at 1:14 AM Akira Ajisaka wrote: > Dear Hadoop developers, > > Given the feedback from the discussion thread [1], I'd like to start > an official vote > thread for the community to vote and start the 3.1 EOL process. > > What this entails: > > (1) an official anno

Re: data node reads blocks sequentially from disk

2020-03-11 Thread Kihwal Lee
When Datanode was initially designed, Linux AIO was still early in its adoption. Kernel support was there and the libraries were almost there. No java support, of course. We would have to write a lot of native code for it and use JNI. Also, AIO means bypassing kernel page cache since you are doing

Re: Changing the default Datanode Volume Choosing policy

2020-05-05 Thread Kihwal Lee
Successfully running on 1,000 clusters over 5 years proves the feature is stable. It does not, however, give me assurance that it will perform well in our env. It will be nice if there is some data on its performance. On obvious concern is, running into grossly unbalanced I/O load among drives. S

Re: Changing the default Datanode Volume Choosing policy

2020-05-12 Thread Kihwal Lee
to me. > > Also worth noting the implementation doesn't try to achieve very fine > grained balance in space. As long as all volumes has the available space > within a threshold (10GB by default), > it falls back to round-robin policy. > > On Tue, May 5, 202

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC1)

2020-06-23 Thread Kihwal Lee
Gabor, If you want to release asap, you can simply revert HDFS-14941 in the release branch for now. It is causing the issue and was committed after 3.1.3. This causes failure of the automated upgrade process and namenode memory leak. Kihwal On Tue, Jun 23, 2020 at 8:47 AM Akira Ajisaka wrote:

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

2020-10-06 Thread Kihwal Lee
Which layout change are you referring to? The only layout change I know of was done in 2.7, IIRC. We backported that to 2.6 and did not see any adverse effects at that time. Is datanode using more heap all the time? Or is it running into trouble when generating full block reports? Kihwal On Mon,

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

2020-10-06 Thread Kihwal Lee
To be clear, we are running 2.8 and 2.10. Although we don't see any issues, I am curious whether the change in heap usage is amplified on dense datanodes. Kihwal On Tue, Oct 6, 2020 at 5:00 PM Kihwal Lee wrote: > Which layout change are you referring to? The only layout change I know

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

2020-10-07 Thread Kihwal Lee
tell which commit increases heap usage worse during upgrade. > > > > On Tue, Oct 6, 2020 at 3:01 PM Kihwal Lee wrote: > >> Which layout change are you referring to? The only layout change I know >> of was done in 2.7, IIRC. We backported that to 2.6 and did not see any >

Re: [E] Re: [DISCUSS] Change project style guidelines to allow line length 100

2021-05-24 Thread Kihwal Lee
+1 for the 100 char limit. But I would have liked 132 columns more. :) Kihwal On Mon, May 24, 2021 at 1:46 PM Sean Busbey wrote: > Hi folks! > > The consensus seems pretty strongly in favor of increasing the line length > limit. Do folks still want to see a formal VOTE thread? > > > > On May 1

Re: [DISCUSS] About the details of JDK-8 support

2015-10-13 Thread Kihwal Lee
sun.security.krb5.KrbApReq was creating a static MD5 digest object and not synchronizing access. This has been fixed in jdk8u60. http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89 One of the visible symptom is RPC reader thread getting ArrayIndexOutOfBoundsException from sun.securi

Re: hadoop-hdfs-client splitoff is going to break code

2015-10-23 Thread Kihwal Lee
I am not sure whether it was mentioned by anyone before, butI noticed that client only changes do not trigger running anytest in hdfs-precommit. This is because hadoop-hdfs-client does nothave any test. Kihwal From: Colin P. McCabe To: "hdfs-dev@hadoop.apache.org" Cc: "common-...@hadoo

Re: hadoop-hdfs-client splitoff is going to break code

2015-10-23 Thread Kihwal Lee
I think a lot of "client-side" tests use MiniDFSCluster. I know mechanical division is possible, but what about test coverage? Kihwal From: Haohui Mai To: hdfs-dev@hadoop.apache.org; Kihwal Lee Cc: "common-...@hadoop.apache.org" Sent: Friday, October 23, 2015

Re: 2.7.2 release plan

2015-10-27 Thread Kihwal Lee
I think we need HDFS-8950 and HDFS-7725 in 2.7.2.It should be easy to backport/cherry-pick HDFS-7725. For HDFS-8950, it will be nice if Ming can chime in. Kihwal From: Tsuyoshi Ozawa To: "common-...@hadoop.apache.org" Cc: Chris Nauroth ; "yarn-...@hadoop.apache.org" ; "hdfs-dev@hadoop

Re: 2.7.2 release plan

2015-10-28 Thread Kihwal Lee
I will try to get them in or bug Daryn. HDFS-8498 doesn't seem a new bug, so I kicked it out to 2.7.3. Kihwal From: Vinod Vavilapalli To: "common-...@hadoop.apache.org" ; Kihwal Lee Cc: "hdfs-dev@hadoop.apache.org" ; Chris Nauroth ; "yarn-...@ha

Re: [VOTE] Release Apache Hadoop 2.7.2 RC0

2015-11-13 Thread Kihwal Lee
We found HDFS-9426. The rolling upgrade finalization is not backward compatible.I.e. 2.7.1 or 2.6.x datanodes will ignore finalization. So -1. Kihwal From: Vinod Kumar Vavilapalli To: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hado

Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

2015-12-21 Thread Kihwal Lee
+1 (binding)- Verified the signature/digest. - I've built the dist tree with the native support from the source.- Brought up a single node cluster and ran a set of basic tests. From: Vinod Kumar Vavilapalli To: Hadoop Common ; hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; ma

Re: [VOTE] Release hadoop-2.0.3-alpha

2013-02-12 Thread Kihwal Lee
+1 Verified checksums. Built from source, deployed and ran a couple of jobs. Kihwal On 2/6/13 9:59 PM, "Arun C Murthy" wrote: >Folks, > >I've created a release candidate (rc0) for hadoop-2.0.3-alpha that I >would like to release. > >This release contains several major enhancements such as QJM

Re: [VOTE] Release Apache Hadoop 0.23.7

2013-04-16 Thread Kihwal Lee
+1 I've checked out, built and deployed release-0.23.7-rc0 to a single node cluster. Ran a couple of sample jobs. Kihwal On 4/11/13 2:55 PM, "Thomas Graves" wrote: >I've created a release candidate (RC0) for hadoop-0.23.7 that I would like >to release. > >This release is a sustaining release wi

Re: [VOTE] Release Apache Hadoop 0.23.8

2013-06-03 Thread Kihwal Lee
+1 I've downloaded & built the RC and ran several tests on a single node cluster. Kihwal On 5/28/13 11:00 AM, "Thomas Graves" wrote: > >I've created a release candidate (RC0) for hadoop-0.23.8 that I would like >to release. > >This release is a sustaining release with several important bug fixe

Re: [VOTE] Release Apache Hadoop 2.0.5-alpha (rc2)

2013-06-04 Thread Kihwal Lee
+1 Built from source and ran a couple of jobs in a pseudo-distributed cluster. Kihwal On 6/3/13 2:51 PM, "Konstantin Boudnik" wrote: >I have rolled out release candidate (rc2) for hadoop-2.0.5-alpha. > >The difference between rc1 and rc2 is the "optimistic release date" is >set for >06/06/2013

Re: Why is FileSystem.createNonRecursive deprecated?

2013-06-13 Thread Kihwal Lee
It seems the transition within hadoop needs to happen before deprecating FileSystem. I think createNonRecursive() was marked deprecated from the beginning under the assumption that this transition would happen soon. We know that hasn't been the case. So the question is, do we transition the portion

Re: data loss after cluster wide power loss

2013-07-03 Thread Kihwal Lee
For the ext3 bug Colin mentioned, see https://bugzilla.redhat.com/show_bug.cgi?id=592961. This was fixed in 2.6.32 and backported in RHEL 5.4 (or CENTOS). This has more to do with file data and affects NN more. Since NN preallocates blocks for edits, almost all data writes are done without modifyin

Re: [VOTE] Release Apache Hadoop 0.23.9

2013-07-08 Thread Kihwal Lee
+1 Downloaded it and ran several sample tests in a pseudo-distributed cluster. Kihwal On 7/1/13 12:20 PM, "Thomas Graves" wrote: >I've created a release candidate (RC0) for hadoop-0.23.9 that I would like >to release. > >The RC is available at: >http://people.apache.org/~tgraves/hadoop-0.23.9-

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-08 Thread Kihwal Lee
Another blocker, HADOOP-9850, has been committed. Kihwal From: Arun C Murthy To: Daryn Sharp Cc: "" ; "mapreduce-...@hadoop.apache.org" ; "yarn-...@hadoop.apache.org" ; "common-...@hadoop.apache.org" Sent: Thursday, August 1, 2013 1:30 PM Subject: Re: [

Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845

2013-08-08 Thread Kihwal Lee
Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672. The version we are using has memory leak and inefficiency issues. We've seen users running into it. Kihwal From: Tsuyoshi OZAWA To: "common-...@hadoop.apache.org" Cc: "hdfs-dev@h

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-16 Thread Kihwal Lee
We have found HADOOP-9880, which prevents Namenode HA from running with security. Kihwal From: Arun C Murthy To: "common-...@hadoop.apache.org" ; "hdfs-dev@hadoop.apache.org" ; "mapreduce-...@hadoop.apache.org" ; "yarn-...@hadoop.apache.org" Sent: Thursda

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-16 Thread Kihwal Lee
We have found HADOOP-9880, which prevents Namenode HA from running with security. Kihwal From: Arun C Murthy To: "common-...@hadoop.apache.org" ; "hdfs-dev@hadoop.apache.org" ; "mapreduce-...@hadoop.apache.org" ; "yarn-...@hadoop.apache.org" Sent: Thurs

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-16 Thread Kihwal Lee
It's your call, Arun.  I.e. as long you believe rc2 meets the expectations and objectives of 2.1.0-beta. Kihwal From: Arun Murthy To: "common-...@hadoop.apache.org" Cc: Kihwal Lee ; "mapreduce-...@hadoop.apache.org" ; "hdfs-d

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-16 Thread Kihwal Lee
I've changed the target version of HADOOP-9880 to 2.1.1.  Please change it back, if you feel that it needs to be in 2.1.0-beta. Kihwal From: Kihwal Lee To: Arun Murthy ; "common-...@hadoop.apache.org" Cc: "mapreduce-...@hadoop.apac

Re: [VOTE] Release Apache Hadoop 2.2.0

2013-10-09 Thread Kihwal Lee
+1  Ran a set of tests against a single node cluster. Kihwal On Monday, October 7, 2013 2:02 AM, Arun C Murthy wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues w

Re: what is different between CorruptBlocks and MissingBlocks in NN FSNamesystem metrics?

2013-11-18 Thread Kihwal Lee
Missing blocks are the blocks with no valid replicas. I.e. data is not available. Corrupt blocks are the ones with at least one corrupt replica. They may have healthy live replicas that are not corrupt, which can be read by users or replicated by the system to replace the corrupt replicas.  "Mis

Re: [VOTE] Release Apache Hadoop 0.23.10

2013-12-03 Thread Kihwal Lee
I've built the tag and ran some basic tests on a single node cluster. +1 (binding) Kihwal On Tuesday, December 3, 2013 12:24 AM, Thomas Graves wrote: Hey Everyone, There have been lots of improvements and bug fixes that have went into branch-0.23 since the 0.23.9 release.  We think its ti

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

2017-12-11 Thread Kihwal Lee
+1 (binding) - checked out the rc1 tag and built the source (-Pdist -Pnative) - brought up a pseudo distributed cluster - ran sample MR jobs - verified web UIs working. On Thu, Dec 7, 2017 at 9:22 PM, Konstantin Shvachko wrote: > Hi everybody, > > I updated CHANGES.txt and fixed documentation li

Re: Apache Hadoop 3.0.1 Release plan

2018-02-08 Thread Kihwal Lee
HADOOP-14060 is a blocker. Daryn will add more detail to the jira or to this thread. On Thu, Feb 8, 2018 at 7:01 AM, Brahma Reddy Battula wrote: > Hi Eddy, > > HDFS-12990 got committed to 3.0.1,can we have RC for 3.0.1 (only YARN-5742 > blocker is open ) ? > > > On Sat, Feb 3, 2018 at 12:40 A

Broken build env

2018-05-17 Thread Kihwal Lee
Simple commit builds are failing often https://builds.apache.org/job/Hadoop-trunk-Commit/ Many trunk builds are failing on H19. "protoc version is 'libprotoc 2.6.1', expected version is '2.5.0' " On H4, a cmake version problem was seen. The commit builds don't seem to be running in a docker con

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

2018-09-12 Thread Kihwal Lee
+1 (binding) - Built from source - Brought up a single node cluster - Checked basic HDFS functions and checked the UIs - Ran several simple jobs. On Mon, Sep 10, 2018 at 7:01 AM 俊平堵 wrote: > Hi all, > > I've created the first release candidate (RC0) for Apache > Hadoop 2.8.5. This is our

Re: NN run progressively slower

2018-09-25 Thread Kihwal Lee
Are you using CMS? How big is young gen? How often does the NN do young gen collection when it is slow? On Tue, Sep 25, 2018 at 4:04 AM Lin,Yiqun(vip.com) wrote: > Hi hdfs developers: > > We meet a bad problem after rolling upgrade our hadoop version from > 2.5.0-cdh5.3.2 to 2.6.0-cdh5.13.1. The

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

2016-01-20 Thread Kihwal Lee
+1 (binding)Checked out the source and built.Ran basic hdfs and mapred tests on a single node cluster Kihwal From: Vinod Kumar Vavilapalli To: Hadoop Common ; hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Sent: Thursday, January 14, 2016 10:

Re: HDFS-8867 Patch

2016-01-25 Thread Kihwal Lee
Sergey, The release 2.7.2 is being voted on. It should get released very soon.The patch applies to 2.7.1 cleanly, so you could try it, if you can do build and dist on your own. Thanks,Kihwal From: Сергей Казаков To: hdfs-dev@hadoop.apache.org Sent: Monday, January 25, 2016 12:35 AM

Re: 'Target Version' field missing in Jira

2016-02-12 Thread Kihwal Lee
It's still here: https://issues.apache.org/jira/plugins/servlet/project-config/HDFS/fields But somehow not showing up on pages. Kihwal From: Arpit Agarwal To: "common-...@hadoop.apache.org" ; "hdfs-dev@hadoop.apache.org" Sent: Friday, February 12, 2016 1:52 PM Subject: 'Target Versi

Re: Looking to a Hadoop 3 release

2016-02-18 Thread Kihwal Lee
Moving Hadoop 3 forward sounds fine. If EC is one of the main motivations, are we getting rid of branch-2.8? Kihwal From: Andrew Wang To: "common-...@hadoop.apache.org" Cc: "yarn-...@hadoop.apache.org" ; "mapreduce-...@hadoop.apache.org" ; hdfs-dev Sent: Thursday, February 18, 201

Re: 2.7.3 release plan

2016-04-06 Thread Kihwal Lee
Just reverted HDFS-8791 from branch-2.7.Eulogy: Although it has ascended to a better version, it did caught an upgrade bug while in branch-2.7. Kihwal From: Vinod Kumar Vavilapalli To: yarn-...@hadoop.apache.org Cc: Hadoop Common ; hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apach

Re: HDFS Federation

2016-04-28 Thread Kihwal Lee
Kun, (1) The client-facing counter part of federation is ViewFileSystem, aka client side mount table.(2) Federation is supported in 2.7. There are test cases bringing up federated mini cluster, so I assume setting up a pseudo distributed cluster is possible. I am not sure whether all support sc

Re: handlerCount

2016-04-28 Thread Kihwal Lee
For name node, start from NameNodeRpcServer. From: Kun Ren To: hdfs-dev@hadoop.apache.org Sent: Thursday, April 28, 2016 4:51 PM Subject: handlerCount Hi Genius, I have a quick question: I remembered I saw the default value for HandlerCout is 10(The number of Handler threads), but

Re: [Federation setup] Adding a new name node to federated cluster

2015-04-02 Thread Kihwal Lee
You might be issuing the refresh command against the dataXfer port, not the rpc port of the datanode. -Kihwal From: Ajith shetty To: "hdfs-dev@hadoop.apache.org" Sent: Wednesday, April 1, 2015 1:43 AM Subject: [Federation setup] Adding a new name node to federated cluster Hi all,

Re: [VOTE] Release Apache Hadoop 2.7.0 RC0

2015-04-15 Thread Kihwal Lee
+1 (binding)Built the source from the tag and ran basic test on pseudo distributed cluster. Kihwal From: Vinod Kumar Vavilapalli To: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Cc: vino...@apache.org Sent: Frida

Re: Problem in NetUtils

2011-10-12 Thread Kihwal Lee
0.21 is not actively maintained. The bug was fixed in other active branches (0.20-security, 0.22, 0.23, etc.) and trunk. Kihwal On 10/12/11 7:50 PM, "Beckham007" wrote: Hi, Today, I use org.apache.hadoop.net.NetUtils Class to change IP address and hostname. normalizeHostName but I think the

PreCommit-HDFS-Build stuck

2012-08-22 Thread Kihwal Lee
#3073 is stuck. It has been saying its running TestHASafMode for more than three hours. It's not timing out. Can Somebody with a Jenkins access please bounce this?

Re: PreCommit-HDFS-Build stuck

2012-08-23 Thread Kihwal Lee
al on the master or slave would have made a difference. Kihwal On 8/22/12 11:49 PM, "Kihwal Lee" wrote: >#3073 is stuck. It has been saying its running TestHASafMode for more >than three hours. It's not timing out. > >Can Somebody with a Jenkins access please bounce this? > >

Re: [DISCUSS] Retire BKJM from trunk?

2016-07-28 Thread Kihwal Lee
+1 From: Brahma Reddy Battula To: "Gangumalla, Uma" ; Rakesh Radhakrishnan ; Sijie Guo Cc: "d...@bookkeeper.apache.org" ; Uma gangumalla ; Vinayakumar B ; "hdfs-dev@hadoop.apache.org" ; "u...@hadoop.apache.org" ; "u...@bookkeeper.apache.org" Sent: Thursday, July 28, 2016 4:21 AM

Re: [Release thread] 2.6.5 release activities

2016-08-12 Thread Kihwal Lee
You might want the snapshot bug fix done in HDFS-7056. This bug creates snapshot filediffs even if you never use snapshot. For 2.6, we will have to do it in a separate jira to pick up the fix only. Related to this, HDFS-9696 might be of interest too. Kihwal From: Chris Trezzo To: Karthi

Re: can someone add me to contributors for hte HDFS jira?

2016-08-16 Thread Kihwal Lee
Done From: Sean Busbey To: "hdfs-dev@hadoop.apache.org" Sent: Tuesday, August 16, 2016 1:05 PM Subject: can someone add me to contributors for hte HDFS jira? Hi! I'm already in the contributor role on the HADOOP jira, could someone add me as one on the HDFS jira? I'd like the abil

Re: Active NameNode Image Download and Edit Log Roll Stuck in FileChannel Force/Truncate

2016-09-13 Thread Kihwal Lee
Is the system busy with I/O when it happens? Any other I/O activities preceding the event? In your case DistCp could have generated extra edits and also namenode daemon and audit log entries.  Depending on configuration, dirty pages can pile up quite a bit on Linux systems with a large memory an

Is anyone seeing this during trunk build?

2016-09-28 Thread Kihwal Lee
I just noticed this during a trunk build. I was doing "mvn clean install -DskipTests".  The build succeeds. Is anyone seeing this?  I am using openjdk8u102. === [WARNING] Unable to process class org/apache/hadoop/hdfs/StripeReader.class in JarAnalyzer File /home1/kihwal/devel/apache/hadoo

Re: Is anyone seeing this during trunk build?

2016-09-29 Thread Kihwal Lee
fault locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.11.3", arch: "x86_64", family: "mac" FYI On Wed, Sep 28, 2016 at 3:54 PM, Kihwal Lee wrote: > I just noticed this during a trunk build. I was doing "mvn clean install &g

Re: Is anyone seeing this during trunk build?

2016-09-30 Thread Kihwal Lee
on did the workaround. I will file a jira to update maven-project-info-reports-plugin. Kihwal From: Arun Suresh To: Kihwal Lee Cc: Ted Yu ; Hdfs-dev ; Hadoop Common Sent: Thursday, September 29, 2016 6:58 PM Subject: Re: Is anyone seeing this during trunk build? It looks l

Re: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to 2.5.2

2016-10-14 Thread Kihwal Lee
It looks like the NN is having trouble after reading in the VERSION file, which contains the new layout version. We can continue the discussion in the jira. Kihwal From: Ravi Prakash To: Dinesh Kumar Prabakaran Cc: user ; hdfs-dev Sent: Thursday, October 13, 2016 3:29 PM Subject: R

Re: RollingUpgrade Rollback openfiles issue

2017-05-25 Thread Kihwal Lee
Hi Vinay, If I rephrase the question, Does a RU rollback snapshot provide a consistent snapshot of the distributed  file system? I don't think we aimed it to be a completely consistent snapshot. It is meant to be a safe place to go back with the old version of software.  This is normally used as

Re: RollingUpgrade Rollback openfiles issue

2017-05-26 Thread Kihwal Lee
Kihwal From: Vinayakumar B To: Kihwal Lee ; "hdfs-dev@hadoop.apache.org" Sent: Friday, May 26, 2017 5:05 AM Subject: RE: RollingUpgrade Rollback openfiles issue Thanks Kihwal, Found another case, which looks more dangerous. 1. File written and closed. FINALIZ

Re: Apache Hadoop 2.8.2 Release Plan

2017-07-21 Thread Kihwal Lee
Thanks for driving the next 2.8 release, Junping. While I was committing a blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 2.8.2. Thanks,Kihwal On Thursday, July 20, 2017, 3:32:16 PM CDT, Junp

Re: [VOTE] Merge HDFS-5535 Rolling Upgrade Improvement

2014-02-27 Thread Kihwal Lee
gt; The other remaining works are: > - Revise the design doc > - Post a test plan (Haohui is working on it.) > - Execute the manual tests (Haohui and Fengdong will work on it.) > > The work was a collective effort of Nathan Roberts, Sanjay Radia, Suresh > Srinivas, Kihwal Lee, Jing

Re: [VOTE] Release Apache Hadoop 2.4.1

2014-06-20 Thread Kihwal Lee
If we ever respin 2.4.1, I strongly suggest HDFS-6527 be included. Kihwal On 6/19/14, 4:56 PM, "Akira AJISAKA" wrote: >I think we should include this issue in 2.4.1, so I uploaded a patch to >fix it. I'll appreciate your review. > >Thanks, >Akira > >(2014/06/18 12:13), Vinod Kumar Vavilapalli

Re: [VOTE] Release Apache Hadoop 0.23.11

2014-06-20 Thread Kihwal Lee
Checked out the source, built and started a single node cluster. Ran a couple of sample jobs. +1 (binding) Kihwal On 6/19/14, 10:14 AM, "Thomas Graves" wrote: >Hey Everyone, > >There have been various bug fixes that have went into >branch-0.23 since the 0.23.10 release. We think its time to d

Re: [VOTE] Release Apache Hadoop 2.4.1

2014-06-20 Thread Kihwal Lee
>> Thanks for the feedback Vinod, Akira & Kihwal. >> >> I'll re-spin rc1 with MAPREDUCE-5830 & HDFS-6527. >> >> @Kihwal - Can you, please, merge HDFS-6527 to branch-2.4 and >>branch-2.4.1? >> >> thanks, >> Arun >> >> On Ju

Re: [VOTE] Change by-laws on release votes: 5 days instead of 7

2014-06-25 Thread Kihwal Lee
+1 (binding) Kihwal On 6/24/14, 3:53 AM, "Arun C Murthy" wrote: >Folks, > > As discussed, I'd like to call a vote on changing our by-laws to change >release votes from 7 days to 5. > > I've attached the change to by-laws I'm proposing. > > Please vote, the vote will the usual period of 7 days.

Re: Jenkins build fails

2014-07-09 Thread Kihwal Lee
Now builds are failing because of this. Please make sure build works with "-Pnative". [exec] CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message): [exec] Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) [exec] Call Stack (most recent

Re: [VOTE] Migration from subversion to git for version control

2014-08-11 Thread Kihwal Lee
+1 (binding) Kihwal On 8/8/14, 9:57 PM, "Karthik Kambatla" wrote: >I have put together this proposal based on recent discussion on this >topic. > >Please vote on the proposal. The vote runs for 7 days. > > 1. Migrate from subversion to git for version control. > 2. Force-push to be disabled

Erratic Jenkins behavior

2015-02-05 Thread Kihwal Lee
I am sure many of us have seen strange jenkins behavior out of the precommit builds. - build artifacts missing - serving build artifact belonging to another build. This also causes wrong precommit results to be posted on the bug. - etc. The latest one I saw is disappearance of the unit test st

Re: Questions about HDFS federation

2015-03-03 Thread Kihwal Lee
I suggest testing without federation first. That is, running two separate yarn/hdfs instances. Once that works properly, you can introduce federation in your config. Kihwal From: xeonmailinglist To: hdfs-dev@hadoop.apache.org Sent: Tuesday, March 3, 2015 4:32 AM Subject: Questions abo

Re: Questions about HDFS federation

2015-03-03 Thread Kihwal Lee
Federation is for sharing the data node storage spaces, not for sharing the data among multiple clusters. Check out viewfs instead. From: xeonmailinglist-yahoo To: hdfs-dev@hadoop.apache.org Sent: Tuesday, March 3, 2015 12:31 PM Subject: Re: Questions about HDFS federation I would

Re: HDFS Scalability Limit?

2019-06-13 Thread Kihwal Lee
Hi Wei-Chiu, We have experience with 5,000 - 6,000 node clusters. Although it ran/runs fine, any heavy hitter activities such as decommissioning needed to be carefully planned. In terms of files and blocks, we have multiple clusters running stable with over 500M files and blocks. Some at over

Re: Block capacity

2019-06-25 Thread Kihwal Lee
It just means that hash collisions will be more frequent above the capacity, causing more walks on hash chains. It is unclear at what level you will see meaningful impact. In any case, it is not really a limit. Kihwal On Tue, Jun 25, 2019 at 2:17 AM Lars Francke wrote: > Hi, > > I stumbled upo

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

2019-08-21 Thread Kihwal Lee
+1 Kihwal On Tue, Aug 20, 2019 at 10:03 PM Wangda Tan wrote: > Hi all, > > This is a vote thread to mark any versions smaller than 2.7 (inclusive), > and 3.0 EOL. This is based on discussions of [1] > > This discussion runs for 7 days and will conclude on Aug 28 Wed. > > Please feel free to sha

Re: [Help] NameNode error

2019-08-27 Thread Kihwal Lee
That's not supposed to happen. What version of Hadoop are you using? It Please file a jira with details including how the namenodes are configured. For the recovery: First and foremost, do not shut down the active namenode. Put it into safe mode and issue a saveNamespace command to create a checkp

Re: Number of bytes per checksum

2011-06-24 Thread Kihwal Lee
Doing CRC32 on a huge data block also reduces its error detection capability. If you need more information on this topic, this paper will be a good starting poing: http://www.ece.cmu.edu/~koopman/networks/dsn02/dsn02_koopman.pdf Kihwal On 6/24/11 9:50 AM, "Doug Cutting" wrote: > A smaller c

Re: corrupted edits log after power failure

2011-09-22 Thread Kihwal Lee
Does the backup process include syncing? On-drive write cache can also trick you. For absolutely critical data, it is a good idea to use a controller with battery-backed write cache or a service/product that guarantees durability. Kihwal On 9/22/11 3:48 AM, "Gabi Kazav" wrote: Hi, I had Powe

[jira] [Resolved] (HDFS-16123) NameNode standby checkpoint is abnormal, because the data disk is full, even if the disk space is restored

2021-07-12 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-16123. --- Resolution: Invalid > NameNode standby checkpoint is abnormal, because the data disk is full, e

[jira] [Created] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-13 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-16127: - Summary: Improper pipeline close recovery causes a permanent write failure or data loss. Key: HDFS-16127 URL: https://issues.apache.org/jira/browse/HDFS-16127 Project

[jira] [Resolved] (HDFS-5441) Wrong use of catalina opts in httpfs.sh

2019-10-31 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-5441. -- Resolution: Won't Fix It moved on to Jetty. No longer relevant. > Wrong use of catalina

[jira] [Created] (HDFS-14948) Improve HttpFS Server

2019-10-31 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-14948: - Summary: Improve HttpFS Server Key: HDFS-14948 URL: https://issues.apache.org/jira/browse/HDFS-14948 Project: Hadoop HDFS Issue Type: Improvement

[jira] [Created] (HDFS-14949) HttpFS does not support getServerDefaults()

2019-10-31 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-14949: - Summary: HttpFS does not support getServerDefaults() Key: HDFS-14949 URL: https://issues.apache.org/jira/browse/HDFS-14949 Project: Hadoop HDFS Issue Type

[jira] [Resolved] (HDFS-4935) add symlink support to HttpFS server side

2020-01-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-4935. -- Resolution: Won't Fix Symlink is currently disabled due to security reasons and bugs. > add

[jira] [Created] (HDFS-15181) Webhdfs getTrashRoot() caused internal AccessControlException

2020-02-18 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15181: - Summary: Webhdfs getTrashRoot() caused internal AccessControlException Key: HDFS-15181 URL: https://issues.apache.org/jira/browse/HDFS-15181 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-15147) LazyPersistTestCase wait logic is error-prone

2020-02-27 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15147. --- Hadoop Flags: Reviewed Resolution: Fixed > LazyPersistTestCase wait logic is error-pr

[jira] [Created] (HDFS-15203) A bug in ViewFileSystemBaseTest

2020-03-03 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15203: - Summary: A bug in ViewFileSystemBaseTest Key: HDFS-15203 URL: https://issues.apache.org/jira/browse/HDFS-15203 Project: Hadoop HDFS Issue Type: Bug

[jira] [Created] (HDFS-15287) HDFS rollingupgrade prepare never finishes

2020-04-20 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15287: - Summary: HDFS rollingupgrade prepare never finishes Key: HDFS-15287 URL: https://issues.apache.org/jira/browse/HDFS-15287 Project: Hadoop HDFS Issue Type: Bug

[jira] [Resolved] (HDFS-15350) Set dfs.client.failover.random.order to true as default

2020-05-12 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15350. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed >

[jira] [Resolved] (HDFS-15348) [SBN Read] IllegalStateException happened when doing failover

2020-05-12 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15348. --- Resolution: Duplicate > [SBN Read] IllegalStateException happened when doing failo

[jira] [Created] (HDFS-15357) Do not trust bad block reports from clients

2020-05-15 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15357: - Summary: Do not trust bad block reports from clients Key: HDFS-15357 URL: https://issues.apache.org/jira/browse/HDFS-15357 Project: Hadoop HDFS Issue Type: Bug

[jira] [Resolved] (HDFS-15287) HDFS rollingupgrade prepare never finishes

2020-06-08 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15287. --- Resolution: Duplicate > HDFS rollingupgrade prepare never finis

[jira] [Created] (HDFS-15421) IBR leak causes standby NN to be stuck in safe mode

2020-06-19 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15421: - Summary: IBR leak causes standby NN to be stuck in safe mode Key: HDFS-15421 URL: https://issues.apache.org/jira/browse/HDFS-15421 Project: Hadoop HDFS Issue Type

[jira] [Created] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.

2020-06-19 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15422: - Summary: Reported IBR is partially replaced with stored info when queuing. Key: HDFS-15422 URL: https://issues.apache.org/jira/browse/HDFS-15422 Project: Hadoop HDFS

[jira] [Created] (HDFS-15726) Client should only complete a file if the last block is finalized

2020-12-10 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15726: - Summary: Client should only complete a file if the last block is finalized Key: HDFS-15726 URL: https://issues.apache.org/jira/browse/HDFS-15726 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-15824) Update to enable TLS >=1.2 as default secure protocols

2021-02-08 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15824. --- Resolution: Invalid > Update to enable TLS >=1.2 as default secure pro

[jira] [Resolved] (HDFS-15825) Using a cryptographically weak Pseudo Random Number Generator (PRNG)

2021-02-08 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15825. --- Resolution: Invalid > Using a cryptographically weak Pseudo Random Number Generator (P

[jira] [Created] (HDFS-16034) Disk failures exceeding the DFIP threshold do not shutdown datanode

2021-05-21 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-16034: - Summary: Disk failures exceeding the DFIP threshold do not shutdown datanode Key: HDFS-16034 URL: https://issues.apache.org/jira/browse/HDFS-16034 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-16034) Disk failures exceeding the DFIP threshold do not shutdown datanode

2021-05-21 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-16034. --- Resolution: Incomplete > Disk failures exceeding the DFIP threshold do not shutdown datan

[jira] [Created] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad

2015-09-30 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-9178: Summary: Slow datanode I/O can cause a wrong node to be marked bad Key: HDFS-9178 URL: https://issues.apache.org/jira/browse/HDFS-9178 Project: Hadoop HDFS Issue

[jira] [Created] (HDFS-9208) Disabling atime may fail clients like distCp

2015-10-07 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-9208: Summary: Disabling atime may fail clients like distCp Key: HDFS-9208 URL: https://issues.apache.org/jira/browse/HDFS-9208 Project: Hadoop HDFS Issue Type: Bug

[jira] [Resolved] (HDFS-5032) Write pipeline failures caused by slow or busy disk may not be handled properly.

2015-10-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-5032. -- Resolution: Fixed > Write pipeline failures caused by slow or busy disk may not be handled > pr

  1   2   3   4   >