[jira] [Created] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-13 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-16127: - Summary: Improper pipeline close recovery causes a permanent write failure or data loss. Key: HDFS-16127 URL: https://issues.apache.org/jira/browse/HDFS-16127 Project

[jira] [Resolved] (HDFS-16123) NameNode standby checkpoint is abnormal, because the data disk is full, even if the disk space is restored

2021-07-12 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-16123. --- Resolution: Invalid > NameNode standby checkpoint is abnormal, because the data disk is full, e

Re: [E] [VOTE] Hadoop 3.1.x EOL

2021-06-07 Thread Kihwal Lee
+1 (binding) On Thu, Jun 3, 2021 at 1:14 AM Akira Ajisaka wrote: > Dear Hadoop developers, > > Given the feedback from the discussion thread [1], I'd like to start > an official vote > thread for the community to vote and start the 3.1 EOL process. > > What this entails: > > (1) an official anno

Re: [E] Re: [DISCUSS] Change project style guidelines to allow line length 100

2021-05-24 Thread Kihwal Lee
+1 for the 100 char limit. But I would have liked 132 columns more. :) Kihwal On Mon, May 24, 2021 at 1:46 PM Sean Busbey wrote: > Hi folks! > > The consensus seems pretty strongly in favor of increasing the line length > limit. Do folks still want to see a formal VOTE thread? > > > > On May 1

[jira] [Resolved] (HDFS-16034) Disk failures exceeding the DFIP threshold do not shutdown datanode

2021-05-21 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-16034. --- Resolution: Incomplete > Disk failures exceeding the DFIP threshold do not shutdown datan

[jira] [Created] (HDFS-16034) Disk failures exceeding the DFIP threshold do not shutdown datanode

2021-05-21 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-16034: - Summary: Disk failures exceeding the DFIP threshold do not shutdown datanode Key: HDFS-16034 URL: https://issues.apache.org/jira/browse/HDFS-16034 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-15825) Using a cryptographically weak Pseudo Random Number Generator (PRNG)

2021-02-08 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15825. --- Resolution: Invalid > Using a cryptographically weak Pseudo Random Number Generator (P

[jira] [Resolved] (HDFS-15824) Update to enable TLS >=1.2 as default secure protocols

2021-02-08 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15824. --- Resolution: Invalid > Update to enable TLS >=1.2 as default secure pro

[jira] [Created] (HDFS-15726) Client should only complete a file if the last block is finalized

2020-12-10 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15726: - Summary: Client should only complete a file if the last block is finalized Key: HDFS-15726 URL: https://issues.apache.org/jira/browse/HDFS-15726 Project: Hadoop HDFS

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

2020-10-07 Thread Kihwal Lee
tell which commit increases heap usage worse during upgrade. > > > > On Tue, Oct 6, 2020 at 3:01 PM Kihwal Lee wrote: > >> Which layout change are you referring to? The only layout change I know >> of was done in 2.7, IIRC. We backported that to 2.6 and did not see any >

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

2020-10-06 Thread Kihwal Lee
To be clear, we are running 2.8 and 2.10. Although we don't see any issues, I am curious whether the change in heap usage is amplified on dense datanodes. Kihwal On Tue, Oct 6, 2020 at 5:00 PM Kihwal Lee wrote: > Which layout change are you referring to? The only layout change I know

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

2020-10-06 Thread Kihwal Lee
Which layout change are you referring to? The only layout change I know of was done in 2.7, IIRC. We backported that to 2.6 and did not see any adverse effects at that time. Is datanode using more heap all the time? Or is it running into trouble when generating full block reports? Kihwal On Mon,

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC1)

2020-06-23 Thread Kihwal Lee
Gabor, If you want to release asap, you can simply revert HDFS-14941 in the release branch for now. It is causing the issue and was committed after 3.1.3. This causes failure of the automated upgrade process and namenode memory leak. Kihwal On Tue, Jun 23, 2020 at 8:47 AM Akira Ajisaka wrote:

[jira] [Created] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.

2020-06-19 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15422: - Summary: Reported IBR is partially replaced with stored info when queuing. Key: HDFS-15422 URL: https://issues.apache.org/jira/browse/HDFS-15422 Project: Hadoop HDFS

[jira] [Created] (HDFS-15421) IBR leak causes standby NN to be stuck in safe mode

2020-06-19 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15421: - Summary: IBR leak causes standby NN to be stuck in safe mode Key: HDFS-15421 URL: https://issues.apache.org/jira/browse/HDFS-15421 Project: Hadoop HDFS Issue Type

[jira] [Resolved] (HDFS-15287) HDFS rollingupgrade prepare never finishes

2020-06-08 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15287. --- Resolution: Duplicate > HDFS rollingupgrade prepare never finis

[jira] [Created] (HDFS-15357) Do not trust bad block reports from clients

2020-05-15 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15357: - Summary: Do not trust bad block reports from clients Key: HDFS-15357 URL: https://issues.apache.org/jira/browse/HDFS-15357 Project: Hadoop HDFS Issue Type: Bug

Re: Changing the default Datanode Volume Choosing policy

2020-05-12 Thread Kihwal Lee
to me. > > Also worth noting the implementation doesn't try to achieve very fine > grained balance in space. As long as all volumes has the available space > within a threshold (10GB by default), > it falls back to round-robin policy. > > On Tue, May 5, 202

[jira] [Resolved] (HDFS-15348) [SBN Read] IllegalStateException happened when doing failover

2020-05-12 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15348. --- Resolution: Duplicate > [SBN Read] IllegalStateException happened when doing failo

[jira] [Resolved] (HDFS-15350) Set dfs.client.failover.random.order to true as default

2020-05-12 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15350. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed >

Re: Changing the default Datanode Volume Choosing policy

2020-05-05 Thread Kihwal Lee
Successfully running on 1,000 clusters over 5 years proves the feature is stable. It does not, however, give me assurance that it will perform well in our env. It will be nice if there is some data on its performance. On obvious concern is, running into grossly unbalanced I/O load among drives. S

[jira] [Created] (HDFS-15287) HDFS rollingupgrade prepare never finishes

2020-04-20 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15287: - Summary: HDFS rollingupgrade prepare never finishes Key: HDFS-15287 URL: https://issues.apache.org/jira/browse/HDFS-15287 Project: Hadoop HDFS Issue Type: Bug

Re: data node reads blocks sequentially from disk

2020-03-11 Thread Kihwal Lee
When Datanode was initially designed, Linux AIO was still early in its adoption. Kernel support was there and the libraries were almost there. No java support, of course. We would have to write a lot of native code for it and use JNI. Also, AIO means bypassing kernel page cache since you are doing

[jira] [Created] (HDFS-15203) A bug in ViewFileSystemBaseTest

2020-03-03 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15203: - Summary: A bug in ViewFileSystemBaseTest Key: HDFS-15203 URL: https://issues.apache.org/jira/browse/HDFS-15203 Project: Hadoop HDFS Issue Type: Bug

[jira] [Resolved] (HDFS-15147) LazyPersistTestCase wait logic is error-prone

2020-02-27 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15147. --- Hadoop Flags: Reviewed Resolution: Fixed > LazyPersistTestCase wait logic is error-pr

[jira] [Created] (HDFS-15181) Webhdfs getTrashRoot() caused internal AccessControlException

2020-02-18 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15181: - Summary: Webhdfs getTrashRoot() caused internal AccessControlException Key: HDFS-15181 URL: https://issues.apache.org/jira/browse/HDFS-15181 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-4935) add symlink support to HttpFS server side

2020-01-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-4935. -- Resolution: Won't Fix Symlink is currently disabled due to security reasons and bugs. > add

[jira] [Created] (HDFS-14949) HttpFS does not support getServerDefaults()

2019-10-31 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-14949: - Summary: HttpFS does not support getServerDefaults() Key: HDFS-14949 URL: https://issues.apache.org/jira/browse/HDFS-14949 Project: Hadoop HDFS Issue Type

[jira] [Created] (HDFS-14948) Improve HttpFS Server

2019-10-31 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-14948: - Summary: Improve HttpFS Server Key: HDFS-14948 URL: https://issues.apache.org/jira/browse/HDFS-14948 Project: Hadoop HDFS Issue Type: Improvement

[jira] [Resolved] (HDFS-5441) Wrong use of catalina opts in httpfs.sh

2019-10-31 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-5441. -- Resolution: Won't Fix It moved on to Jetty. No longer relevant. > Wrong use of catalina

[jira] [Created] (HDFS-14892) Close the output stream if createWrappedOutputStream() fails

2019-10-04 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-14892: - Summary: Close the output stream if createWrappedOutputStream() fails Key: HDFS-14892 URL: https://issues.apache.org/jira/browse/HDFS-14892 Project: Hadoop HDFS

Re: [Help] NameNode error

2019-08-27 Thread Kihwal Lee
That's not supposed to happen. What version of Hadoop are you using? It Please file a jira with details including how the namenodes are configured. For the recovery: First and foremost, do not shut down the active namenode. Put it into safe mode and issue a saveNamespace command to create a checkp

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

2019-08-21 Thread Kihwal Lee
+1 Kihwal On Tue, Aug 20, 2019 at 10:03 PM Wangda Tan wrote: > Hi all, > > This is a vote thread to mark any versions smaller than 2.7 (inclusive), > and 3.0 EOL. This is based on discussions of [1] > > This discussion runs for 7 days and will conclude on Aug 28 Wed. > > Please feel free to sha

Re: Block capacity

2019-06-25 Thread Kihwal Lee
It just means that hash collisions will be more frequent above the capacity, causing more walks on hash chains. It is unclear at what level you will see meaningful impact. In any case, it is not really a limit. Kihwal On Tue, Jun 25, 2019 at 2:17 AM Lars Francke wrote: > Hi, > > I stumbled upo

Re: HDFS Scalability Limit?

2019-06-13 Thread Kihwal Lee
Hi Wei-Chiu, We have experience with 5,000 - 6,000 node clusters. Although it ran/runs fine, any heavy hitter activities such as decommissioning needed to be carefully planned. In terms of files and blocks, we have multiple clusters running stable with over 500M files and blocks. Some at over

[jira] [Created] (HDFS-14521) Suppress setReplication logging while replaying edits

2019-05-29 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-14521: - Summary: Suppress setReplication logging while replaying edits Key: HDFS-14521 URL: https://issues.apache.org/jira/browse/HDFS-14521 Project: Hadoop HDFS Issue

[jira] [Created] (HDFS-14464) Remove unnecessary log message from DFSInputStream

2019-05-02 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-14464: - Summary: Remove unnecessary log message from DFSInputStream Key: HDFS-14464 URL: https://issues.apache.org/jira/browse/HDFS-14464 Project: Hadoop HDFS Issue Type

[jira] [Created] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock

2018-11-26 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-14101: - Summary: Random failure of testListCorruptFilesCorruptedBlock Key: HDFS-14101 URL: https://issues.apache.org/jira/browse/HDFS-14101 Project: Hadoop HDFS Issue

Re: NN run progressively slower

2018-09-25 Thread Kihwal Lee
Are you using CMS? How big is young gen? How often does the NN do young gen collection when it is slow? On Tue, Sep 25, 2018 at 4:04 AM Lin,Yiqun(vip.com) wrote: > Hi hdfs developers: > > We meet a bad problem after rolling upgrade our hadoop version from > 2.5.0-cdh5.3.2 to 2.6.0-cdh5.13.1. The

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

2018-09-12 Thread Kihwal Lee
+1 (binding) - Built from source - Brought up a single node cluster - Checked basic HDFS functions and checked the UIs - Ran several simple jobs. On Mon, Sep 10, 2018 at 7:01 AM 俊平堵 wrote: > Hi all, > > I've created the first release candidate (RC0) for Apache > Hadoop 2.8.5. This is our

[jira] [Resolved] (HDFS-13889) The hadoop3.x client have compatible problem with hadoop2.x cluster

2018-09-04 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-13889. --- Resolution: Duplicate > The hadoop3.x client have compatible problem with hadoop2.x clus

[jira] [Resolved] (HDFS-13751) "hdfs dfs -du -h /*" shows local dir instead of HDFS dir

2018-07-20 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-13751. --- Resolution: Invalid > "hdfs dfs -du -h /*" shows local dir instea

Broken build env

2018-05-17 Thread Kihwal Lee
Simple commit builds are failing often https://builds.apache.org/job/Hadoop-trunk-Commit/ Many trunk builds are failing on H19. "protoc version is 'libprotoc 2.6.1', expected version is '2.5.0' " On H4, a cmake version problem was seen. The commit builds don't seem to be running in a docker con

[jira] [Resolved] (HDFS-13206) IllegalStateException: Unable to finalize edits file

2018-03-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-13206. --- Resolution: Invalid > IllegalStateException: Unable to finalize edits f

[jira] [Resolved] (HDFS-13169) Ambari UI deploy fails during startup of Ambari Metrics

2018-02-20 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-13169. --- Resolution: Invalid > Ambari UI deploy fails during startup of Ambari Metr

[jira] [Reopened] (HDFS-13120) Snapshot diff could be corrupted after concat

2018-02-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-13120: --- This breaks 2.8 compilation. Reverting... > Snapshot diff could be corrupted after con

Re: Apache Hadoop 3.0.1 Release plan

2018-02-08 Thread Kihwal Lee
HADOOP-14060 is a blocker. Daryn will add more detail to the jira or to this thread. On Thu, Feb 8, 2018 at 7:01 AM, Brahma Reddy Battula wrote: > Hi Eddy, > > HDFS-12990 got committed to 3.0.1,can we have RC for 3.0.1 (only YARN-5742 > blocker is open ) ? > > > On Sat, Feb 3, 2018 at 12:40 A

[jira] [Resolved] (HDFS-4411) Asserts are disabled in unit tests

2017-12-12 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-4411. -- Resolution: Invalid > Asserts are disabled in unit te

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

2017-12-11 Thread Kihwal Lee
+1 (binding) - checked out the rc1 tag and built the source (-Pdist -Pnative) - brought up a pseudo distributed cluster - ran sample MR jobs - verified web UIs working. On Thu, Dec 7, 2017 at 9:22 PM, Konstantin Shvachko wrote: > Hi everybody, > > I updated CHANGES.txt and fixed documentation li

[jira] [Created] (HDFS-12441) Suppress UnresolvedPathException in namenode log

2017-09-13 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-12441: - Summary: Suppress UnresolvedPathException in namenode log Key: HDFS-12441 URL: https://issues.apache.org/jira/browse/HDFS-12441 Project: Hadoop HDFS Issue Type

[jira] [Resolved] (HDFS-6964) NN fails to fix under replication leading to data loss

2017-08-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-6964. -- Resolution: Duplicate > NN fails to fix under replication leading to data l

[jira] [Resolved] (HDFS-12272) chooseRemoteRack() semantics broken in trunk

2017-08-07 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-12272. --- Resolution: Invalid > chooseRemoteRack() semantics broken in tr

[jira] [Created] (HDFS-12272) chooseRemoteRack() semantics broken in trunk

2017-08-07 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-12272: - Summary: chooseRemoteRack() semantics broken in trunk Key: HDFS-12272 URL: https://issues.apache.org/jira/browse/HDFS-12272 Project: Hadoop HDFS Issue Type: Bug

[jira] [Created] (HDFS-12270) Allow more spreading of replicas during block placement

2017-08-07 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-12270: - Summary: Allow more spreading of replicas during block placement Key: HDFS-12270 URL: https://issues.apache.org/jira/browse/HDFS-12270 Project: Hadoop HDFS Issue

Re: Apache Hadoop 2.8.2 Release Plan

2017-07-21 Thread Kihwal Lee
Thanks for driving the next 2.8 release, Junping. While I was committing a blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 2.8.2. Thanks,Kihwal On Thursday, July 20, 2017, 3:32:16 PM CDT, Junp

[jira] [Resolved] (HDFS-11589) Unable to remove dead node after datanode decommission

2017-06-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-11589. --- Resolution: Duplicate > Unable to remove dead node after datanode decommiss

[jira] [Created] (HDFS-12008) Improve the available-space block placement policy

2017-06-21 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-12008: - Summary: Improve the available-space block placement policy Key: HDFS-12008 URL: https://issues.apache.org/jira/browse/HDFS-12008 Project: Hadoop HDFS Issue Type

[jira] [Created] (HDFS-11945) Internal lease recovery may not be retried for a long time

2017-06-07 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11945: - Summary: Internal lease recovery may not be retried for a long time Key: HDFS-11945 URL: https://issues.apache.org/jira/browse/HDFS-11945 Project: Hadoop HDFS

Re: RollingUpgrade Rollback openfiles issue

2017-05-26 Thread Kihwal Lee
Kihwal From: Vinayakumar B To: Kihwal Lee ; "hdfs-dev@hadoop.apache.org" Sent: Friday, May 26, 2017 5:05 AM Subject: RE: RollingUpgrade Rollback openfiles issue Thanks Kihwal, Found another case, which looks more dangerous. 1. File written and closed. FINALIZ

Re: RollingUpgrade Rollback openfiles issue

2017-05-25 Thread Kihwal Lee
Hi Vinay, If I rephrase the question, Does a RU rollback snapshot provide a consistent snapshot of the distributed  file system? I don't think we aimed it to be a completely consistent snapshot. It is meant to be a safe place to go back with the old version of software.  This is normally used as

[jira] [Resolved] (HDFS-11852) Under-repicated block never completes because of failure in commitBlockSynchronization()

2017-05-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-11852. --- Resolution: Duplicate > Under-repicated block never completes because of failure

[jira] [Resolved] (HDFS-11820) Thread safety in logEdit?

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-11820. --- Resolution: Invalid > Thread safety in logEdit? > - > >

[jira] [Created] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-12 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11817: - Summary: A faulty node can cause a lease leak and NPE on accessing data Key: HDFS-11817 URL: https://issues.apache.org/jira/browse/HDFS-11817 Project: Hadoop HDFS

[jira] [Created] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11742: - Summary: Revert the core changes from HDFS-8818 Key: HDFS-11742 URL: https://issues.apache.org/jira/browse/HDFS-11742 Project: Hadoop HDFS Issue Type: Bug

[jira] [Resolved] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-8818. -- Resolution: Fixed > Allow Balancer to run faster > > >

[jira] [Reopened] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-8818: -- Reopening for revert. git won't revert it cleanly since there have been other changes since the commi

[jira] [Created] (HDFS-11729) Improve NNStorageRetentionManager failure handling.

2017-05-01 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11729: - Summary: Improve NNStorageRetentionManager failure handling. Key: HDFS-11729 URL: https://issues.apache.org/jira/browse/HDFS-11729 Project: Hadoop HDFS Issue Type

[jira] [Created] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-27 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11714: - Summary: Newly added NN storage directory won't get initialized and cause space exhaustion Key: HDFS-11714 URL: https://issues.apache.org/jira/browse/HDFS-11714 Pr

[jira] [Created] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11691: - Summary: Add a proper scheme to the datanode links in NN web UI Key: HDFS-11691 URL: https://issues.apache.org/jira/browse/HDFS-11691 Project: Hadoop HDFS Issue

[jira] [Created] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-03-31 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11609: - Summary: Some blocks can be permanently lost if nodes are decommissioned while dead Key: HDFS-11609 URL: https://issues.apache.org/jira/browse/HDFS-11609 Project: Hadoop

[jira] [Resolved] (HDFS-11207) Unnecessary incompatible change of NNHAStatusHeartbeat.state in DatanodeProtocolProtos

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-11207. --- Resolution: Done > Unnecessary incompatible change of NNHAStatusHeartbeat.state

[jira] [Reopened] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-11094: --- I've reverted this, as HDFS-5079 was reverted. Please rework the patch to reflect that. > S

[jira] [Created] (HDFS-11198) NN UI should link DN web address using hostnames

2016-12-01 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11198: - Summary: NN UI should link DN web address using hostnames Key: HDFS-11198 URL: https://issues.apache.org/jira/browse/HDFS-11198 Project: Hadoop HDFS Issue Type

[jira] [Reopened] (HDFS-10455) Logging the username when deny the setOwner operation

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-10455: --- Reverted the commits. > Logging the username when deny the setOwner operat

[jira] [Created] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11069: - Summary: Tighten the authorization of datanode RPC Key: HDFS-11069 URL: https://issues.apache.org/jira/browse/HDFS-11069 Project: Hadoop HDFS Issue Type: Bug

[jira] [Created] (HDFS-11053) Unnecessary superuser check in versionRequest()

2016-10-25 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11053: - Summary: Unnecessary superuser check in versionRequest() Key: HDFS-11053 URL: https://issues.apache.org/jira/browse/HDFS-11053 Project: Hadoop HDFS Issue Type

[jira] [Reopened] (HDFS-10960) TestDataNodeHotSwapVolumes#testRemoveVolumeBeingWritten fails at disk error verification after volume remove

2016-10-14 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-10960: --- > TestDataNodeHotSwapVolumes#testRemoveVolumeBeingWritten fails at disk error > verification

[jira] [Created] (HDFS-11015) Enforce timeout in balancer

2016-10-14 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11015: - Summary: Enforce timeout in balancer Key: HDFS-11015 URL: https://issues.apache.org/jira/browse/HDFS-11015 Project: Hadoop HDFS Issue Type: Bug

Re: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to 2.5.2

2016-10-14 Thread Kihwal Lee
It looks like the NN is having trouble after reading in the VERSION file, which contains the new layout version. We can continue the discussion in the jira. Kihwal From: Ravi Prakash To: Dinesh Kumar Prabakaran Cc: user ; hdfs-dev Sent: Thursday, October 13, 2016 3:29 PM Subject: R

[jira] [Created] (HDFS-11000) webhdfs PUT does not work if requests are routed to call queue.

2016-10-12 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11000: - Summary: webhdfs PUT does not work if requests are routed to call queue. Key: HDFS-11000 URL: https://issues.apache.org/jira/browse/HDFS-11000 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized

2016-10-11 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-8041. -- Resolution: Won't Fix While this jira makes balancing more aggressive when free space is low, there

Re: Is anyone seeing this during trunk build?

2016-09-30 Thread Kihwal Lee
on did the workaround. I will file a jira to update maven-project-info-reports-plugin. Kihwal From: Arun Suresh To: Kihwal Lee Cc: Ted Yu ; Hdfs-dev ; Hadoop Common Sent: Thursday, September 29, 2016 6:58 PM Subject: Re: Is anyone seeing this during trunk build? It looks l

Re: Is anyone seeing this during trunk build?

2016-09-29 Thread Kihwal Lee
fault locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.11.3", arch: "x86_64", family: "mac" FYI On Wed, Sep 28, 2016 at 3:54 PM, Kihwal Lee wrote: > I just noticed this during a trunk build. I was doing "mvn clean install &g

Is anyone seeing this during trunk build?

2016-09-28 Thread Kihwal Lee
I just noticed this during a trunk build. I was doing "mvn clean install -DskipTests".  The build succeeds. Is anyone seeing this?  I am using openjdk8u102. === [WARNING] Unable to process class org/apache/hadoop/hdfs/StripeReader.class in JarAnalyzer File /home1/kihwal/devel/apache/hadoo

Re: Active NameNode Image Download and Edit Log Roll Stuck in FileChannel Force/Truncate

2016-09-13 Thread Kihwal Lee
Is the system busy with I/O when it happens? Any other I/O activities preceding the event? In your case DistCp could have generated extra edits and also namenode daemon and audit log entries.  Depending on configuration, dirty pages can pile up quite a bit on Linux systems with a large memory an

[jira] [Created] (HDFS-10857) Rolling upgrade can make data unavailable when the cluster has many failed volumes

2016-09-12 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-10857: - Summary: Rolling upgrade can make data unavailable when the cluster has many failed volumes Key: HDFS-10857 URL: https://issues.apache.org/jira/browse/HDFS-10857 Project

[jira] [Created] (HDFS-10841) Fix findbugs warning in branch-2 introduced by path handling improvements

2016-09-06 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-10841: - Summary: Fix findbugs warning in branch-2 introduced by path handling improvements Key: HDFS-10841 URL: https://issues.apache.org/jira/browse/HDFS-10841 Project: Hadoop

[jira] [Reopened] (HDFS-10763) Open files can leak permanently due to inconsistent lease update

2016-08-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-10763: --- Reopening to fix branch-2.7. Apparently we can't simply eliminate open file leak. A deleted file

Re: can someone add me to contributors for hte HDFS jira?

2016-08-16 Thread Kihwal Lee
Done From: Sean Busbey To: "hdfs-dev@hadoop.apache.org" Sent: Tuesday, August 16, 2016 1:05 PM Subject: can someone add me to contributors for hte HDFS jira? Hi! I'm already in the contributor role on the HADOOP jira, could someone add me as one on the HDFS jira? I'd like the abil

[jira] [Created] (HDFS-10763) Open files can leak permanently due to inconsistent lease update

2016-08-15 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-10763: - Summary: Open files can leak permanently due to inconsistent lease update Key: HDFS-10763 URL: https://issues.apache.org/jira/browse/HDFS-10763 Project: Hadoop HDFS

Re: [Release thread] 2.6.5 release activities

2016-08-12 Thread Kihwal Lee
You might want the snapshot bug fix done in HDFS-7056. This bug creates snapshot filediffs even if you never use snapshot. For 2.6, we will have to do it in a separate jira to pick up the fix only. Related to this, HDFS-9696 might be of interest too. Kihwal From: Chris Trezzo To: Karthi

[jira] [Reopened] (HDFS-9696) Garbage snapshot records lingering forever

2016-08-12 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-9696: -- Assignee: (was: Yongjun Zhang) > Garbage snapshot records lingering fore

[jira] [Created] (HDFS-10723) TestRenameWhileOpen tries to restart NN with the same rpc port

2016-08-04 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-10723: - Summary: TestRenameWhileOpen tries to restart NN with the same rpc port Key: HDFS-10723 URL: https://issues.apache.org/jira/browse/HDFS-10723 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-6687) nn.getNamesystem() may return NPE from JspHelper

2016-08-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-6687. -- Resolution: Done > nn.getNamesystem() may return NPE from JspHel

Re: [DISCUSS] Retire BKJM from trunk?

2016-07-28 Thread Kihwal Lee
+1 From: Brahma Reddy Battula To: "Gangumalla, Uma" ; Rakesh Radhakrishnan ; Sijie Guo Cc: "d...@bookkeeper.apache.org" ; Uma gangumalla ; Vinayakumar B ; "hdfs-dev@hadoop.apache.org" ; "u...@hadoop.apache.org" ; "u...@bookkeeper.apache.org" Sent: Thursday, July 28, 2016 4:21 AM

[jira] [Created] (HDFS-10480) Add an admin command to list currently open files

2016-06-02 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-10480: - Summary: Add an admin command to list currently open files Key: HDFS-10480 URL: https://issues.apache.org/jira/browse/HDFS-10480 Project: Hadoop HDFS Issue Type

[jira] [Resolved] (HDFS-10417) Improve error message from checkBlockLocalPathAccess

2016-05-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-10417. --- Resolution: Fixed > Improve error message from checkBlockLocalPathAcc

[jira] [Reopened] (HDFS-10417) Improve error message from checkBlockLocalPathAccess

2016-05-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-10417: --- > Improve error message from checkBlockLocalPathAcc

Re: handlerCount

2016-04-28 Thread Kihwal Lee
For name node, start from NameNodeRpcServer. From: Kun Ren To: hdfs-dev@hadoop.apache.org Sent: Thursday, April 28, 2016 4:51 PM Subject: handlerCount Hi Genius, I have a quick question: I remembered I saw the default value for HandlerCout is 10(The number of Handler threads), but

[jira] [Resolved] (HDFS-9958) BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages.

2016-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-9958. -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.7.3 Committed this to trunk

  1   2   3   4   >