Introduce Read Write Lock to Datanode

2020-02-04 Thread Stephen O'Donnell
I would like to reopen an old topic, which is to introduce a Read Write lock to the datanode. In the current trunk, a RentrentLock is used, so it is always exclusive. However there are many code paths in the DN where an exclusive lock is not necessary and a read lock would suffice. We know the Re

Alternative Decommission Monitor Implementation

2020-02-04 Thread Stephen O'Donnell
Hi All, In https://issues.apache.org/jira/browse/HDFS-14854 we committed a new decommission monitor to trunk, which is disabled by default. The new implementation hopes to be an improvement over the original monitor, but without running it on a real cluster it is hard to know for sure. I would li

Re: Restrict Frequency of BlockReport To Namenode startup and failover

2020-02-07 Thread Stephen O'Donnell
Are you seeing this problem on the 3.x branch, and if so, did the problem exist before you upgraded to 3.x? I am wondering if the situation is better or worse since moving to 3.x. Also, do you believe the issue is driven by the namenode holding its lock for too long while it processes each block r

Changing the default Datanode Volume Choosing policy

2020-04-28 Thread Stephen O'Donnell
Hi, A long time back there was a Jira raised to change the default volume choosing policy from Round Robin to Available Space: https://issues.apache.org/jira/browse/HDFS-8538 At the time there were some objections / concerns about using available space. In the 5 years since then, at Cloudera we

Re: Changing the default Datanode Volume Choosing policy

2020-04-30 Thread Stephen O'Donnell
on in Hadoop 3.4.0 >> > >> > (1) it doesn't break any existing applications I am aware of. >> > (2) No noticeable performance regression in any cases observed. >> > >> > I feel compelled to make a feature the default if it is strictly better. >&

mvsite failing with yetus-dl: gpg unable to import

2020-05-28 Thread Stephen O'Donnell
Hi All, I have a patch (HADOOP-14698) where the mvnsite step is failing with: ERROR: yetus-dl: gpg unable to import /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/patchprocess/KEYS_YETUS I've tried running it a few times in case it was a transient issue, but no luck. Doe

Re: [RESULT][VOTE] Rlease Apache Hadoop-3.3.0

2020-07-15 Thread Stephen O'Donnell
Hi All, Sorry for being a bit late to this, but I wonder if we have a potential blocker to this release. In Cloudera we have recently encountered a serious dataloss issue in HDFS surrounding snapshots. To hit the dataloss issue, you must have HDFS-13101 and HDFS-15012 on the build (which branch-3

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC3)

2020-07-16 Thread Stephen O'Donnell
Hi Gabor, We recently discovered a HDFS data loss issue in any build which uses snapshots containing HDFS-13101 but not including HDFS-15313. Unfortunately 3.1.4 falls into this category: git log origin/branch-3.1.4 | egrep "HDFS-(15313|13101)" HDFS-15012. NN fails to parse Edit logs after a

Re: [DISCUSS] fate of branch-2.9

2020-08-27 Thread Stephen O'Donnell
+1 for EOL of 2.9. On Thu, Aug 27, 2020 at 1:17 PM Sunil Govindan wrote: > +1 > > Thanks > Sunil > > On Thu, Aug 27, 2020 at 3:49 PM Xiaoqiao He wrote: > > > +1 for putting 2.9 release lines to EOL. > > > > Thanks, > > Hexiaoqiao > > > > On Thu, Aug 27, 2020 at 3:14 PM Mingliang Liu wrote: > >

Re: [VOTE] End of Life Hadoop 2.9

2020-09-01 Thread Stephen O'Donnell
+1 Thanks, Stephen. On Tue, Sep 1, 2020 at 7:25 AM Masatake Iwasaki wrote: > +1 > > Thanks, > Masatake Iwasaki > > On 2020/09/01 4:09, Wei-Chiu Chuang wrote: > > Dear fellow Hadoop developers, > > > > Given the overwhelming feedback from the discussion thread > > https://s.apache.org/hadoop2.

Re: Alternative Decommission Monitor Implementation

2020-09-28 Thread Stephen O'Donnell
sion monitor. > > We're planning to adopt this after upgrading to HDFS 3.3.0. > > > > Thanks, > > Akira > > > > On Wed, Feb 5, 2020 at 3:57 AM Wei-Chiu Chuang > wrote: > >> > >> @Akira Ajisaka you said you'd be interested,

Re: Collect feedback for HDFS-15638

2020-10-18 Thread Stephen O'Donnell
I agree with Owen on this - I don't think this is a feature we should add to HDFS. If managing the permissions for Hive tables is becoming a big overhead for you, you should look into something like Sentry. It allows you to manage the permissions of all the files and folders under Hive tables in a

Re: Collect feedback for HDFS-15638

2020-10-19 Thread Stephen O'Donnell
So we have to disable the plugin in NN and map Sentry policies > to HDFS ACL. It works great so far. This is the only major issue we see. > > On Sun, Oct 18, 2020 at 1:19 AM Stephen O'Donnell > wrote: > > > I agree with Owen on this - I don't think this is a feature we sh

Re: Collect feedback for HDFS-15638

2020-10-26 Thread Stephen O'Donnell
hat is another reason > > why I propose this HDFS-15638 to use table-level directory ACLs instead > of > > file-level. > > > > It seems GCP/GCS already has the prefix-based permission granting. Check > > it out here > > https://cloud.google.com/iam/docs/conf

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC2

2020-11-10 Thread Stephen O'Donnell
I compiled a native build from source and started a non-HA, non-HA secure (TLS and Kerberos) and a HA hdfs cluster using docker. On each, I created a few files, looked around the webUI etc. Everything looks fine. The only strange thing, was on the Namenode webUI under the overview section - the "u

Re: RPC logging in ipc Server

2021-04-12 Thread Stephen O'Donnell
I have not tried to do this, but as we (Cloudera) deal with more and more performance related problems, I feel something like this is needed. It is a tricky problem due to the number of requests the NN handles and how performance sensitive it is. At the IPC Server level, we should be able to know

Re: [DISCUSS] Change project style guidelines to allow line length 100

2021-05-20 Thread Stephen O'Donnell
I am +1 on increasing the line length to 100. As for changes to address existing style issues - I think that is more pain than it's worth. It will make backports much harder, and we have quite a few active branches, not to mention those who maintain custom builds. To ease the backport problems th

[jira] [Resolved] (HDFS-15131) FoldedTreeSet appears to degrade over time

2021-06-21 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-15131. -- Resolution: Won't Fix HDFS-13671 is now committed which removes the FoldedTr

[jira] [Created] (HDFS-16144) Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions)

2021-07-26 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-16144: Summary: Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions) Key: HDFS-16144 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Resolved] (HDFS-16132) SnapshotDiff report fails with invalid path assertion with external Attribute provider

2021-07-28 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16132. -- Resolution: Won't Fix Closing this as it is no longer relevant after

[jira] [Resolved] (HDFS-16153) Avoid evaluation of LOG.debug statement in QuorumJournalManager

2021-08-06 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16153. -- Resolution: Fixed > Avoid evaluation of LOG.debug statement in QuorumJourna

[jira] [Created] (HDFS-16190) Combine some tests in TestOzoneManagerHAWithData to reuse mini-Clusters

2021-08-27 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-16190: Summary: Combine some tests in TestOzoneManagerHAWithData to reuse mini-Clusters Key: HDFS-16190 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Created] (HDFS-16252) Correct docs for dfs.http.client.retry.policy.spec

2021-10-04 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-16252: Summary: Correct docs for dfs.http.client.retry.policy.spec Key: HDFS-16252 URL: https://issues.apache.org/jira/browse/HDFS-16252 Project: Hadoop

[jira] [Created] (HDFS-16259) Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger)

2021-10-06 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-16259: Summary: Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) Key: HDFS-16259 URL: https://issues.apach

[jira] [Resolved] (HDFS-16272) Int overflow in computing safe length during EC block recovery

2021-10-18 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16272. -- Resolution: Fixed Committed down the active branches. Thanks for the contrib

[jira] [Resolved] (HDFS-16259) Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger)

2021-11-02 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16259. -- Resolution: Fixed Thanks for the review [~weichiu] and the discussion on

[jira] [Resolved] (HDFS-16286) Debug tool to verify the correctness of erasure coding on file

2021-11-03 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16286. -- Resolution: Fixed Committed down the active 3.x branches. Thanks for the contrib

[jira] [Resolved] (HDFS-16343) Add some debug logs when the dfsUsed are not used during Datanode startup

2021-11-23 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16343. -- Resolution: Fixed > Add some debug logs when the dfsUsed are not used during

[jira] [Resolved] (HDFS-16386) Reduce DataNode load when FsDatasetAsyncDiskService is working

2021-12-20 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16386. -- Resolution: Fixed > Reduce DataNode load when FsDatasetAsyncDiskService is

[jira] [Resolved] (HDFS-16391) Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService

2021-12-21 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16391. -- Resolution: Fixed > Avoid evaluation of LOG.debug stat

[jira] [Resolved] (HDFS-16408) Ensure LeaseRecheckIntervalMs is greater than zero

2022-01-05 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16408. -- Resolution: Fixed > Ensure LeaseRecheckIntervalMs is greater t

[jira] [Resolved] (HDFS-16262) Async refresh of cached locations in DFSInputStream

2022-01-25 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16262. -- Resolution: Fixed I have committed this to trunk and branch-3.3. There are conf

[jira] [Created] (HDFS-16496) Snapshot diff on snapshotable directory fails with not snapshottable error

2022-03-06 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-16496: Summary: Snapshot diff on snapshotable directory fails with not snapshottable error Key: HDFS-16496 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Resolved] (HDFS-16496) Snapshot diff on snapshotable directory fails with not snapshottable error

2022-03-08 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16496. -- Fix Version/s: 3.4.0 Resolution: Fixed > Snapshot diff on snap

[jira] [Created] (HDFS-16530) setReplication debug log creates a new string even if debug is disabled

2022-04-05 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-16530: Summary: setReplication debug log creates a new string even if debug is disabled Key: HDFS-16530 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Created] (HDFS-16531) Avoid setReplication logging an edit record if old replication equals the new value

2022-04-05 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-16531: Summary: Avoid setReplication logging an edit record if old replication equals the new value Key: HDFS-16531 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Resolved] (HDFS-16530) setReplication debug log creates a new string even if debug is disabled

2022-04-06 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16530. -- Resolution: Fixed > setReplication debug log creates a new string even if

[jira] [Resolved] (HDFS-16531) Avoid setReplication logging an edit record if old replication equals the new value

2022-04-20 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16531. -- Resolution: Abandoned Reverted this change down the branches. Sorry for causin

[jira] [Created] (HDFS-16583) DatanodeAdminDefaultMonitor can get stuck in an infinite loop

2022-05-19 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-16583: Summary: DatanodeAdminDefaultMonitor can get stuck in an infinite loop Key: HDFS-16583 URL: https://issues.apache.org/jira/browse/HDFS-16583 Project: H

[jira] [Created] (HDFS-16610) Make fsck read timeout configurable

2022-05-31 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-16610: Summary: Make fsck read timeout configurable Key: HDFS-16610 URL: https://issues.apache.org/jira/browse/HDFS-16610 Project: Hadoop HDFS Issue

[jira] [Reopened] (HDFS-16583) DatanodeAdminDefaultMonitor can get stuck in an infinite loop

2022-05-31 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell reopened HDFS-16583: -- Reopening to add a branch-3.2 PR. > DatanodeAdminDefaultMonitor can get stu

[jira] [Resolved] (HDFS-16610) Make fsck read timeout configurable

2022-06-07 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16610. -- Resolution: Fixed > Make fsck read timeout conf

[jira] [Resolved] (HDFS-14548) Cannot create snapshot when the snapshotCounter reaches MaxSnapshotID

2023-02-27 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-14548. -- Resolution: Duplicate > Cannot create snapshot when the snapshotCounter

[jira] [Created] (HDFS-16942) Send error to datanode if FBR is rejected due to bad lease

2023-03-07 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-16942: Summary: Send error to datanode if FBR is rejected due to bad lease Key: HDFS-16942 URL: https://issues.apache.org/jira/browse/HDFS-16942 Project: Hadoop

[jira] [Resolved] (HDFS-16942) Send error to datanode if FBR is rejected due to bad lease

2023-03-11 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16942. -- Resolution: Fixed > Send error to datanode if FBR is rejected due to b

[jira] [Created] (HDDS-2349) QueryNode does not respect null values for opState or state

2019-10-23 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDDS-2349: --- Summary: QueryNode does not respect null values for opState or state Key: HDDS-2349 URL: https://issues.apache.org/jira/browse/HDDS-2349 Project: H

[jira] [Created] (HDDS-2446) ContainerReplica should contain DatanodeInfo rather than DatanodeDetails

2019-11-08 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDDS-2446: --- Summary: ContainerReplica should contain DatanodeInfo rather than DatanodeDetails Key: HDDS-2446 URL: https://issues.apache.org/jira/browse/HDDS-2446 Pr

[jira] [Created] (HDDS-2447) Allow datanodes to operate with simulated containers

2019-11-08 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDDS-2447: --- Summary: Allow datanodes to operate with simulated containers Key: HDDS-2447 URL: https://issues.apache.org/jira/browse/HDDS-2447 Project: Hadoop Distri

[jira] [Created] (HDDS-2448) Delete container command should used a thread pool

2019-11-08 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDDS-2448: --- Summary: Delete container command should used a thread pool Key: HDDS-2448 URL: https://issues.apache.org/jira/browse/HDDS-2448 Project: Hadoop Distributed

[jira] [Created] (HDDS-2450) Datanode ReplicateContainer thread pool should be configurable

2019-11-08 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDDS-2450: --- Summary: Datanode ReplicateContainer thread pool should be configurable Key: HDDS-2450 URL: https://issues.apache.org/jira/browse/HDDS-2450 Project: H

[jira] [Created] (HDDS-2449) Delete block command should use a thread pool

2019-11-08 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDDS-2449: --- Summary: Delete block command should use a thread pool Key: HDDS-2449 URL: https://issues.apache.org/jira/browse/HDDS-2449 Project: Hadoop Distributed

[jira] [Created] (HDDS-2459) Refactor ReplicationManager to consider maintenance states

2019-11-11 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDDS-2459: --- Summary: Refactor ReplicationManager to consider maintenance states Key: HDDS-2459 URL: https://issues.apache.org/jira/browse/HDDS-2459 Project: H

[jira] [Created] (HDDS-2592) Add Datanode command to allow the datanode to persist its admin state

2019-11-20 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDDS-2592: --- Summary: Add Datanode command to allow the datanode to persist its admin state Key: HDDS-2592 URL: https://issues.apache.org/jira/browse/HDDS-2592 Pr

[jira] [Created] (HDDS-2593) DatanodeAdminMonitor should track unreplicated containers and complete the admin workflow accordingly

2019-11-20 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDDS-2593: --- Summary: DatanodeAdminMonitor should track unreplicated containers and complete the admin workflow accordingly Key: HDDS-2593 URL: https://issues.apache.org/jira/b

[jira] [Created] (HDFS-15131) FoldedTreeSet appears to degrade over time

2020-01-17 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15131: Summary: FoldedTreeSet appears to degrade over time Key: HDFS-15131 URL: https://issues.apache.org/jira/browse/HDFS-15131 Project: Hadoop HDFS

[jira] [Created] (HDFS-15140) Replace FoldedTreeSet in Datanode with SortedSet or TreeMap

2020-01-22 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15140: Summary: Replace FoldedTreeSet in Datanode with SortedSet or TreeMap Key: HDFS-15140 URL: https://issues.apache.org/jira/browse/HDFS-15140 Project: H

[jira] [Created] (HDFS-15150) Introduce read write lock to Datanode

2020-01-30 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15150: Summary: Introduce read write lock to Datanode Key: HDFS-15150 URL: https://issues.apache.org/jira/browse/HDFS-15150 Project: Hadoop HDFS Issue

[jira] [Created] (HDFS-15160) Allow ReplicaMap and callers of ReplicaMap.replicas to use readLock

2020-02-10 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15160: Summary: Allow ReplicaMap and callers of ReplicaMap.replicas to use readLock Key: HDFS-15160 URL: https://issues.apache.org/jira/browse/HDFS-15160 Pr

[jira] [Created] (HDFS-15283) Cache pool MAXTTL is not persisted and restored on cluster restart

2020-04-16 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15283: Summary: Cache pool MAXTTL is not persisted and restored on cluster restart Key: HDFS-15283 URL: https://issues.apache.org/jira/browse/HDFS-15283 Pr

[jira] [Created] (HDFS-15372) Files in snapshots no longer see attribute provider permissions

2020-05-22 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15372: Summary: Files in snapshots no longer see attribute provider permissions Key: HDFS-15372 URL: https://issues.apache.org/jira/browse/HDFS-15372 Pr

[jira] [Resolved] (HDFS-15386) ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories

2020-06-09 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-15386. -- Resolution: Fixed > ReplicaNotFoundException keeps happening in DN after

[jira] [Created] (HDFS-15415) Reduce locking in Datanode DirectoryScanner

2020-06-17 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15415: Summary: Reduce locking in Datanode DirectoryScanner Key: HDFS-15415 URL: https://issues.apache.org/jira/browse/HDFS-15415 Project: Hadoop

[jira] [Reopened] (HDFS-15313) Ensure inodes in active filesytem are not deleted during snapshot delete

2020-07-15 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell reopened HDFS-15313: -- Reopening to see if we can trigger jenkins on the 3.1 and 2.10 patches. > Ensur

[jira] [Created] (HDFS-15540) Directories protected from delete can still be moved to the trash

2020-08-26 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15540: Summary: Directories protected from delete can still be moved to the trash Key: HDFS-15540 URL: https://issues.apache.org/jira/browse/HDFS-15540 Pr

[jira] [Created] (HDFS-15573) Only log warning if considerLoad and considerStorageType are both true

2020-09-11 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15573: Summary: Only log warning if considerLoad and considerStorageType are both true Key: HDFS-15573 URL: https://issues.apache.org/jira/browse/HDFS-15573 Pr

[jira] [Created] (HDFS-15574) Remove unnecessary sort of block list in DirectoryScanner

2020-09-11 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15574: Summary: Remove unnecessary sort of block list in DirectoryScanner Key: HDFS-15574 URL: https://issues.apache.org/jira/browse/HDFS-15574 Project: Hadoop

[jira] [Reopened] (HDFS-15574) Remove unnecessary sort of block list in DirectoryScanner

2020-09-15 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell reopened HDFS-15574: -- Reopening to backport to earlier branches. > Remove unnecessary sort of block

[jira] [Created] (HDFS-15583) Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1

2020-09-17 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15583: Summary: Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1 Key: HDFS-15583 URL: https://issues.apache.org/jira/b

[jira] [Created] (HDFS-15621) Datanode DirectoryScanner uses excessive memory

2020-10-09 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15621: Summary: Datanode DirectoryScanner uses excessive memory Key: HDFS-15621 URL: https://issues.apache.org/jira/browse/HDFS-15621 Project: Hadoop

[jira] [Created] (HDFS-15725) Lease Recovery never completes for a committed block which the DNs never finalize

2020-12-10 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15725: Summary: Lease Recovery never completes for a committed block which the DNs never finalize Key: HDFS-15725 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Resolved] (HDFS-15795) EC: Wrong checksum when reconstruction was failed by exception

2021-02-02 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-15795. -- Resolution: Fixed > EC: Wrong checksum when reconstruction was failed by e

[jira] [Resolved] (HDFS-15818) Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig

2021-02-09 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-15818. -- Resolution: Fixed > Fix TestFsDatasetImpl.testReadLockCanBeDisabled

[jira] [Created] (HDFS-15907) Reduce Memory Overhead of AclFeature by avoiding AtomicInteger

2021-03-19 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15907: Summary: Reduce Memory Overhead of AclFeature by avoiding AtomicInteger Key: HDFS-15907 URL: https://issues.apache.org/jira/browse/HDFS-15907 Project: H

[jira] [Reopened] (HDFS-15160) ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock

2021-03-24 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell reopened HDFS-15160: -- Reopening to backport to branch-3.3 > ReplicaMap, Disk Balancer, Directory Sca

[jira] [Created] (HDFS-15919) BlockPoolManager should log stack trace if unable to get Namenode addresses

2021-03-24 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15919: Summary: BlockPoolManager should log stack trace if unable to get Namenode addresses Key: HDFS-15919 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Created] (HDFS-15937) Reduce memory used during datanode layout upgrade

2021-03-30 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15937: Summary: Reduce memory used during datanode layout upgrade Key: HDFS-15937 URL: https://issues.apache.org/jira/browse/HDFS-15937 Project: Hadoop

[jira] [Created] (HDFS-15942) Increase Quota initialization threads

2021-04-01 Thread Stephen O'Donnell (Jira)
Stephen O'Donnell created HDFS-15942: Summary: Increase Quota initialization threads Key: HDFS-15942 URL: https://issues.apache.org/jira/browse/HDFS-15942 Project: Hadoop HDFS Issue

[jira] [Resolved] (HDFS-15937) Reduce memory used during datanode layout upgrade

2021-04-08 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-15937. -- Resolution: Fixed > Reduce memory used during datanode layout

[jira] [Reopened] (HDFS-15850) Superuser actions should be reported to external enforcers

2021-04-21 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell reopened HDFS-15850: -- > Superuser actions should be reported to external e

[jira] [Resolved] (HDFS-15621) Datanode DirectoryScanner uses excessive memory

2021-04-26 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-15621. -- Resolution: Fixed > Datanode DirectoryScanner uses excessiv

[jira] [Created] (HDFS-12910) Secure Datanode Starter should log the port when it

2017-12-08 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-12910: Summary: Secure Datanode Starter should log the port when it Key: HDFS-12910 URL: https://issues.apache.org/jira/browse/HDFS-12910 Project: Hadoop

[jira] [Created] (HDFS-13170) Port webhdfs unmaskedpermission parameter to HTTPFS

2018-02-19 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-13170: Summary: Port webhdfs unmaskedpermission parameter to HTTPFS Key: HDFS-13170 URL: https://issues.apache.org/jira/browse/HDFS-13170 Project: Hadoop

[jira] [Created] (HDFS-13727) Log full stack trace if DiskBalancer exits with an unhandle exceptiopn

2018-07-11 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-13727: Summary: Log full stack trace if DiskBalancer exits with an unhandle exceptiopn Key: HDFS-13727 URL: https://issues.apache.org/jira/browse/HDFS-13727 Pr

[jira] [Created] (HDFS-13728) Disk Balaner should not fail if volume usage is greater than capacity

2018-07-11 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-13728: Summary: Disk Balaner should not fail if volume usage is greater than capacity Key: HDFS-13728 URL: https://issues.apache.org/jira/browse/HDFS-13728 Pr

[jira] [Created] (HDFS-11215) Log Namenode thread dump on unexpected exits

2016-12-06 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-11215: Summary: Log Namenode thread dump on unexpected exits Key: HDFS-11215 URL: https://issues.apache.org/jira/browse/HDFS-11215 Project: Hadoop

[jira] [Created] (HDFS-14200) Add emptyTrash option to purge trash immediately

2019-01-11 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14200: Summary: Add emptyTrash option to purge trash immediately Key: HDFS-14200 URL: https://issues.apache.org/jira/browse/HDFS-14200 Project: Hadoop

[jira] [Created] (HDFS-14333) Datanode fails to start if any disk has errors during Namenode registration

2019-03-04 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14333: Summary: Datanode fails to start if any disk has errors during Namenode registration Key: HDFS-14333 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Created] (HDFS-14359) Inherited ACL permissions masked when parent directory does not exist (mkdir -p)

2019-03-11 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14359: Summary: Inherited ACL permissions masked when parent directory does not exist (mkdir -p) Key: HDFS-14359 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Created] (HDFS-14389) getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured

2019-03-24 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14389: Summary: getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured Key: HDFS-14389 URL: https://issues.apache.org/jira/browse

[jira] [Created] (HDFS-14459) ClosedChannelException silently ignored in FsVolumeList.addBlockPool()

2019-04-26 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14459: Summary: ClosedChannelException silently ignored in FsVolumeList.addBlockPool() Key: HDFS-14459 URL: https://issues.apache.org/jira/browse/HDFS-14459 Pr

[jira] [Created] (HDDS-1645) Change the version of Pico CLI to the latest 3.x release - 3.9.6

2019-06-05 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDDS-1645: --- Summary: Change the version of Pico CLI to the latest 3.x release - 3.9.6 Key: HDDS-1645 URL: https://issues.apache.org/jira/browse/HDDS-1645 Project: H

[jira] [Created] (HDDS-1660) Use Picocli for Ozone Manager

2019-06-07 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDDS-1660: --- Summary: Use Picocli for Ozone Manager Key: HDDS-1660 URL: https://issues.apache.org/jira/browse/HDDS-1660 Project: Hadoop Distributed Data

[jira] [Created] (HDFS-14560) Allow block replication parameters to be refreshable

2019-06-11 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14560: Summary: Allow block replication parameters to be refreshable Key: HDFS-14560 URL: https://issues.apache.org/jira/browse/HDFS-14560 Project: Hadoop

[jira] [Created] (HDFS-14579) In refreshNodes, avoid performing a DNS lookup while holding the write lock

2019-06-18 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14579: Summary: In refreshNodes, avoid performing a DNS lookup while holding the write lock Key: HDFS-14579 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Created] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

2019-06-28 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14617: Summary: Improve fsimage load time by writing sub-sections to the fsimage index Key: HDFS-14617 URL: https://issues.apache.org/jira/browse/HDFS-14617 Pr

[jira] [Created] (HDFS-14624) When decommissioning a node, log remaining blocks to replicate periodically

2019-07-02 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14624: Summary: When decommissioning a node, log remaining blocks to replicate periodically Key: HDFS-14624 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Created] (HDFS-14626) Decommission all nodes hosting last block of open file succeeds unexpectedly

2019-07-02 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14626: Summary: Decommission all nodes hosting last block of open file succeeds unexpectedly Key: HDFS-14626 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Created] (HDFS-14637) Namenode may not replicate blocks to meet the policy after enabling upgradeDomain

2019-07-08 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14637: Summary: Namenode may not replicate blocks to meet the policy after enabling upgradeDomain Key: HDFS-14637 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Created] (HDFS-14642) processMisReplicatedBlocks does not return correct processed count

2019-07-10 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14642: Summary: processMisReplicatedBlocks does not return correct processed count Key: HDFS-14642 URL: https://issues.apache.org/jira/browse/HDFS-14642 Pr

[jira] [Created] (HDFS-14663) HTTPFS ListStatus_Batch does not return batches as expected

2019-07-23 Thread Stephen O'Donnell (JIRA)
Stephen O'Donnell created HDFS-14663: Summary: HTTPFS ListStatus_Batch does not return batches as expected Key: HDFS-14663 URL: https://issues.apache.org/jira/browse/HDFS-14663 Project: H

  1   2   >