mpl
>
> org.apache.hadoop.ipc.WeightedTimeCostProvider
>
>
>
>
>
> ipc.8020.decay-scheduler.blacklisted.users.enabled
>
> true
>
>
>
>
>
> ipc.8020.decay-scheduler.decay-factor
>
> 0.01
>
>
>
>
I submitted the original 2.8 cost-based FCQ patch (thanks to community
members for porting to other branches). We've been running with it since
early 2019 on all clusters. Multiple clusters run at a baseline of ~30k+
ops/sec with some bursting over 100k ops/sec.
If you are looking at the overall
Daryn Sharp created HDFS-14534:
--
Summary: NetworkTopology's fair lock has poor performance
Key: HDFS-14534
URL: https://issues.apache.org/jira/browse/HDFS-14534
Project: Hadoop HDFS
Issue
Daryn Sharp created HDFS-14533:
--
Summary: Datanode short circuit cache can become blocked
Key: HDFS-14533
URL: https://issues.apache.org/jira/browse/HDFS-14533
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-14532:
--
Summary: Datanode's BlockSender checksum buffer is too big
Key: HDFS-14532
URL: https://issues.apache.org/jira/browse/HDFS-14532
Project: Hadoop HDFS
Issue
Daryn Sharp created HDFS-14531:
--
Summary: Datanode's ScanInfo requires excessive memory
Key: HDFS-14531
URL: https://issues.apache.org/jira/browse/HDFS-14531
Project: Hadoop HDFS
Issue
-1 pending additional info. After a cursory scan, I have serious concerns
regarding the design. This seems like a feature that should have been
purely implemented in hdfs w/o touching the common IPC layer.
The biggest issue in the alignment context. It's purpose appears to be for
allowing handl
; encryption performance) and HADOOP-13836
> > <https://issues.apache.org/jira/browse/HADOOP-13836> (Securing Hadoop
> RPC
> > using SSL). But it looks like both attempts have not been progressing.
> >
> > During the recent Hadoop contributor meetup, Daryn Sharp ment
Daryn Sharp created HDFS-13929:
--
Summary: DFSClient leaks data streamer sockets
Key: HDFS-13929
URL: https://issues.apache.org/jira/browse/HDFS-13929
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-13465:
--
Summary: Overlapping lease recoveries cause NPE in NN
Key: HDFS-13465
URL: https://issues.apache.org/jira/browse/HDFS-13465
Project: Hadoop HDFS
Issue Type: Bug
I’m generally neutral and looked foremost at developer impact. Ie. Will
it be so intertwined with hdfs that each project risks destabilizing the
other? Will developers with no expertise in ozone will be impeded? I
think the answer is currently no. These are the intersections and some
concerns
Daryn Sharp created HDFS-13112:
--
Summary: Token expiration edits may cause log corruption or
deadlock
Key: HDFS-13112
URL: https://issues.apache.org/jira/browse/HDFS-13112
Project: Hadoop HDFS
Daryn Sharp created HDFS-13111:
--
Summary: Close recovery may incorrectly mark blocks corrupt
Key: HDFS-13111
URL: https://issues.apache.org/jira/browse/HDFS-13111
Project: Hadoop HDFS
Issue
[
https://issues.apache.org/jira/browse/HDFS-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp resolved HDFS-13038.
Resolution: Not A Problem
> User with no permission on file is able to run getfacl for that f
Daryn Sharp created HDFS-12914:
--
Summary: Block report leases cause missing blocks until next report
Key: HDFS-12914
URL: https://issues.apache.org/jira/browse/HDFS-12914
Project: Hadoop HDFS
Daryn Sharp created HDFS-12907:
--
Summary: Allow read-only access to reserved raw for non-superusers
Key: HDFS-12907
URL: https://issues.apache.org/jira/browse/HDFS-12907
Project: Hadoop HDFS
Daryn Sharp created HDFS-12747:
--
Summary: Lease monitor may infinitely loop on the same lease
Key: HDFS-12747
URL: https://issues.apache.org/jira/browse/HDFS-12747
Project: Hadoop HDFS
Issue
[
https://issues.apache.org/jira/browse/HDFS-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp resolved HDFS-4493.
---
Resolution: Works for Me
Release Note: This issue was fixed years ago when authenticated url was
Daryn Sharp created HDFS-12705:
--
Summary: WebHdfsFileSystem exceptions should retain the caused by
exception
Key: HDFS-12705
URL: https://issues.apache.org/jira/browse/HDFS-12705
Project: Hadoop HDFS
Daryn Sharp created HDFS-12704:
--
Summary: FBR may corrupt block state
Key: HDFS-12704
URL: https://issues.apache.org/jira/browse/HDFS-12704
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-12703:
--
Summary: Exceptions are fatal to decommissioning monitor
Key: HDFS-12703
URL: https://issues.apache.org/jira/browse/HDFS-12703
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-12658:
--
Summary: Lease renewal causes connection flapping
Key: HDFS-12658
URL: https://issues.apache.org/jira/browse/HDFS-12658
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-12657:
--
Summary: Operations based on inode id must not fallback to the path
Key: HDFS-12657
URL: https://issues.apache.org/jira/browse/HDFS-12657
Project: Hadoop HDFS
Daryn Sharp created HDFS-12648:
--
Summary: DN should provide feedback to NN for throttling commands
Key: HDFS-12648
URL: https://issues.apache.org/jira/browse/HDFS-12648
Project: Hadoop HDFS
Daryn Sharp created HDFS-12647:
--
Summary: DN commands processing should be async
Key: HDFS-12647
URL: https://issues.apache.org/jira/browse/HDFS-12647
Project: Hadoop HDFS
Issue Type: Sub-task
Daryn Sharp created HDFS-12646:
--
Summary: Avoid IO while holding the FsDataset lock
Key: HDFS-12646
URL: https://issues.apache.org/jira/browse/HDFS-12646
Project: Hadoop HDFS
Issue Type: Sub
Daryn Sharp created HDFS-12645:
--
Summary: FSDatasetImpl lock will stall BP service actors and may
cause missing blocks
Key: HDFS-12645
URL: https://issues.apache.org/jira/browse/HDFS-12645
Project
Daryn Sharp created HDFS-12639:
--
Summary: BPOfferService lock may stall all service actors
Key: HDFS-12639
URL: https://issues.apache.org/jira/browse/HDFS-12639
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-12184:
--
Summary: Avoid redundant ancestor metadata traversals for
listStatus child entries.
Key: HDFS-12184
URL: https://issues.apache.org/jira/browse/HDFS-12184
Project: Hadoop
Daryn Sharp created HDFS-12174:
--
Summary: DN commitBlockSynchronization needs delivery guarantee
Key: HDFS-12174
URL: https://issues.apache.org/jira/browse/HDFS-12174
Project: Hadoop HDFS
Issue
Daryn Sharp created HDFS-12173:
--
Summary: MiniDFSCluster cannot reliably use NameNode#stop
Key: HDFS-12173
URL: https://issues.apache.org/jira/browse/HDFS-12173
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-12172:
--
Summary: Reduce EZ lookup overhead
Key: HDFS-12172
URL: https://issues.apache.org/jira/browse/HDFS-12172
Project: Hadoop HDFS
Issue Type: Improvement
Daryn Sharp created HDFS-12171:
--
Summary: Reduce IIP object allocations for inode lookup
Key: HDFS-12171
URL: https://issues.apache.org/jira/browse/HDFS-12171
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-12143:
--
Summary: Improve performance of getting and removing inode features
Key: HDFS-12143
URL: https://issues.apache.org/jira/browse/HDFS-12143
Project: Hadoop HDFS
Daryn Sharp created HDFS-12142:
--
Summary: Files may be closed before streamer is done
Key: HDFS-12142
URL: https://issues.apache.org/jira/browse/HDFS-12142
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-12140:
--
Summary: Remove BPOfferService lock contention to get block pool id
Key: HDFS-12140
URL: https://issues.apache.org/jira/browse/HDFS-12140
Project: Hadoop HDFS
Daryn Sharp created HDFS-12137:
--
Summary: DN dataset lock should be fair
Key: HDFS-12137
URL: https://issues.apache.org/jira/browse/HDFS-12137
Project: Hadoop HDFS
Issue Type: Improvement
Daryn Sharp created HDFS-12136:
--
Summary: BlockSender performance regression due to volume scanner
edge case
Key: HDFS-12136
URL: https://issues.apache.org/jira/browse/HDFS-12136
Project: Hadoop HDFS
Daryn Sharp created HDFS-12070:
--
Summary: Failed block recovery leaves files open indefinitely and
at risk for data loss
Key: HDFS-12070
URL: https://issues.apache.org/jira/browse/HDFS-12070
Project
Daryn Sharp created HDFS-12049:
--
Summary: Recommissioning live nodes stalls the NN
Key: HDFS-12049
URL: https://issues.apache.org/jira/browse/HDFS-12049
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-11648:
--
Summary: Lazy construct the IIP pathname
Key: HDFS-11648
URL: https://issues.apache.org/jira/browse/HDFS-11648
Project: Hadoop HDFS
Issue Type: Improvement
Daryn Sharp created HDFS-11641:
--
Summary: Reduce cost of audit logging by using FileStatus instead
of HdfsFileStatus
Key: HDFS-11641
URL: https://issues.apache.org/jira/browse/HDFS-11641
Project: Hadoop
Daryn Sharp created HDFS-11379:
--
Summary: DFSInputStream may infinite loop requesting block
locations
Key: HDFS-11379
URL: https://issues.apache.org/jira/browse/HDFS-11379
Project: Hadoop HDFS
Daryn Sharp created HDFS-11310:
--
Summary: Reduce the performance impact of the balancer (trunk port)
Key: HDFS-11310
URL: https://issues.apache.org/jira/browse/HDFS-11310
Project: Hadoop HDFS
Daryn Sharp created HDFS-10997:
--
Summary: Reduce number of path resolving methods
Key: HDFS-10997
URL: https://issues.apache.org/jira/browse/HDFS-10997
Project: Hadoop HDFS
Issue Type: Sub-task
Daryn Sharp created HDFS-10980:
--
Summary: Optimize check for existence of parent directory
Key: HDFS-10980
URL: https://issues.apache.org/jira/browse/HDFS-10980
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-10979:
--
Summary: Pass IIP for FSDirDeleteOp methods
Key: HDFS-10979
URL: https://issues.apache.org/jira/browse/HDFS-10979
Project: Hadoop HDFS
Issue Type: Sub-task
Daryn Sharp created HDFS-10956:
--
Summary: Remove rename/delete performance penalty when not using
snapshots
Key: HDFS-10956
URL: https://issues.apache.org/jira/browse/HDFS-10956
Project: Hadoop HDFS
Daryn Sharp created HDFS-10955:
--
Summary: Pass IIP for FSDirAttr methods
Key: HDFS-10955
URL: https://issues.apache.org/jira/browse/HDFS-10955
Project: Hadoop HDFS
Issue Type: Sub-task
Daryn Sharp created HDFS-10940:
--
Summary: Reduce performance penalty of block caching when not used
Key: HDFS-10940
URL: https://issues.apache.org/jira/browse/HDFS-10940
Project: Hadoop HDFS
Daryn Sharp created HDFS-10939:
--
Summary: Reduce performance penalty of encryption zones
Key: HDFS-10939
URL: https://issues.apache.org/jira/browse/HDFS-10939
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-10851:
--
Summary: FSDirStatAndListingOp: stop passing path as string
Key: HDFS-10851
URL: https://issues.apache.org/jira/browse/HDFS-10851
Project: Hadoop HDFS
Issue
Daryn Sharp created HDFS-10850:
--
Summary: getEZForPath should NOT throw FNF
Key: HDFS-10850
URL: https://issues.apache.org/jira/browse/HDFS-10850
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-10789:
--
Summary: Route webhdfs through the RPC call queue
Key: HDFS-10789
URL: https://issues.apache.org/jira/browse/HDFS-10789
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-10779:
--
Summary: Rename does not need to re-solve destination
Key: HDFS-10779
URL: https://issues.apache.org/jira/browse/HDFS-10779
Project: Hadoop HDFS
Issue Type: Sub
Daryn Sharp created HDFS-10772:
--
Summary: Reduce byte/string conversions for get listing
Key: HDFS-10772
URL: https://issues.apache.org/jira/browse/HDFS-10772
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-10768:
--
Summary: Optimize mkdir ops
Key: HDFS-10768
URL: https://issues.apache.org/jira/browse/HDFS-10768
Project: Hadoop HDFS
Issue Type: Sub-task
Components
Daryn Sharp created HDFS-10762:
--
Summary: Pass IIP for file status related methods
Key: HDFS-10762
URL: https://issues.apache.org/jira/browse/HDFS-10762
Project: Hadoop HDFS
Issue Type: Sub
Daryn Sharp created HDFS-10745:
--
Summary: Directly resolve paths into INodesInPath
Key: HDFS-10745
URL: https://issues.apache.org/jira/browse/HDFS-10745
Project: Hadoop HDFS
Issue Type: Sub
Daryn Sharp created HDFS-10744:
--
Summary: Internally optimize path component resolution
Key: HDFS-10744
URL: https://issues.apache.org/jira/browse/HDFS-10744
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-10743:
--
Summary: MiniDFSCluster test runtimes can be drastically reduce
Key: HDFS-10743
URL: https://issues.apache.org/jira/browse/HDFS-10743
Project: Hadoop HDFS
Issue
Daryn Sharp created HDFS-10722:
--
Summary: Fix race condition in
TestEditLog#testBatchedSyncWithClosedLogs
Key: HDFS-10722
URL: https://issues.apache.org/jira/browse/HDFS-10722
Project: Hadoop HDFS
[
https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp reopened HDFS-10301:
> BlockReport retransmissions may lead to storages falsely being declared
> zombie if storage
Daryn Sharp created HDFS-10711:
--
Summary: Optimize FSPermissionChecker group membership check
Key: HDFS-10711
URL: https://issues.apache.org/jira/browse/HDFS-10711
Project: Hadoop HDFS
Issue
Daryn Sharp created HDFS-10674:
--
Summary: Optimize creating a full path from an inode
Key: HDFS-10674
URL: https://issues.apache.org/jira/browse/HDFS-10674
Project: Hadoop HDFS
Issue Type: Sub
Daryn Sharp created HDFS-10673:
--
Summary: Optimize FSPermissionChecker's internal path usage
Key: HDFS-10673
URL: https://issues.apache.org/jira/browse/HDFS-10673
Project: Hadoop HDFS
Daryn Sharp created HDFS-10662:
--
Summary: Optimize UTF8 string/byte conversions
Key: HDFS-10662
URL: https://issues.apache.org/jira/browse/HDFS-10662
Project: Hadoop HDFS
Issue Type: Sub-task
Daryn Sharp created HDFS-10656:
--
Summary: Optimize conversion of byte arrays back to path string
Key: HDFS-10656
URL: https://issues.apache.org/jira/browse/HDFS-10656
Project: Hadoop HDFS
Issue
Daryn Sharp created HDFS-10655:
--
Summary: Fix path related byte array conversion bugs
Key: HDFS-10655
URL: https://issues.apache.org/jira/browse/HDFS-10655
Project: Hadoop HDFS
Issue Type: Sub
Daryn Sharp created HDFS-10653:
--
Summary: Optimize conversion from path string to components
Key: HDFS-10653
URL: https://issues.apache.org/jira/browse/HDFS-10653
Project: Hadoop HDFS
Issue
Daryn Sharp created HDFS-10619:
--
Summary: Cache path in InodesInPath
Key: HDFS-10619
URL: https://issues.apache.org/jira/browse/HDFS-10619
Project: Hadoop HDFS
Issue Type: Sub-task
Daryn Sharp created HDFS-10616:
--
Summary: Improve performance of path handling
Key: HDFS-10616
URL: https://issues.apache.org/jira/browse/HDFS-10616
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-10343:
--
Summary: BlockManager#createLocatedBlocks may return blocks on
failed storages
Key: HDFS-10343
URL: https://issues.apache.org/jira/browse/HDFS-10343
Project: Hadoop HDFS
Daryn Sharp created HDFS-10342:
--
Summary: BlockManager#createLocatedBlocks should not check corrupt
replicas if none are corrupt
Key: HDFS-10342
URL: https://issues.apache.org/jira/browse/HDFS-10342
Daryn Sharp created HDFS-10326:
--
Summary: Disable setting tcp socket send/receive buffers for write
pipelines
Key: HDFS-10326
URL: https://issues.apache.org/jira/browse/HDFS-10326
Project: Hadoop HDFS
Daryn Sharp created HDFS-9566:
-
Summary: Remove expensive getStorages method
Key: HDFS-9566
URL: https://issues.apache.org/jira/browse/HDFS-9566
Project: Hadoop HDFS
Issue Type: Improvement
Daryn Sharp created HDFS-9557:
-
Summary: Reduce object allocation in PB conversion
Key: HDFS-9557
URL: https://issues.apache.org/jira/browse/HDFS-9557
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-9287:
-
Summary: Block placement completely fails if too many nodes are
decommissioning
Key: HDFS-9287
URL: https://issues.apache.org/jira/browse/HDFS-9287
Project: Hadoop HDFS
Daryn Sharp created HDFS-9258:
-
Summary: NN should indicate which nodes are stale
Key: HDFS-9258
URL: https://issues.apache.org/jira/browse/HDFS-9258
Project: Hadoop HDFS
Issue Type: Improvement
Daryn Sharp created HDFS-9198:
-
Summary: Coalesce IBR processing in the NN
Key: HDFS-9198
URL: https://issues.apache.org/jira/browse/HDFS-9198
Project: Hadoop HDFS
Issue Type: Improvement
Daryn Sharp created HDFS-9107:
-
Summary: Prevent NN's unrecoverable death spiral after full GC
Key: HDFS-9107
URL: https://issues.apache.org/jira/browse/HDFS-9107
Project: Hadoop HDFS
Issue
Daryn Sharp created HDFS-9017:
-
Summary: UI shows wrong last contact for dead nodes
Key: HDFS-9017
URL: https://issues.apache.org/jira/browse/HDFS-9017
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-8776:
-
Summary: Decom manager should not be active on standby
Key: HDFS-8776
URL: https://issues.apache.org/jira/browse/HDFS-8776
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-8675:
-
Summary: IBRs from dead DNs go into infinite loop
Key: HDFS-8675
URL: https://issues.apache.org/jira/browse/HDFS-8675
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-8674:
-
Summary: Improve performance of postponed block scans
Key: HDFS-8674
URL: https://issues.apache.org/jira/browse/HDFS-8674
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-8616:
-
Summary: Cherry pick HDFS-6495 for excess block leak
Key: HDFS-8616
URL: https://issues.apache.org/jira/browse/HDFS-8616
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-8498:
-
Summary: Blocks can be committed with wrong size
Key: HDFS-8498
URL: https://issues.apache.org/jira/browse/HDFS-8498
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-8492:
-
Summary: DN should notify NN when client requests a missing block
Key: HDFS-8492
URL: https://issues.apache.org/jira/browse/HDFS-8492
Project: Hadoop HDFS
Issue
Daryn Sharp created HDFS-8491:
-
Summary: DN shutdown race conditions with open xceivers
Key: HDFS-8491
URL: https://issues.apache.org/jira/browse/HDFS-8491
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-8486:
-
Summary: DN startup may cause severe data loss
Key: HDFS-8486
URL: https://issues.apache.org/jira/browse/HDFS-8486
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-8402:
-
Summary: Fsck exit codes are not reliable
Key: HDFS-8402
URL: https://issues.apache.org/jira/browse/HDFS-8402
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-8133:
-
Summary: Improve readability of deleted block check
Key: HDFS-8133
URL: https://issues.apache.org/jira/browse/HDFS-8133
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-7990:
-
Summary: IBR delete ack should not be delayed
Key: HDFS-7990
URL: https://issues.apache.org/jira/browse/HDFS-7990
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-7967:
-
Summary: Reduce the performance impact of the balancer
Key: HDFS-7967
URL: https://issues.apache.org/jira/browse/HDFS-7967
Project: Hadoop HDFS
Issue Type: Sub
Daryn Sharp created HDFS-7964:
-
Summary: Add support for async edit logging
Key: HDFS-7964
URL: https://issues.apache.org/jira/browse/HDFS-7964
Project: Hadoop HDFS
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/HDFS-7586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp resolved HDFS-7586.
---
Resolution: Not a Problem
> HFTP does not work when namenode bind on wildc
Daryn Sharp created HDFS-7607:
-
Summary: Use random rack-local node for webhdfs opens to avoid OOM
on DNs
Key: HDFS-7607
URL: https://issues.apache.org/jira/browse/HDFS-7607
Project: Hadoop HDFS
Daryn Sharp created HDFS-7597:
-
Summary: Clients seeking over webhdfs may crash the NN
Key: HDFS-7597
URL: https://issues.apache.org/jira/browse/HDFS-7597
Project: Hadoop HDFS
Issue Type
Daryn Sharp created HDFS-7457:
-
Summary: DatanodeID generates excessive garbage
Key: HDFS-7457
URL: https://issues.apache.org/jira/browse/HDFS-7457
Project: Hadoop HDFS
Issue Type: Bug
Daryn Sharp created HDFS-7435:
-
Summary: PB encoding of block reports is very inefficient
Key: HDFS-7435
URL: https://issues.apache.org/jira/browse/HDFS-7435
Project: Hadoop HDFS
Issue Type
1 - 100 of 256 matches
Mail list logo
