[jira] [Created] (HDFS-13092) Reduce verbosity for ThrottledAsyncChecker.java:schedule

[Mukul Kumar Singh (JIRA)]

Mukul Kumar Singh created HDFS-13092:


 Summary: Reduce verbosity for ThrottledAsyncChecker.java:schedule
 Key: HDFS-13092
 URL: https://issues.apache.org/jira/browse/HDFS-13092
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh


ThrottledAsyncChecker.java:schedule prints a log message every time a disk 
check is scheduled. However if the previous check was triggered lesser than the 
frequency at "minMsBetweenChecks" then the task is not scheduled. This jira 
will reduce the log verbosity by printing the message only when the task will 
be scheduled.

{code}
2018-01-29 00:51:44,467 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/2/hadoop/hdfs/data/current
2018-01-29 00:51:44,470 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/2/hadoop/hdfs/data/current
2018-01-29 00:51:44,477 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/4/hadoop/hdfs/data/current
2018-01-29 00:51:44,480 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/4/hadoop/hdfs/data/current
2018-01-29 00:51:44,486 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/11/hadoop/hdfs/data/current
2018-01-29 00:51:44,501 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/13/hadoop/hdfs/data/current
2018-01-29 00:51:44,507 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/11/hadoop/hdfs/data/current
2018-01-29 00:51:44,533 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/2/hadoop/hdfs/data/current
2018-01-29 00:51:44,536 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/12/hadoop/hdfs/data/current
2018-01-29 00:51:44,543 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/10/hadoop/hdfs/data/current
2018-01-29 00:51:44,544 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/2/hadoop/hdfs/data/current
2018-01-29 00:51:44,548 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/3/hadoop/hdfs/data/current
2018-01-29 00:51:44,549 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/5/hadoop/hdfs/data/current
2018-01-29 00:51:44,550 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/6/hadoop/hdfs/data/current
2018-01-29 00:51:44,551 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/2/hadoop/hdfs/data/current
2018-01-29 00:51:44,552 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/10/hadoop/hdfs/data/current
2018-01-29 00:51:44,552 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/8/hadoop/hdfs/data/current
2018-01-29 00:51:44,552 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/12/hadoop/hdfs/data/current
2018-01-29 00:51:44,554 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/9/hadoop/hdfs/data/current
2018-01-29 00:51:44,555 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/8/hadoop/hdfs/data/current
2018-01-29 00:51:44,555 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/14/hadoop/hdfs/data/current
2018-01-29 00:51:44,560 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/12/hadoop/hdfs/data/current
2018-01-29 00:51:44,560 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/12/hadoop/hdfs/data/current
2018-01-29 00:51:44,564 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/5/hadoop/hdfs/data/current
2018-01-29 00:51:44,564 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/12/hadoop/hdfs/data/current
2018-01-29 00:51:44,570 INFO  checker.ThrottledAsyncChecker 
(ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for 
/grid/5/hado

Re: Hadoop 3.1.0 release discussion

[Daniel Templeton]

I added my comments on that JIRA.  Looks like YARN-7292 is marked as a 
blocker for 3.1, and I would tend to agree with that.  Let's see what we 
can to do get profiles nailed down so that 3.1 can go forward.


Daniel

On 1/18/18 10:25 AM, Wangda Tan wrote:

Thanks Daniel,

We need to make a decision about this: 
https://issues.apache.org/jira/browse/YARN-7292, I believe this is the 
JIRA you mentioned correct? Please let me know if there's anything 
else. And let's move the discussion on JIRA.


The good news is that resource profile is merged to trunk already so 
we can finish that before code freeze date (Feb 08).


+ Sunil as well.

Thanks,
Wangda



On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton > wrote:


What's the status on resource profiles?  I believe there are still
a couple of open JIRAs to rethink some of the design choices.

Daniel


On 1/17/18 11:33 AM, Wangda Tan wrote:

Hi All,

Since we're fast approaching previously proposed feature
freeze date (Jan
30, about 13 days from today). If you've any features which
live in a
branch and targeted to 3.1.0, please reply this email thread.
Ideally, we
should finish branch merging before feature freeze date.

Here's an updated 3.1.0 feature status:

1. Merged & Completed features:
* (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
* (Wangda) YARN-6223: GPU support on YARN. Features in trunk
and works
end-to-end.
* (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
services.
* (Steve Loughran): HADOOP-13786: S3Guard committer for
zero-rename commits.
* (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
of Leaf
Queues While Doing Queue Mapping.
* (Chris Douglas) HDFS-9806: HDFS Tiered Storage.

2. Features close to finish:
* (Zhankun) YARN-5983: FPGA support. Majority implementations
completed and
merged to trunk. Except for UI/documentation.
* (Uma) HDFS-10285: HDFS SPS. Majority implementations are
done, some
discussions going on about implementation.
* (Arun Suresh / Kostas / Wangda). YARN-6592: New
SchedulingRequest and
anti-affinity support. Close to finish, on track to be merged
before Jan 30.

3. Tentative features:
* (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
containers. Only one pending patch. Plan to finish before Jan 7th.
* (Haibo Chen). YARN-1011: Resource overcommitment. Looks
challenging to be
done before Jan 2018.
* (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
Looks
challenging to be done before Jan 2018.
* (Varun V) YARN-5673: container-executor write. Given
security refactoring
of c-e (YARN-6623) is already landed, IMHO other stuff may be
moved to 3.2.

Thanks,
Wangda




On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan
mailto:wheele...@gmail.com>> wrote:

Hi all,

Congratulations on the 3.0.0-GA release!

As we discussed in the previous email thread [1], I'd like
to restart
3.1.0 release plans.

a) Quick summary:
a.1 Release status
We started 3.1 release discussion on Sep 6, 2017 [1]. As
of today,
there’re 232 patches loaded on 3.1.0 alone [2], besides 6
open blockers and
22 open critical issues.

a.2 Release date update
Considering delays of 3.0-GA release by month-and-a-half,
I propose to
move the dates as follows
  - feature freeze date from Dec 15, 2017, to Jan 30, 2018
- last date for
any branches to get merged too;
  - code freeze (blockers & critical only) date to Feb 08,
2018;
  - release voting start by Feb 18, 2018, leaving time for
at least two RCx
  - release date from Jan 15, 2018, to Feb 28, 2018;

Unlike before, I added an additional milestone for
release-vote-start so
that we can account for voting time-period also.

This overall is still 5 1/2 months of release-timeline
unlike the faster
cadence we hoped for, but this, in my opinion, is the
best-updated timeline
given the delays of the final release of 3.0-GA.

b) Individual feature status:
I spoke to several feature owners and checked the status
of un-finished
features, following are status of features planned to 3.1.0:

b.1 Merged & Completed features:
* (Sunil) YARN-5881: Support absolute value in
Capac

[jira] [Created] (HDFS-13093) Quota set don't compute usage of unspecified storage policy content

[liaoyuxiangqin (JIRA)]

liaoyuxiangqin created HDFS-13093:
-

 Summary: Quota set don't compute usage of unspecified storage 
policy content
 Key: HDFS-13093
 URL: https://issues.apache.org/jira/browse/HDFS-13093
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 3.1.0
 Environment: hdfs: hadoop-3.1.0-SNAPSHOT

node:1 namenode, 9 datanodes
Reporter: liaoyuxiangqin


test as following steps:
 1. hdfs dfs -mkdir /hot
 2. hdfs dfs -put 1G.img /hot/file1
 3. hdfs dfsadmin -setSpaceQuota 6442450944 -storageType DISK /hot
 4. hdfs storagepolicies -setStoragePolicy -path /hot -policy HOT
 5. hdfs dfs -count -q -h -v -t DISK /hot
{code:java}
SSD_QUOTA REM_SSD_QUOTA DISK_QUOTA REM_DISK_QUOTA ARCHIVE_QUOTA 
REM_ARCHIVE_QUOTA PROVIDED_QUOTA REM_PROVIDED_QUOTA PATHNAME
 none inf 6 G 6 G none inf none inf /hot{code}
In step5 i speculation the remaining quota is 3G(quota - 1G*3 replicas )，but 6G 
actually.
 if i change the turn of step3 and step4, then the remaining quota equal to 
what I think 3G.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Hadoop 3.1.0 release discussion

[Wangda Tan]

Hi Daniel,

I agree, let's move the discussion to YARN-7292 and get it resolved before
3.1.0

Thanks,
Wangda

On Wed, Jan 31, 2018 at 4:35 PM, Daniel Templeton 
wrote:

> I added my comments on that JIRA.  Looks like YARN-7292 is marked as a
> blocker for 3.1, and I would tend to agree with that.  Let's see what we
> can to do get profiles nailed down so that 3.1 can go forward.
>
> Daniel
>
>
> On 1/18/18 10:25 AM, Wangda Tan wrote:
>
> Thanks Daniel,
>
> We need to make a decision about this: https://issues.apache.
> org/jira/browse/YARN-7292, I believe this is the JIRA you mentioned
> correct? Please let me know if there's anything else. And let's move the
> discussion on JIRA.
>
> The good news is that resource profile is merged to trunk already so we
> can finish that before code freeze date (Feb 08).
>
> + Sunil as well.
>
> Thanks,
> Wangda
>
>
>
> On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton 
> wrote:
>
>> What's the status on resource profiles?  I believe there are still a
>> couple of open JIRAs to rethink some of the design choices.
>>
>> Daniel
>>
>>
>> On 1/17/18 11:33 AM, Wangda Tan wrote:
>>
>>> Hi All,
>>>
>>> Since we're fast approaching previously proposed feature freeze date (Jan
>>> 30, about 13 days from today). If you've any features which live in a
>>> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
>>> should finish branch merging before feature freeze date.
>>>
>>> Here's an updated 3.1.0 feature status:
>>>
>>> 1. Merged & Completed features:
>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>> end-to-end.
>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>> commits.
>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>> Queues While Doing Queue Mapping.
>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>>>
>>> 2. Features close to finish:
>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>> and
>>> merged to trunk. Except for UI/documentation.
>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>> discussions going on about implementation.
>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>> anti-affinity support. Close to finish, on track to be merged before Jan
>>> 30.
>>>
>>> 3. Tentative features:
>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>> be
>>> done before Jan 2018.
>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>> challenging to be done before Jan 2018.
>>> * (Varun V) YARN-5673: container-executor write. Given security
>>> refactoring
>>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
>>> 3.2.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>>
>>>
>>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan  wrote:
>>>
>>> Hi all,

 Congratulations on the 3.0.0-GA release!

 As we discussed in the previous email thread [1], I'd like to restart
 3.1.0 release plans.

 a) Quick summary:
 a.1 Release status
 We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
 there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers
 and
 22 open critical issues.

 a.2 Release date update
 Considering delays of 3.0-GA release by month-and-a-half, I propose to
 move the dates as follows
   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
 for
 any branches to get merged too;
   - code freeze (blockers & critical only) date to Feb 08, 2018;
   - release voting start by Feb 18, 2018, leaving time for at least two
 RCx
   - release date from Jan 15, 2018, to Feb 28, 2018;

 Unlike before, I added an additional milestone for release-vote-start so
 that we can account for voting time-period also.

 This overall is still 5 1/2 months of release-timeline unlike the faster
 cadence we hoped for, but this, in my opinion, is the best-updated
 timeline
 given the delays of the final release of 3.0-GA.

 b) Individual feature status:
 I spoke to several feature owners and checked the status of un-finished
 features, following are status of features planned to 3.1.0:

 b.1 Merged & Completed features:
 * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
 * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
 end-to-end.
 * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
 * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
 commits.
 * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creati

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/121/

[Jan 28, 2018 9:24:49 PM] (jianhe) YARN-7765. Fixed an issue that kerberos tgt 
not found when NM posting
[Jan 29, 2018 6:30:14 AM] (xiao) HDFS-12974. Exception message is not printed 
when creating an encryption
[Jan 29, 2018 3:13:04 PM] (aajisaka) YARN-7698. A misleading variable's name in
[Jan 29, 2018 11:52:04 PM] (kihwal) HDFS-12574. Add CryptoInputStream to 
WebHdfsFileSystem read call.
[Jan 30, 2018 8:25:02 AM] (sammi.chen) HADOOP-15027. AliyunOSS: Support 
multi-thread pre-read to improve




-1 overall


The following subsystems voted -1:
asflicense unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-hdfs:47 
   bkjournal:8 
   hadoop-yarn-server-resourcemanager:2 
   hadoop-yarn-client:4 
   hadoop-yarn-applications-distributedshell:1 
   hadoop-mapreduce-client-jobclient:2 
   hadoop-distcp:3 
   hadoop-extras:1 

Failed junit tests :

   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog 
   hadoop.hdfs.server.namenode.TestNameNodeRpcServer 
   hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy 
   
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
 
   hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.tools.TestIntegration 
   hadoop.tools.TestDistCpViewFs 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend 
   org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement 
   org.apache.hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages 
   org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade 
   org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints 
   org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics 
   org.apache.hadoop.hdfs.server.namenode.TestAddBlockRetry 
   org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM 
   org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport 
   org.apache.hadoop.hdfs.server.namenode.TestINodeFile 
   org.apache.hadoop.hdfs.server.namenode.TestEditLog 
   org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer 
   org.apache.hadoop.hdfs.server.namenode.TestStreamFile 
   org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot 
   org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot 
   org.apache.hadoop.hdfs.server.namenode.ha.TestNNHealthCheck 
   org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits 
   
org.apache.hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled 
   org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing 
   org.apache.hadoop.hdfs.server.namenode.TestNameNodeReconfigure 
   org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits 
   org.apache.hadoop.hdfs.server.namenode.TestFileTruncate 
   org.apache.hadoop.hdfs.server.namenode.TestNameNodeStatusMXBean 
   org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures 
   org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite 
   org.apache.hadoop.hdfs.server.namenode.TestMetaSave 
   org.apache.hadoop.hdfs.server.namenode.TestCheckpoint 
   org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs 
   org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes 
   org.apache.hadoop.hdfs.server.namenode.TestFSDirectory 
   org.apache.hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA 
   org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication 
   org.apache.hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA 
   org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions 
   org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover 
   org.apache.hadoop.hdfs.server.namenode.TestDeleteRace 
   org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby 
   org.apache.hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA 
   org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover 
   org.apache.hadoop.hdfs.server.namenode.ha.TestQuotasWithHA 
   org.apache.hadoop.hdfs.server.namenode.TestAuditLogge

Re: [VOTE] Merge YARN-6592 feature branch to trunk

[Arun Suresh]

Here's my +1 as well.
With 6 +1s and no -1s, the VOTE passes.

Thanks to everyone who participated.
Will be merging this shortly.

Cheers
-Arun

On Jan 26, 2018 5:46 PM, "Weiwei Yang"  wrote:

> +1
> We are also looking forward to this : )
>
> --
> Weiwei
>
> On 27 Jan 2018, 9:16 AM +0800, Wangda Tan , wrote:
> +1
>
> On Sat, Jan 27, 2018 at 2:05 AM, Chris Douglas  > wrote:
> +1 Looking forward to this. -C
>
> On Fri, Jan 26, 2018 at 7:28 AM, Arun Suresh  asur...@apache.org>> wrote:
> > Hello yarn-dev@
> >
> > Based on the positive feedback from the DISCUSS thread [1], I'd like to
> > start a formal vote to merge YARN-6592 [2] to trunk. The vote will run
> for 5
> > days, and will end Jan 31 7:30AM PDT.
> >
> > This feature adds support for placing containers in YARN using rich
> > placement constraints. For example, this can be used by applications to
> > co-locate containers on a node or rack (affinity constraint), spread
> > containers across nodes or racks (anti-affinity constraint), or even
> specify
> > the maximum number of containers on a node/rack (cardinality constraint).
> >
> > We have integrated this feature into the Distributed-Shell application
> for
> > feature testing. We have performed end-to-end testing on moderately-sized
> > clusters to verify that constraints work fine. Performance tests have
> been
> > done via both SLS tests and Capacity Scheduler performance unit tests,
> and
> > no regression was found. We have opened a JIRA to track Jenkins
> acceptance
> > of the aggregated patch [3]. Documentation is in the process of being
> > completed [4]. You can also check our design document for more details
> [5].
> >
> > Config flags are needed to enable this feature and it should not have any
> > effect on YARN when turned off. Once merged, we plan to work on further
> > improvements, which can be tracked in the umbrella YARN-7812 [6].
> >
> > Kindly do take a look at the branch and raise issues/concerns that need
> to
> > be addressed before the merge.
> >
> > Many thanks to Konstantinos, Wangda, Panagiotis, Weiwei, and Sunil for
> their
> > contributions to this effort, as well as Subru, Chris, Carlo, and Vinod
> for
> > their inputs and discussions.
> >
> >
> > Cheers
> > -Arun
> >
> > [1]
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201
> 801.mbox/%3CCAMreUaz%3DGnsjOLZ%3Dem2x%3DQS7qh27euCWNw6Bo_
> 4Cu%2BfXnXhyNA%40mail.gmail.com%3E
> > [2] https://issues.apache.org/jira/browse/YARN-6592
> > [3] https://issues.apache.org/jira/browse/YARN-7792
> > [4] https://issues.apache.org/jira/browse/YARN-7780
> > [5]
> > https://issues.apache.org/jira/secure/attachment/12867869/
> YARN-6592-Rich-Placement-Constraints-Design-V1.pdf
> > [6] https://issues.apache.org/jira/browse/YARN-7812
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org yarn-dev-unsubscr...@hadoop.apache.org>
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org yarn-dev-h...@hadoop.apache.org>
>
>
>

Re: Hadoop 3.1.0 release discussion

[Arun Suresh]

Hey Wangda

Update: YARN-6592 has been merged to trunk.

Cheers
-Arun

On Wed, Jan 31, 2018 at 12:58 AM, Wangda Tan  wrote:

> Hi Daniel,
>
> I agree, let's move the discussion to YARN-7292 and get it resolved before
> 3.1.0
>
> Thanks,
> Wangda
>
> On Wed, Jan 31, 2018 at 4:35 PM, Daniel Templeton 
> wrote:
>
> > I added my comments on that JIRA.  Looks like YARN-7292 is marked as a
> > blocker for 3.1, and I would tend to agree with that.  Let's see what we
> > can to do get profiles nailed down so that 3.1 can go forward.
> >
> > Daniel
> >
> >
> > On 1/18/18 10:25 AM, Wangda Tan wrote:
> >
> > Thanks Daniel,
> >
> > We need to make a decision about this: https://issues.apache.
> > org/jira/browse/YARN-7292, I believe this is the JIRA you mentioned
> > correct? Please let me know if there's anything else. And let's move the
> > discussion on JIRA.
> >
> > The good news is that resource profile is merged to trunk already so we
> > can finish that before code freeze date (Feb 08).
> >
> > + Sunil as well.
> >
> > Thanks,
> > Wangda
> >
> >
> >
> > On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton 
> > wrote:
> >
> >> What's the status on resource profiles?  I believe there are still a
> >> couple of open JIRAs to rethink some of the design choices.
> >>
> >> Daniel
> >>
> >>
> >> On 1/17/18 11:33 AM, Wangda Tan wrote:
> >>
> >>> Hi All,
> >>>
> >>> Since we're fast approaching previously proposed feature freeze date
> (Jan
> >>> 30, about 13 days from today). If you've any features which live in a
> >>> branch and targeted to 3.1.0, please reply this email thread. Ideally,
> we
> >>> should finish branch merging before feature freeze date.
> >>>
> >>> Here's an updated 3.1.0 feature status:
> >>>
> >>> 1. Merged & Completed features:
> >>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> >>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> >>> end-to-end.
> >>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
> >>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> >>> commits.
> >>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> >>> Queues While Doing Queue Mapping.
> >>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
> >>>
> >>> 2. Features close to finish:
> >>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
> >>> and
> >>> merged to trunk. Except for UI/documentation.
> >>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> >>> discussions going on about implementation.
> >>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> >>> anti-affinity support. Close to finish, on track to be merged before
> Jan
> >>> 30.
> >>>
> >>> 3. Tentative features:
> >>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> >>> containers. Only one pending patch. Plan to finish before Jan 7th.
> >>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to
> >>> be
> >>> done before Jan 2018.
> >>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> >>> challenging to be done before Jan 2018.
> >>> * (Varun V) YARN-5673: container-executor write. Given security
> >>> refactoring
> >>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
> >>> 3.2.
> >>>
> >>> Thanks,
> >>> Wangda
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan 
> wrote:
> >>>
> >>> Hi all,
> 
>  Congratulations on the 3.0.0-GA release!
> 
>  As we discussed in the previous email thread [1], I'd like to restart
>  3.1.0 release plans.
> 
>  a) Quick summary:
>  a.1 Release status
>  We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>  there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
> blockers
>  and
>  22 open critical issues.
> 
>  a.2 Release date update
>  Considering delays of 3.0-GA release by month-and-a-half, I propose to
>  move the dates as follows
>    - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
>  for
>  any branches to get merged too;
>    - code freeze (blockers & critical only) date to Feb 08, 2018;
>    - release voting start by Feb 18, 2018, leaving time for at least
> two
>  RCx
>    - release date from Jan 15, 2018, to Feb 28, 2018;
> 
>  Unlike before, I added an additional milestone for release-vote-start
> so
>  that we can account for voting time-period also.
> 
>  This overall is still 5 1/2 months of release-timeline unlike the
> faster
>  cadence we hoped for, but this, in my opinion, is the best-updated
>  timeline
>  given the delays of the final release of 3.0-GA.
> 
>  b) Individual feature status:
>  I spoke to several feature owners and checked the status of
> un-finished
>  features, following are status of features planned to 3.1.0:
> 
>  b.1 Merged

[jira] [Created] (HDFS-13094) Refactor TestJournalNode

[Bharat Viswanadham (JIRA)]

Bharat Viswanadham created HDFS-13094:
-

 Summary: Refactor TestJournalNode
 Key: HDFS-13094
 URL: https://issues.apache.org/jira/browse/HDFS-13094
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


This Jira is created from the review comment from  [~arpitagarwal] in 
HDFS-13062.
We have used this testName to add testcase-specific behavior in the past but it 
is fragile.

Perhaps we should open a separate Jira to move this behavior to 
testcase-specific init routines by using a test base class and derived classes 
for individual unit tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13095) Improve slice tree traversal implementation

[Rakesh R (JIRA)]

Rakesh R created HDFS-13095:
---

 Summary: Improve slice tree traversal implementation
 Key: HDFS-13095
 URL: https://issues.apache.org/jira/browse/HDFS-13095
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Rakesh R
Assignee: Rakesh R


This task is to refine the existing slice tree traversal logic in 
[ReencryptionHandler|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ReencryptionHandler.java#L74]
 class.

Please refer Daryn's review comments
{quote}*FSTreeTraverser*
 I need to study this more but I have grave concerns this will work correctly 
in a mutating namesystem.  Ex. renames and deletes esp. in combination with 
snapshots. Looks like there's a chance it will go off in the weeds when 
backtracking out of a renamed directory.

traverseDir may NPE if it's traversing a tree in a snapshot and one of the 
ancestors is deleted.

Not sure why it's bothering to re-check permissions during the crawl.  The 
storage policy is inherited by the entire tree, regardless of whether the 
sub-contents are accessible.  The effect of this patch is the storage policy is 
enforced for all readable files, non-readable violate the new storage policy, 
new non-readable will conform to the new storage policy.  Very convoluted.  
Since new files will conform, should just process the entire tree.
{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13096) HDFS group quota

[Ruslan Dautkhanov (JIRA)]

Ruslan Dautkhanov created HDFS-13096:


 Summary: HDFS group quota
 Key: HDFS-13096
 URL: https://issues.apache.org/jira/browse/HDFS-13096
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: datanode, fs, hdfs, nn
Affects Versions: 3.0.0, 2.7.5, 2.8.3
Reporter: Ruslan Dautkhanov


We have groups of people that have their own set of HDFS directories. 
For example, they have HDFS staging place for new files:
/datascience
/analysts 
... 
but at the same time they have Hive warehouse directory 
/hivewarehouse/datascience
/hivewarehouse/analysts 
... 
on top of that they also have some files stored under /user/${username}/ 

It's always been a challenge to maintain a combined quota on all HDFS locations 
a particular group of people owns. As we're currently forced to put a 
particular quota for each directory independently.

It would be great if HDFS would have a quota tied either
- to a set of HDFS locations ;
- or to a group of people (where `group`is defined as which HDFS group a 
particular file/directory belongs to).

Linux allows to define quotas at group level, i.e. `edquota -g devel` etc.. 
would be great to have the same at HDFS level.

Other thoughts and ideas?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-11419) BlockPlacementPolicyDefault is choosing datanode in an inefficient way

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDFS-11419.
--
Resolution: Done

> BlockPlacementPolicyDefault is choosing datanode in an inefficient way
> --
>
> Key: HDFS-11419
> URL: https://issues.apache.org/jira/browse/HDFS-11419
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
>
> Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up 
> calling into {{chooseRandom}}, which will first find a random datanode by 
> calling
> {code}DatanodeDescriptor chosenNode = chooseDataNode(scope, 
> excludedNodes);{code}, then it checks whether that returned datanode 
> satisfies storage type requirement
> {code}storage = chooseStorage4Block(
>   chosenNode, blocksize, results, entry.getKey());{code}
> If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes, 
> and runs the loop again until {{numOfReplicas}} is down to 0.
> A problem here is that, storage type is not being considered only until after 
> a random node is already returned.  We've seen a case where a cluster has a 
> large number of datanodes, while only a few satisfy the storage type 
> condition. So, for the most part, this code blindly picks random datanodes 
> that do not satisfy the storage type requirement.
> To make matters worse, the way {{NetworkTopology#chooseRandom}} works is 
> that, given a set of excluded nodes, it first finds a random datanodes, then 
> if it is in excluded nodes set, try find another random nodes. So the more 
> excluded nodes there are, the more likely a random node will be in the 
> excluded set, in which case we basically wasted one iteration.
> Therefore, this JIRA proposes to augment/modify the relevant classes in a way 
> that datanodes can be found more efficiently. There are currently two 
> different high level solutions we are considering:
> 1. add some field to Node base types to describe the storage type info, and 
> when searching for a node, we take into account such field(s), and do not 
> return node that does not meet the storage type requirement.
> 2. change {{NetworkTopology}} class to be aware of storage types, e.g. for 
> one storage type, there is one tree subset that connects all the nodes with 
> that type. And one search happens on only one such subset. So unexpected 
> storage types are simply not in the search space. 
> Thanks [~szetszwo] for the offline discussion, and thanks [~linyiqun] for 
> pointing out a wrong statement (corrected now) in the description. Any 
> further comments are more than welcome.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13097) [SPS]: Fix the branch review comments(Part1)

[Surendra Singh Lilhore (JIRA)]

Surendra Singh Lilhore created HDFS-13097:
-

 Summary: [SPS]: Fix the branch review comments(Part1)
 Key: HDFS-13097
 URL: https://issues.apache.org/jira/browse/HDFS-13097
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS-10285
Reporter: Surendra Singh Lilhore
Assignee: Surendra Singh Lilhore


Fix the branch review comment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13098) RBF: Datanodes interacting with Routers

[JIRA]

Íñigo Goiri created HDFS-13098:
--

 Summary: RBF: Datanodes interacting with Routers
 Key: HDFS-13098
 URL: https://issues.apache.org/jira/browse/HDFS-13098
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Íñigo Goiri


Datanodes talk to particular Namenodes. We could use the Router infrastructure 
for the Datanodes to register/heartbeating into them and the Routers would 
forward this to particular Namenodes. This would make the assignment of 
Datanodes to subclusters potentially more dynamic.

The implementation would potentially include:
* Router to implement part of DatanodeProtocol
* Forwarding DN messages into Routers
* Policies to assign datanodes to subclusters
* Datanodes to make blockpool configuration dynamic



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/674/

[Jan 30, 2018 5:42:38 PM] (eyang) YARN-7811.  Fixed a bug in user defined 
docker network settings. 
[Jan 30, 2018 11:25:03 PM] (wwei) HDFS-12528. Add an option to not disable 
short-circuit reads on
[Jan 31, 2018 2:27:18 AM] (inigoiri) HDFS-13083. RBF: Fix doc error setting up 
client. Contributed by
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-6593. [API] Introduce Placement 
Constraint object. (Konstantinos
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-6594. [API] Introduce 
SchedulingRequest object. (Konstantinos
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-6595. [API] Add Placement 
Constraints at the application level.
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7448. [API] Add SchedulingRequest 
to the AllocateRequest.
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7522. Introduce 
AllocationTagsManager to associate allocation tags
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7670. Modifications to the 
ResourceScheduler API to support
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7669. API and interface 
modifications for placement constraint
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7653. Node group support for 
AllocationTagsManager. (Panagiotis
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-6596. Introduce Placement 
Constraint Manager module. (Konstantinos
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7612. Add Processor Framework for 
Rich Placement Constraints.
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7613. Implement Basic algorithm 
for constraint based placement.
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7682. Expose canSatisfyConstraints 
utility function to validate a
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7681. Double-check placement 
constraints in scheduling phase before
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7696. Add container tags to 
ContainerTokenIdentifier, api.Container
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-6619. AMRMClient Changes to use 
the PlacementConstraint and
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7709. Remove SELF from 
TargetExpression type. (Konstantinos
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-6599. Support anti-affinity 
constraint via AppPlacementAllocator.
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7745. Allow DistributedShell to 
take a placement specification for
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7774. Miscellaneous fixes to the 
PlacementProcessor. (asuresh)
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7763. Allow Constraints specified 
in the SchedulingRequest to
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7788. Factor out management of 
temp tags from
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7779. Display allocation tags in 
RM web UI and expose same through
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7783. Add validation step to 
ensure constraints are not violated
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7807. Assume intra-app 
anti-affinity as default for scheduling
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7795. Fix jenkins issues of 
YARN-6592 branch. (Sunil G via asuresh)
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7784. Fix Cluster metrics when 
placement processor is enabled.
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-6597. Add RMContainer recovery 
test to verify tag population in the
[Jan 31, 2018 9:30:17 AM] (arun suresh) YARN-7780. Documentation for Placement 
Constraints. (Konstantinos
[Jan 31, 2018 12:10:36 PM] (sunilg) YARN-7802. [UI2] Application regex search 
did not work properly with app
[Jan 31, 2018 1:43:08 PM] (wangda) YARN-7828. Clicking on yarn service should 
take to component tab. (Sunil
[Jan 31, 2018 1:44:42 PM] (wangda) YARN-7861. [UI2] Logs page shows duplicated 
containers with ATS. (Sunil
[Jan 31, 2018 3:39:43 PM] (jlowe) YARN-2185. Use pipes when localizing 
archives. Contributed by Miklos
[Jan 31, 2018 9:51:08 AM] (arun suresh) YARN-7822. Constraint satisfaction 
checker support for composite OR and




-1 overall


The following subsystems voted -1:
findbugs mvnsite unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage 
   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.