[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #72: HDDS-2341. Validate tar entry path during extraction

[GitBox]

adoroszlai opened a new pull request #72: HDDS-2341. Validate tar entry path 
during extraction
URL: https://github.com/apache/hadoop-ozone/pull/72
 
 
   ## What changes were proposed in this pull request?
   
   Add a check to verify tar entries during extraction: they should not be able 
to escape (by using `../` or absolute directory names) from the expected 
archive directory.
   
   https://issues.apache.org/jira/browse/HDDS-2341
   
   ## How was this patch tested?
   
   Added unit test.
   
   Verified in docker-compose cluster that container replication still works 
normally.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2346) Replication manager config has wrong description

[Attila Doroszlai (Jira)]

Attila Doroszlai created HDDS-2346:
--

 Summary: Replication manager config has wrong description
 Key: HDDS-2346
 URL: https://issues.apache.org/jira/browse/HDDS-2346
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: SCM
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai


Replication manager's configuration for its own interval:

{code:title=https://github.com/apache/hadoop-ozone/blob/eb1d77e3206fab1a4ac0573507c9deb2b56b9ea1/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java#L808-L822}
@Config(key = "thread.interval",
type = ConfigType.TIME,
defaultValue = "300s",
tags = {SCM, OZONE},
description = "When a heartbeat from the data node arrives on SCM, "
+ "It is queued for processing with the time stamp of when the "
+ "heartbeat arrived. There is a heartbeat processing thread "
+ "inside "
+ "SCM that runs at a specified interval. This value controls how "
+ "frequently this thread is run.\n\n"
+ "There are some assumptions build into SCM such as this "
+ "value should allow the heartbeat processing thread to run at "
+ "least three times more frequently than heartbeats and at least "
+ "five times more than stale node detection time. "
+ "If you specify a wrong value, SCM will gracefully refuse to "
+ "run. "
+ "For more info look at the node manager tests in SCM.\n"
+ "\n"
+ "In short, you don't need to change this."
)
{code}

duplicates SCM heartbeat interval doc:

{code:title=https://github.com/apache/hadoop-ozone/blob/eb1d77e3206fab1a4ac0573507c9deb2b56b9ea1/hadoop-hdds/common/src/main/resources/ozone-default.xml#L973-L991}
  
ozone.scm.heartbeat.thread.interval
3s
OZONE, MANAGEMENT

  When a heartbeat from the data node arrives on SCM, It is queued for
  processing with the time stamp of when the heartbeat arrived. There is a
  heartbeat processing thread inside SCM that runs at a specified interval.
  This value controls how frequently this thread is run.

  There are some assumptions build into SCM such as this value should allow
  the heartbeat processing thread to run at least three times more
  frequently than heartbeats and at least five times more than stale node
  detection time. If you specify a wrong value, SCM will gracefully refuse
  to run. For more info look at the node manager tests in SCM.

  In short, you don't need to change this.

  
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.10.0 (RC0)

[Zhankun Tang]

Thanks for the effort, Jonathan!

+1 (non-binding) on RC0.
 - Set up a single node cluster with the binary tarball
 - Run Spark Pi and pySpark job

BR,
Zhankun

On Tue, 22 Oct 2019 at 14:31, Konstantin Shvachko 
wrote:

> +1 on RC0.
> - Verified signatures
> - Built from sources
> - Ran unit tests for new features
> - Checked artifacts on Nexus, made sure the sources are present.
>
> Thanks
> --Konstantin
>
>
> On Wed, Oct 16, 2019 at 6:01 PM Jonathan Hung 
> wrote:
>
> > Hi folks,
> >
> > This is the first release candidate for the first release of Apache
> Hadoop
> > 2.10 line. It contains 361 fixes/improvements since 2.9 [1]. It includes
> > features such as:
> >
> > - User-defined resource types
> > - Native GPU support as a schedulable resource type
> > - Consistent reads from standby node
> > - Namenode port based selective encryption
> > - Improvements related to rolling upgrade support from 2.x to 3.x
> >
> > The RC0 artifacts are at:
> http://home.apache.org/~jhung/hadoop-2.10.0-RC0/
> >
> > RC tag is release-2.10.0-RC0.
> >
> > The maven artifacts are hosted here:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1241/
> >
> > My public key is available here:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > The vote will run for 5 weekdays, until Wednesday, October 23 at 6:00 pm
> > PDT.
> >
> > Thanks,
> > Jonathan Hung
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0)
> >
>

[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #73: HDDS-2346. Replication manager config has wrong description

[GitBox]

adoroszlai opened a new pull request #73: HDDS-2346. Replication manager config 
has wrong description
URL: https://github.com/apache/hadoop-ozone/pull/73
 
 
   ## What changes were proposed in this pull request?
   
   Copy the correct description from [old 
`ozone-default.xml`](https://github.com/apache/hadoop/pull/773/files#diff-1e26884a3be02959ddb7c4c65591f630).
  Reformatted to make checkstyle happy.
   
   https://issues.apache.org/jira/browse/HDDS-2346
   
   ## How was this patch tested?
   
   Verified the generated config property in 
`hadoop-hdds/server-scm/target/classes/ozone-default-generated.xml`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #7: HDDS-1228. Chunk Scanner Checkpoints

[GitBox]

adoroszlai commented on a change in pull request #7: HDDS-1228. Chunk Scanner 
Checkpoints
URL: https://github.com/apache/hadoop-ozone/pull/7#discussion_r337399474
 
 

 ##
 File path: 
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerDataScanOrder.java
 ##
 @@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.container.common.impl;
+
+import org.apache.hadoop.ozone.container.common.interfaces.Container;
+
+import java.time.Instant;
+import java.util.Comparator;
+import java.util.Optional;
+
+/**
+ * Orders containers:
+ * 1. containers not yet scanned first,
+ * 2. then least recently scanned first,
+ * 3. ties are broken by containerID.
+ */
+public class ContainerDataScanOrder implements Comparator> {
+
+  public static final Comparator> INSTANCE =
+  new ContainerDataScanOrder();
+
+  @Override
+  public int compare(Container o1, Container o2) {
+ContainerData d1 = o1.getContainerData();
+ContainerData d2 = o2.getContainerData();
+
+Optional scan1 = d1.lastDataScanTime();
+boolean scanned1 = scan1.isPresent();
+Optional scan2 = d2.lastDataScanTime();
+boolean scanned2 = scan2.isPresent();
+
+int result = Boolean.compare(scanned1, scanned2);
+if (0 == result && scanned1 && scanned2) {
+  result = scan1.get().compareTo(scan2.get());
+}
+if (0 == result) {
+  result = Long.compare(d1.getContainerID(), d2.getContainerID());
 
 Review comment:
   Thanks @arp7 for reviewing it again.
   
   The first check compares presence of scan timestamps and takes care of the 
case you mention:
   
   
https://github.com/apache/hadoop-ozone/blob/019f5d313bea56ec3009fbe9f773d21a85cfe54b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerDataScanOrder.java#L47
   
   The unit test also checks it:
   
   
https://github.com/apache/hadoop-ozone/blob/019f5d313bea56ec3009fbe9f773d21a85cfe54b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerSet.java#L212-L224


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bshashikant commented on a change in pull request #54: HDDS-2281. ContainerStateMachine#handleWriteChunk should ignore close container exception

[GitBox]

bshashikant commented on a change in pull request #54: HDDS-2281. 
ContainerStateMachine#handleWriteChunk should ignore close container exception
URL: https://github.com/apache/hadoop-ozone/pull/54#discussion_r337403878
 
 

 ##
 File path: 
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java
 ##
 @@ -460,6 +460,10 @@ private ExecutorService getCommandExecutor(
 LOG.error(gid + ": writeChunk writeStateMachineData failed: 
blockId"
 + write.getBlockID() + " logIndex " + entryIndex + " chunkName 
"
 + write.getChunkData().getChunkName() + e);
+metrics.incNumWriteDataFails();
+// write chunks go in parallel. It's possible that one write chunk
+// see the stateMachine is marked unhealthy by other parallel 
thread.
+stateMachineHealthy.set(false);
 
 Review comment:
   > So question; if a thread has marked the container as unhealthy why should 
a write be successful at all ?
   
   If a container is marked unhealthy, write will marked fail and log Append 
wlll fail in ratis. This is juts incrementing fail count metrics here and 
marking the stateMachine for the pipeline unhealthy so that now new ratis 
snapshots can be taken .


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bshashikant commented on a change in pull request #54: HDDS-2281. ContainerStateMachine#handleWriteChunk should ignore close container exception

[GitBox]

bshashikant commented on a change in pull request #54: HDDS-2281. 
ContainerStateMachine#handleWriteChunk should ignore close container exception
URL: https://github.com/apache/hadoop-ozone/pull/54#discussion_r337406349
 
 

 ##
 File path: 
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java
 ##
 @@ -460,6 +460,10 @@ private ExecutorService getCommandExecutor(
 LOG.error(gid + ": writeChunk writeStateMachineData failed: 
blockId"
 + write.getBlockID() + " logIndex " + entryIndex + " chunkName 
"
 + write.getChunkData().getChunkName() + e);
+metrics.incNumWriteDataFails();
+// write chunks go in parallel. It's possible that one write chunk
+// see the stateMachine is marked unhealthy by other parallel 
thread.
+stateMachineHealthy.set(false);
 
 Review comment:
   > I know this patch is merged, but I have no way of understanding what this 
means -- so I appreciate some comments or feedback that explains what happens 
here?
   
   Updated the description to add clarity to this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bshashikant commented on issue #40: HDDS-2285. GetBlock and ReadChunk command from the client should be s…

[GitBox]

bshashikant commented on issue #40: HDDS-2285. GetBlock and ReadChunk command 
from the client should be s…
URL: https://github.com/apache/hadoop-ozone/pull/40#issuecomment-544884381
 
 
   Thanks @hanishakoneru for working on this. I am thinking of a case where, 
let's say two threads trying to read two different keys using the same 
XceiverClientGrpc object as it is shared object, one thread issues a getBlock 
for block1 to dn1, dn1 got cached and further read chunk calls for all chunks 
for all blocks(block1 and block2) will go the same dn. The idea of shuffling 
the datanodes was basically to distribute the load across the datanodes in a 
pipeline, but for the same block it should always be tried on the same order of 
datanodes. I think the patch won't address both these concerns here. What do 
you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bshashikant edited a comment on issue #40: HDDS-2285. GetBlock and ReadChunk command from the client should be s…

[GitBox]

bshashikant edited a comment on issue #40: HDDS-2285. GetBlock and ReadChunk 
command from the client should be s…
URL: https://github.com/apache/hadoop-ozone/pull/40#issuecomment-544884381
 
 
   Thanks @hanishakoneru for working on this. I am thinking of a case where, 
let's say two threads trying to read two different blocks using the same 
XceiverClientGrpc object as it is shared object, one thread issues a getBlock 
for block1 to dn1, dn1 got cached and further read chunk calls for all chunks 
for all blocks(block1 and block2) will go the same dn. The idea of shuffling 
the datanodes was basically to distribute the load across the datanodes in a 
pipeline, but for the same block it should always be tried on the same order of 
datanodes. I think the patch won't address both these concerns here. What do 
you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bshashikant commented on issue #65: HDDS-2334. Dummy chunk manager fails with length mismatch error

[GitBox]

bshashikant commented on issue #65: HDDS-2334. Dummy chunk manager fails with 
length mismatch error
URL: https://github.com/apache/hadoop-ozone/pull/65#issuecomment-544886654
 
 
   Thanks @adoroszlai for working on this, The patch looks good tom me. I am +1 
on the change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2347) XCeiverClientGrpc's parallel use leads to NPE

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2347:
--

 Summary: XCeiverClientGrpc's parallel use leads to NPE
 Key: HDDS-2347
 URL: https://issues.apache.org/jira/browse/HDDS-2347
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Ozone Client
Reporter: Istvan Fajth
Assignee: Istvan Fajth


This issue came up when testing Hive with ORC tables on Ozone storage backend, 
I so far I could not reproduce it locally within a JUnit test but the issue.

I am attaching a diff file that shows what logging I have added in 
XCevierClientGrpc and in KeyInputStream to get the results that made me arrive 
to the following understanding of the scenario:
- Hive starts a couple of threads to work on the table data during query 
execution
- There is one RPCClient that is being used by these threads
- The threads are opening different stream to read from the same key in ozone
- The InputStreams internally are using the same XCeiverClientGrpc
- XCeiverClientGrpc throws the following NPE intermittently:

{code}
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:398)
at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:295)
at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:259)
at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:242)
at 
org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
at 
org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:169)
at 
org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
at 
org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:224)
at 
org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:173)
at 
org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
at 
org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
at 
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
at 
org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1708)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1596)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
{code}

I have two proposals to fix this issue, one is the easy answer to put 
synchronization to the XCeiverClientGrpc code, the other one is a bit more 
complicated, let me explain below.

Naively I would assume that when I get a client SPI instance from 
XCeiverClientManager, that instance is ready to use. In fact it is not, and 
when the user of the SPI instance sends the first request that is the point 
when the client gets essentially ready. Now if we put synchronization to this 
code, that is the easy solution, but my pragmatic half screams for a better 
solution, that ensures that the Manager essentially manages the clients that is 
giving to it's users, and the clients themselves are not getting ready by 
accident.
I am working on a proposal that moves things around a bit, and looking for 
possible other solutions that does not feel hacky as I feel with the easy 
solution.

I am attaching the followings:
- a diff that shows the added extended logging in XCeiverClientGrpc and 
KeyInputStream.
- a job log snippet from a Hive query that shows the relevant output from the 
extensive logging added by the diff in a cluster.
- later a proposal for

[GitHub] [hadoop-ozone] timmylicheng commented on issue #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

timmylicheng commented on issue #28: HDDS-1569 Support creating multiple 
pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#issuecomment-544932882
 
 
   /retest


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14920) Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission

[Fei Hui (Jira)]

Fei Hui created HDFS-14920:
--

 Summary: Erasure Coding: Decommission may hang If one or more 
datanodes are out of service during decommission  
 Key: HDFS-14920
 URL: https://issues.apache.org/jira/browse/HDFS-14920
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: ec
Affects Versions: 3.1.3, 3.2.1, 3.0.3
Reporter: Fei Hui


Decommission test hangs in our clusters.
Have seen the messages as follow
{quote}
2019-10-22 15:58:51,514 TRACE 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Block 
blk_-9223372035600425840_372987973 numExpected=9, numLive=5
2019-10-22 15:58:51,514 INFO BlockStateChange: Block: 
blk_-9223372035600425840_372987973, Expected Replicas: 9, live replicas: 5, 
corrupt replicas: 0, decommissioned replicas: 0, decommissioning replicas: 4, 
maintenance replicas: 0, live entering maintenance replicas: 0, excess 
replicas: 0, Is Open File: false, Datanodes having this block: 
10.255.43.57:50010 10.255.53.12:50010 10.255.63.12:50010 10.255.62.39:50010 
10.255.37.36:50010 10.255.33.15:50010 10.255.69.29:50010 10.255.51.13:50010 
10.255.64.15:50010 , Current Datanode: 10.255.69.29:50010, Is current datanode 
decommissioning: true, Is current datanode entering maintenance: false
2019-10-22 15:58:51,514 DEBUG 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Node 
10.255.69.29:50010 still has 1 blocks to replicate before it is a candidate to 
finish Decommission In Progress
{quote}

After digging the source code and cluster log,  guess it happens as follow 
steps.
# Storage strategy is RS-6-3-1024k.
# EC block b consists of b0, b1, b2, b3, b4, b5, b6, b7, b8, b0 is from 
datanode dn0, b1 is from datanode dn1, ...etc
# At the beginning dn0 is in decommission progress, b0 is replicated 
successfully, and dn0 is staill in decommission progress.
# Later b1, b2, b3 in decommission progress, and dn4 containing b4 is out of 
service, so need to reconstruct, and create ErasureCodingWork to do it, in the 
ErasureCodingWork, additionalReplRequired is 4
# Because hasAllInternalBlocks is false, Will call 
ErasureCodingWork#addTaskToDatanode -> 
DatanodeDescriptor#addBlockToBeErasureCoded, and send BlockECReconstructionInfo 
task to Datanode
# DataNode can not reconstruction the block because targets is 4, greater than 
3( parity number).

There is a problem as follow, from BlockManager.java
{code}
  // should reconstruct all the internal blocks before scheduling
  // replication task for decommissioning node(s).
  if (additionalReplRequired - numReplicas.decommissioning() -
  numReplicas.liveEnteringMaintenanceReplicas() > 0) {
additionalReplRequired = additionalReplRequired -
numReplicas.decommissioning() -
numReplicas.liveEnteringMaintenanceReplicas();
  }
{code}
Should reconstruction firstly and then replicate for decommissioning. Because 
numReplicas.decommissioning() is 4, and additionalReplRequired is 4, that's 
wrong.
numReplicas.decommissioning() should be 3, it should exclude live replica. If 
so, additionalReplRequired will be 1, reconstruction will schedule as expected. 
After that, decommission goes on.
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/

[Oct 21, 2019 9:36:42 PM] (kihwal) HDFS-12749. DN may not send block report to 
NN after NN restart.




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client
 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 335] 

Failed junit tests :

   hadoop.util.TestReadWriteDiskValidator 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.registry.secure.TestSecureLogins 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt
  [328K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-compile-cc-root-jdk1.8.0_222.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-compile-javac-root-jdk1.8.0_222.txt
  [308K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-patch-shellcheck.txt
  [72K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/whitespace-tabs.txt
  [1.3M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/xml.txt
  [12K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_222.txt
  [1.1M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [160K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [236K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/482/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-regi

[jira] [Created] (HDDS-2348) remove log4j properties for org.apache.hadoop.ozone

[luhuachao (Jira)]

luhuachao created HDDS-2348:
---

 Summary: remove log4j properties for org.apache.hadoop.ozone
 Key: HDDS-2348
 URL: https://issues.apache.org/jira/browse/HDDS-2348
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Ozone Manager
Affects Versions: 0.5.0
Reporter: luhuachao


The log in package org.apache.hadoop.ozone cannot be logged to .log file ;such 
as OM startup_msg .



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: How should we do about dependency update?

[Sean Busbey]

speaking with my HBase hat on instead of my Hadoop hat, when the
Hadoop project publishes that there's a CVE but does not include a
maintenance release that mitigates it for a given minor release line,
we assume that means the Hadoop project is saying that release line is
EOM and should be abandoned.

I don't know if that's an accurate interpretation in all cases.

With my Hadoop hat on, I think downstream projects should use the
interfaces we say are safe to use and those interfaces should not
include dependencies where practical. I don't know how often a CVE
comes along for things like our logging API dependency, for example.
But downstream folks should definitely not rely on dependencies we use
for internal service, so I'm surprised that a version change for Jetty
would impact downstream.


On Mon, Oct 21, 2019 at 12:33 PM Wei-Chiu Chuang  wrote:
>
> Hi Hadoop developers,
>
> I've always had this question and I don't know the answer.
>
> For the last few months I finally spent time to deal with the vulnerability
> reports from our internal dependency check tools.
>
> Say in HADOOP-16152 
> we update Jetty from 9.3.27 to 9.4.20 because of CVE-2019-16869, should I
> cherrypick the fix into all lower releases?
> This is not a trivial change, and it breaks downstreams like Tez. On the
> other hand, it doesn't seem reasonable if I put this fix only in trunk, and
> left older releases vulnerable. What's the expectation of downstream
> applications w.r.t breaking compatibility vs fixing security issues?
>
> Thoughts?



--
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] adoroszlai commented on issue #6: HDDS-2214. TestSCMContainerPlacementRackAware has an intermittent fai…

[GitBox]

adoroszlai commented on issue #6: HDDS-2214. TestSCMContainerPlacementRackAware 
has an intermittent fai…
URL: https://github.com/apache/hadoop-ozone/pull/6#issuecomment-545039722
 
 
   @ChenSammi could you please review this fix when you have some time?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #73: HDDS-2346. Replication manager config has wrong description

[GitBox]

bharatviswa504 merged pull request #73: HDDS-2346. Replication manager config 
has wrong description
URL: https://github.com/apache/hadoop-ozone/pull/73
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #73: HDDS-2346. Replication manager config has wrong description

[GitBox]

bharatviswa504 commented on issue #73: HDDS-2346. Replication manager config 
has wrong description
URL: https://github.com/apache/hadoop-ozone/pull/73#issuecomment-545043970
 
 
   Thank You @adoroszlai for the contribution.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #27: HDDS-2278. Run S3 test suite on OM HA cluster.

[GitBox]

bharatviswa504 commented on issue #27: HDDS-2278. Run S3 test suite on OM HA 
cluster.
URL: https://github.com/apache/hadoop-ozone/pull/27#issuecomment-545044627
 
 
   /retest


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] adoroszlai commented on issue #73: HDDS-2346. Replication manager config has wrong description

[GitBox]

adoroszlai commented on issue #73: HDDS-2346. Replication manager config has 
wrong description
URL: https://github.com/apache/hadoop-ozone/pull/73#issuecomment-545044798
 
 
   Thanks @bharatviswa504 for reviewing and merging this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1297/

[Oct 21, 2019 12:50:00 PM] (ayushsaxena) HDFS-14916. RBF: line breaks are 
missing from the output of 'hdfs
[Oct 21, 2019 1:05:12 PM] (ayushsaxena) HDFS-14913. Correct the value of 
available count in
[Oct 21, 2019 8:16:49 PM] (eyang) YARN-9922. Fixed YARN javadoc errors from 
YARN-9699.   
[Oct 21, 2019 8:30:50 PM] (ericp) YARN-9915: Fix FindBug issue in QueueMetrics. 
Contributed by Prabhu
[Oct 21, 2019 9:33:15 PM] (kihwal) HDFS-12749. DN may not send block report to 
NN after NN restart.
[Oct 22, 2019 12:31:43 AM] (weichiu) HDFS-13901. INode access time is ignored 
because of race between open




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core
 
   Class org.apache.hadoop.applications.mawo.server.common.TaskStatus 
implements Cloneable but does not define or use clone method At 
TaskStatus.java:does not define or use clone method At TaskStatus.java:[lines 
39-346] 
   Equals method for 
org.apache.hadoop.applications.mawo.server.worker.WorkerId assumes the argument 
is of type WorkerId At WorkerId.java:the argument is of type WorkerId At 
WorkerId.java:[line 114] 
   
org.apache.hadoop.applications.mawo.server.worker.WorkerId.equals(Object) does 
not check for null argument At WorkerId.java:null argument At 
WorkerId.java:[lines 114-115] 

FindBugs :

   module:hadoop-cloud-storage-project/hadoop-cos 
   Redundant nullcheck of dir, which is known to be non-null in 
org.apache.hadoop.fs.cosn.BufferPool.createDir(String) Redundant null check at 
BufferPool.java:is known to be non-null in 
org.apache.hadoop.fs.cosn.BufferPool.createDir(String) Redundant null check at 
BufferPool.java:[line 66] 
   org.apache.hadoop.fs.cosn.CosNInputStream$ReadBuffer.getBuffer() may 
expose internal representation by returning CosNInputStream$ReadBuffer.buffer 
At CosNInputStream.java:by returning CosNInputStream$ReadBuffer.buffer At 
CosNInputStream.java:[line 87] 
   Found reliance on default encoding in 
org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFile(String, File, 
byte[]):in org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFile(String, 
File, byte[]): new String(byte[]) At CosNativeFileSystemStore.java:[line 199] 
   Found reliance on default encoding in 
org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFileWithRetry(String, 
InputStream, byte[], long):in 
org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFileWithRetry(String, 
InputStream, byte[], long): new String(byte[]) At 
CosNativeFileSystemStore.java:[line 178] 
   org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.uploadPart(File, 
String, String, int) may fail to clean up java.io.InputStream Obligation to 
clean up resource created at CosNativeFileSystemStore.java:fail to clean up 
java.io.InputStream Obligation to clean up resource created at 
CosNativeFileSystemStore.java:[line 252] is not discharged 

FindBugs :

   module:hadoop-ozone/csi 
   Useless control flow in 
csi.v1.Csi$CapacityRange$Builder.maybeForceBuilderInitialization() At Csi.java: 
At Csi.java:[line 15977] 
   Class csi.v1.Csi$ControllerExpandVolumeRequest defines non-transient 
non-serializable instance field secrets_ In Csi.java:instance field secrets_ In 
Csi.java 
   Useless control flow in 
csi.v1.Csi$ControllerExpandVolumeRequest$Builder.maybeForceBuilderInitialization()
 At Csi.java: At Csi.java:[line 50408] 
   Useless control flow in 
csi.v1.Csi$ControllerExpandVolumeResponse$Builder.maybeForceBuilderInitialization()
 At Csi.ja

Re: How should we do about dependency update?

[Wei-Chiu Chuang]

Hi Sean,
Thanks for the valuable feedback.
Good point on not using dependency classes in public API parameters. One
example is HADOOP-15502
 (blame
me for breaking the API)

>From what I know, the biggest risk is that downstreamers include
dependencies from Hadoop implicitly. Therefore, if Hadoop updates a
dependency that has a breaking change, the downstream application breaks,
sometimes during compile time, sometimes at runtime.

Jetty is probably not the best example. But take Guava as an example, when
we updated Guava from 11.0 to 27.0, it breaks downstreamers like crazy --
Hive, Tez, Pheonix, Oozie all have to make changes. Probably not Hadoop's
responsibility that they don't use shaded Hadoop client artifacts, but
maybe we can add release note and state potential breaking changes.

On Tue, Oct 22, 2019 at 7:43 AM Sean Busbey 
wrote:

> speaking with my HBase hat on instead of my Hadoop hat, when the
> Hadoop project publishes that there's a CVE but does not include a
> maintenance release that mitigates it for a given minor release line,
> we assume that means the Hadoop project is saying that release line is
> EOM and should be abandoned.
>
> I don't know if that's an accurate interpretation in all cases.
>
> With my Hadoop hat on, I think downstream projects should use the
> interfaces we say are safe to use and those interfaces should not
> include dependencies where practical. I don't know how often a CVE
> comes along for things like our logging API dependency, for example.
> But downstream folks should definitely not rely on dependencies we use
> for internal service, so I'm surprised that a version change for Jetty
> would impact downstream.
>
>
> On Mon, Oct 21, 2019 at 12:33 PM Wei-Chiu Chuang 
> wrote:
> >
> > Hi Hadoop developers,
> >
> > I've always had this question and I don't know the answer.
> >
> > For the last few months I finally spent time to deal with the
> vulnerability
> > reports from our internal dependency check tools.
> >
> > Say in HADOOP-16152 
> > we update Jetty from 9.3.27 to 9.4.20 because of CVE-2019-16869, should I
> > cherrypick the fix into all lower releases?
> > This is not a trivial change, and it breaks downstreams like Tez. On the
> > other hand, it doesn't seem reasonable if I put this fix only in trunk,
> and
> > left older releases vulnerable. What's the expectation of downstream
> > applications w.r.t breaking compatibility vs fixing security issues?
> >
> > Thoughts?
>
>
>
> --
> busbey
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[GitHub] [hadoop-ozone] xiaoyuyao commented on issue #6: HDDS-2214. TestSCMContainerPlacementRackAware has an intermittent fai…

[GitBox]

xiaoyuyao commented on issue #6: HDDS-2214. TestSCMContainerPlacementRackAware 
has an intermittent fai…
URL: https://github.com/apache/hadoop-ozone/pull/6#issuecomment-545070633
 
 
   Thanks @elek  for fixing  this and @adoroszlai  for the review. The change 
LGTM. I will merge it shortly. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao merged pull request #6: HDDS-2214. TestSCMContainerPlacementRackAware has an intermittent fai…

[GitBox]

xiaoyuyao merged pull request #6: HDDS-2214. TestSCMContainerPlacementRackAware 
has an intermittent fai…
URL: https://github.com/apache/hadoop-ozone/pull/6
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: How should we do about dependency update?

[Steve Loughran]

We don't have a complete set of shaded artefacts -so it's not fair to point
at the downstream users and say it's "your own fault". We need to do more
here ourselves.


Here: it is a CVE, they should upgrade anyway. Guava is special because it
has been so brittle cross versions and so widely used by so many
applications.

On Tue, Oct 22, 2019 at 6:15 PM Wei-Chiu Chuang  wrote:

> Hi Sean,
> Thanks for the valuable feedback.
> Good point on not using dependency classes in public API parameters. One
> example is HADOOP-15502
>  (blame
> me for breaking the API)
>
> From what I know, the biggest risk is that downstreamers include
> dependencies from Hadoop implicitly. Therefore, if Hadoop updates a
> dependency that has a breaking change, the downstream application breaks,
> sometimes during compile time, sometimes at runtime.
>
> Jetty is probably not the best example. But take Guava as an example, when
> we updated Guava from 11.0 to 27.0, it breaks downstreamers like crazy --
> Hive, Tez, Pheonix, Oozie all have to make changes. Probably not Hadoop's
> responsibility that they don't use shaded Hadoop client artifacts, but
> maybe we can add release note and state potential breaking changes.
>
> On Tue, Oct 22, 2019 at 7:43 AM Sean Busbey 
> wrote:
>
> > speaking with my HBase hat on instead of my Hadoop hat, when the
> > Hadoop project publishes that there's a CVE but does not include a
> > maintenance release that mitigates it for a given minor release line,
> > we assume that means the Hadoop project is saying that release line is
> > EOM and should be abandoned.
> >
> > I don't know if that's an accurate interpretation in all cases.
> >
> > With my Hadoop hat on, I think downstream projects should use the
> > interfaces we say are safe to use and those interfaces should not
> > include dependencies where practical. I don't know how often a CVE
> > comes along for things like our logging API dependency, for example.
> > But downstream folks should definitely not rely on dependencies we use
> > for internal service, so I'm surprised that a version change for Jetty
> > would impact downstream.
> >
> >
> > On Mon, Oct 21, 2019 at 12:33 PM Wei-Chiu Chuang 
> > wrote:
> > >
> > > Hi Hadoop developers,
> > >
> > > I've always had this question and I don't know the answer.
> > >
> > > For the last few months I finally spent time to deal with the
> > vulnerability
> > > reports from our internal dependency check tools.
> > >
> > > Say in HADOOP-16152 <
> https://issues.apache.org/jira/browse/HADOOP-16152>
> > > we update Jetty from 9.3.27 to 9.4.20 because of CVE-2019-16869,
> should I
> > > cherrypick the fix into all lower releases?
> > > This is not a trivial change, and it breaks downstreams like Tez. On
> the
> > > other hand, it doesn't seem reasonable if I put this fix only in trunk,
> > and
> > > left older releases vulnerable. What's the expectation of downstream
> > > applications w.r.t breaking compatibility vs fixing security issues?
> > >
> > > Thoughts?
> >
> >
> >
> > --
> > busbey
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>

[GitHub] [hadoop-ozone] swagle commented on a change in pull request #50: HDDS-2131. Optimize replication type and creation time calculation in S3 MPU list call.

[GitBox]

swagle commented on a change in pull request #50: HDDS-2131. Optimize 
replication type and creation time calculation in S3 MPU list call.
URL: https://github.com/apache/hadoop-ozone/pull/50#discussion_r337670962
 
 

 ##
 File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
 ##
 @@ -893,17 +893,18 @@ private OmMultipartInfo createMultipartInfo(OmKeyArgs 
keyArgs,
   // Not checking if there is an already key for this in the keyTable, as
   // during final complete multipart upload we take care of this.
 
-
+  long currentTime = Time.now();
   Map partKeyInfoMap = new HashMap<>();
   OmMultipartKeyInfo multipartKeyInfo = new OmMultipartKeyInfo(
-  multipartUploadID, partKeyInfoMap);
+  multipartUploadID, currentTime, keyArgs.getType(),
 
 Review comment:
   Hi @bharatviswa504, I had already changed this in my previous patch.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] swagle commented on a change in pull request #50: HDDS-2131. Optimize replication type and creation time calculation in S3 MPU list call.

[GitBox]

swagle commented on a change in pull request #50: HDDS-2131. Optimize 
replication type and creation time calculation in S3 MPU list call.
URL: https://github.com/apache/hadoop-ozone/pull/50#discussion_r337670962
 
 

 ##
 File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
 ##
 @@ -893,17 +893,18 @@ private OmMultipartInfo createMultipartInfo(OmKeyArgs 
keyArgs,
   // Not checking if there is an already key for this in the keyTable, as
   // during final complete multipart upload we take care of this.
 
-
+  long currentTime = Time.now();
   Map partKeyInfoMap = new HashMap<>();
   OmMultipartKeyInfo multipartKeyInfo = new OmMultipartKeyInfo(
-  multipartUploadID, partKeyInfoMap);
+  multipartUploadID, currentTime, keyArgs.getType(),
 
 Review comment:
   Hi @bharatviswa504, I had already changed this in my previous patch. Since 
it is the only constructor code would not compile otherwise.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] swagle commented on issue #50: HDDS-2131. Optimize replication type and creation time calculation in S3 MPU list call.

[GitBox]

swagle commented on issue #50: HDDS-2131. Optimize replication type and 
creation time calculation in S3 MPU list call.
URL: https://github.com/apache/hadoop-ozone/pull/50#issuecomment-545094626
 
 
   Hi @bharatviswa504 the tests are failing since we do not have RATIS-707 fix, 
it is unrelated. Do you want me to rebase and push?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.10.0 (RC0)

[Jonathan Hung]

Thanks Konstantin and Zhankun. Unfortunately a feature slipped our radar
(HDFS-14667). Since this is the first of a minor release, we would like to
get it into 2.10.0.

HDFS-14667 has been committed to branch-2.10.0, I will be rolling an RC1
shortly.

Jonathan Hung


On Tue, Oct 22, 2019 at 1:39 AM Zhankun Tang  wrote:

> Thanks for the effort, Jonathan!
>
> +1 (non-binding) on RC0.
>  - Set up a single node cluster with the binary tarball
>  - Run Spark Pi and pySpark job
>
> BR,
> Zhankun
>
> On Tue, 22 Oct 2019 at 14:31, Konstantin Shvachko 
> wrote:
>
>> +1 on RC0.
>> - Verified signatures
>> - Built from sources
>> - Ran unit tests for new features
>> - Checked artifacts on Nexus, made sure the sources are present.
>>
>> Thanks
>> --Konstantin
>>
>>
>> On Wed, Oct 16, 2019 at 6:01 PM Jonathan Hung 
>> wrote:
>>
>> > Hi folks,
>> >
>> > This is the first release candidate for the first release of Apache
>> Hadoop
>> > 2.10 line. It contains 361 fixes/improvements since 2.9 [1]. It includes
>> > features such as:
>> >
>> > - User-defined resource types
>> > - Native GPU support as a schedulable resource type
>> > - Consistent reads from standby node
>> > - Namenode port based selective encryption
>> > - Improvements related to rolling upgrade support from 2.x to 3.x
>> >
>> > The RC0 artifacts are at:
>> http://home.apache.org/~jhung/hadoop-2.10.0-RC0/
>> >
>> > RC tag is release-2.10.0-RC0.
>> >
>> > The maven artifacts are hosted here:
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1241/
>> >
>> > My public key is available here:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > The vote will run for 5 weekdays, until Wednesday, October 23 at 6:00 pm
>> > PDT.
>> >
>> > Thanks,
>> > Jonathan Hung
>> >
>> > [1]
>> >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0)
>> >
>>
>

[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #50: HDDS-2131. Optimize replication type and creation time calculation in S3 MPU list call.

[GitBox]

bharatviswa504 merged pull request #50: HDDS-2131. Optimize replication type 
and creation time calculation in S3 MPU list call.
URL: https://github.com/apache/hadoop-ozone/pull/50
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #50: HDDS-2131. Optimize replication type and creation time calculation in S3 MPU list call.

[GitBox]

bharatviswa504 commented on issue #50: HDDS-2131. Optimize replication type and 
creation time calculation in S3 MPU list call.
URL: https://github.com/apache/hadoop-ozone/pull/50#issuecomment-545105335
 
 
   Thank You @swagle for the verification and for the contribution and 
@anuengineer for the review.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2131) Optimize replication type and creation time calculation in S3 MPU list call

[Bharat Viswanadham (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham resolved HDDS-2131.
--
Fix Version/s: 0.5.0
   Resolution: Fixed

> Optimize replication type and creation time calculation in S3 MPU list call
> ---
>
> Key: HDDS-2131
> URL: https://issues.apache.org/jira/browse/HDDS-2131
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Marton Elek
>Assignee: Siddharth Wagle
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Based on the review from [~bharatviswa]:
> {code}
>  
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
>   metadataManager.getOpenKeyTable();
>   OmKeyInfo omKeyInfo =
>   openKeyTable.get(upload.getDbKey());
> {code}
> {quote}Here we are reading openKeyTable only for getting creation time. If we 
> can have this information in omMultipartKeyInfo, we could avoid DB calls for 
> openKeyTable.
> To do this, We can set creationTime in OmMultipartKeyInfo during 
> initiateMultipartUpload . In this way, we can get all the required 
> information from the MultipartKeyInfo table.
> And also StorageClass is missing from the returned OmMultipartUpload, as 
> listMultipartUploads shows StorageClass information. For this, if we can 
> return replicationType and depending on this value, we can set StorageClass 
> in the listMultipartUploads Response.
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 2.10.0 (RC1)

[Jonathan Hung]

Hi folks,

This is the second release candidate for the first release of Apache Hadoop
2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes
features such as:

- User-defined resource types
- Native GPU support as a schedulable resource type
- Consistent reads from standby node
- Namenode port based selective encryption
- Improvements related to rolling upgrade support from 2.x to 3.x
- Cost based fair call queue

The RC1 artifacts are at: http://home.apache.org/~jhung/hadoop-2.10.0-RC1/

RC tag is release-2.10.0-RC1.

The maven artifacts are hosted here:
https://repository.apache.org/content/repositories/orgapachehadoop-1243/

My public key is available here:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm PDT.

Thanks,
Jonathan Hung

[1]
https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0)

Re: [VOTE] Release Apache Hadoop 2.10.0 (RC0)

[Eric Badger]

Hi Jonathan,

Thanks for putting this RC together. You stated that there are improvements
related to rolling upgrades from 2.x to 3.x and I know I have seen multiple
JIRAs getting committed to that effect. Could you describe any tests that
you have done to verify rolling upgrade compatibility for 3.x servers
talking to 2.x clients and vice versa?

Thanks,

Eric

On Tue, Oct 22, 2019 at 1:49 PM Jonathan Hung  wrote:

> Thanks Konstantin and Zhankun. Unfortunately a feature slipped our radar
> (HDFS-14667). Since this is the first of a minor release, we would like to
> get it into 2.10.0.
>
> HDFS-14667 has been committed to branch-2.10.0, I will be rolling an RC1
> shortly.
>
> Jonathan Hung
>
>
> On Tue, Oct 22, 2019 at 1:39 AM Zhankun Tang  wrote:
>
> > Thanks for the effort, Jonathan!
> >
> > +1 (non-binding) on RC0.
> >  - Set up a single node cluster with the binary tarball
> >  - Run Spark Pi and pySpark job
> >
> > BR,
> > Zhankun
> >
> > On Tue, 22 Oct 2019 at 14:31, Konstantin Shvachko 
> > wrote:
> >
> >> +1 on RC0.
> >> - Verified signatures
> >> - Built from sources
> >> - Ran unit tests for new features
> >> - Checked artifacts on Nexus, made sure the sources are present.
> >>
> >> Thanks
> >> --Konstantin
> >>
> >>
> >> On Wed, Oct 16, 2019 at 6:01 PM Jonathan Hung 
> >> wrote:
> >>
> >> > Hi folks,
> >> >
> >> > This is the first release candidate for the first release of Apache
> >> Hadoop
> >> > 2.10 line. It contains 361 fixes/improvements since 2.9 [1]. It
> includes
> >> > features such as:
> >> >
> >> > - User-defined resource types
> >> > - Native GPU support as a schedulable resource type
> >> > - Consistent reads from standby node
> >> > - Namenode port based selective encryption
> >> > - Improvements related to rolling upgrade support from 2.x to 3.x
> >> >
> >> > The RC0 artifacts are at:
> >> http://home.apache.org/~jhung/hadoop-2.10.0-RC0/
> >> >
> >> > RC tag is release-2.10.0-RC0.
> >> >
> >> > The maven artifacts are hosted here:
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1241/
> >> >
> >> > My public key is available here:
> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >
> >> > The vote will run for 5 weekdays, until Wednesday, October 23 at 6:00
> pm
> >> > PDT.
> >> >
> >> > Thanks,
> >> > Jonathan Hung
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0)
> >> >
> >>
> >
>

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337784877
 
 

 ##
 File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/states/Node2PipelineMap.java
 ##
 @@ -71,6 +71,10 @@ public synchronized void addPipeline(Pipeline pipeline) {
   UUID dnId = details.getUuid();
   dn2ObjectMap.computeIfAbsent(dnId, k -> ConcurrentHashMap.newKeySet())
   .add(pipeline.getId());
+  dn2ObjectMap.computeIfPresent(dnId, (k, v) -> {
+v.add(pipeline.getId());
 
 Review comment:
   why do we need to add the dn->pipelineId in 75 since line 73 already add it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337785909
 
 

 ##
 File path: hadoop-hdds/common/src/main/resources/ozone-default.xml
 ##
 @@ -846,10 +846,17 @@
 
   
   
-ozone.scm.datanode.max.pipeline.engagement
-5
+  ozone.scm.datanode.max.pipeline.engagement
+  0
+  OZONE, SCM, PIPELINE
+  Max number of pipelines per datanode can be engaged in.
 
 Review comment:
   Can we add some description for the meaning of default 0? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337788038
 
 

 ##
 File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelinePlacementPolicy.java
 ##
 @@ -76,11 +80,46 @@ public PipelinePlacementPolicy(
* Returns true if this node meets the criteria.
*
* @param datanodeDetails DatanodeDetails
+   * @param nodesRequired nodes required count
* @return true if we have enough space.
*/
   @VisibleForTesting
-  boolean meetCriteria(DatanodeDetails datanodeDetails, long heavyNodeLimit) {
-return (nodeManager.getPipelinesCount(datanodeDetails) <= heavyNodeLimit);
+  boolean meetCriteria(DatanodeDetails datanodeDetails, int nodesRequired) {
+if (heavyNodeCriteria == 0) {
+  // no limit applied.
+  return true;
+}
+// Datanodes from pipeline in some states can also be considered available
+// for pipeline allocation. Thus the number of these pipeline shall be
+// deducted from total heaviness calculation.
+int pipelineNumDeductable = 0;
+Set pipelines = nodeManager.getPipelines(datanodeDetails);
+for (PipelineID pid : pipelines) {
+  Pipeline pipeline;
+  try {
+pipeline = stateManager.getPipeline(pid);
+  } catch (PipelineNotFoundException e) {
+LOG.error("Pipeline not found in pipeline state manager during" +
+" pipeline creation. PipelineID: " + pid +
+" exception: " + e.getMessage());
+continue;
 
 Review comment:
   This seems to be a case where the PipelineStateMap mismatch with the 
Node2PipelineMap. Do you see cases from unit tests or cluster?  


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #27: HDDS-2278. Run S3 test suite on OM HA cluster.

[GitBox]

bharatviswa504 commented on issue #27: HDDS-2278. Run S3 test suite on OM HA 
cluster.
URL: https://github.com/apache/hadoop-ozone/pull/27#issuecomment-545204726
 
 
   Newly added tests are passing, test failures are not related to this patch.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #27: HDDS-2278. Run S3 test suite on OM HA cluster.

[GitBox]

dineshchitlangia merged pull request #27: HDDS-2278. Run S3 test suite on OM HA 
cluster.
URL: https://github.com/apache/hadoop-ozone/pull/27
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia commented on issue #27: HDDS-2278. Run S3 test suite on OM HA cluster.

[GitBox]

dineshchitlangia commented on issue #27: HDDS-2278. Run S3 test suite on OM HA 
cluster.
URL: https://github.com/apache/hadoop-ozone/pull/27#issuecomment-545211362
 
 
   Thanks @elek and @adoroszlai for reviews. Thanks @bharatviswa504 for this 
contribution. Committed this change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.10.0 (RC0)

[Jonathan Hung]

Hi Eric, we've run some basic HDFS commands with a 3.2.1 namenode and
2.10.0 clients and datanodes. Everything worked as expected.

Jonathan Hung


On Tue, Oct 22, 2019 at 3:04 PM Eric Badger 
wrote:

> Hi Jonathan,
>
> Thanks for putting this RC together. You stated that there are
> improvements related to rolling upgrades from 2.x to 3.x and I know I have
> seen multiple JIRAs getting committed to that effect. Could you describe
> any tests that you have done to verify rolling upgrade compatibility
> for 3.x servers talking to 2.x clients and vice versa?
>
> Thanks,
>
> Eric
>
> On Tue, Oct 22, 2019 at 1:49 PM Jonathan Hung 
> wrote:
>
>> Thanks Konstantin and Zhankun. Unfortunately a feature slipped our radar
>> (HDFS-14667). Since this is the first of a minor release, we would like to
>> get it into 2.10.0.
>>
>> HDFS-14667 has been committed to branch-2.10.0, I will be rolling an RC1
>> shortly.
>>
>> Jonathan Hung
>>
>>
>> On Tue, Oct 22, 2019 at 1:39 AM Zhankun Tang  wrote:
>>
>> > Thanks for the effort, Jonathan!
>> >
>> > +1 (non-binding) on RC0.
>> >  - Set up a single node cluster with the binary tarball
>> >  - Run Spark Pi and pySpark job
>> >
>> > BR,
>> > Zhankun
>> >
>> > On Tue, 22 Oct 2019 at 14:31, Konstantin Shvachko > >
>> > wrote:
>> >
>> >> +1 on RC0.
>> >> - Verified signatures
>> >> - Built from sources
>> >> - Ran unit tests for new features
>> >> - Checked artifacts on Nexus, made sure the sources are present.
>> >>
>> >> Thanks
>> >> --Konstantin
>> >>
>> >>
>> >> On Wed, Oct 16, 2019 at 6:01 PM Jonathan Hung 
>> >> wrote:
>> >>
>> >> > Hi folks,
>> >> >
>> >> > This is the first release candidate for the first release of Apache
>> >> Hadoop
>> >> > 2.10 line. It contains 361 fixes/improvements since 2.9 [1]. It
>> includes
>> >> > features such as:
>> >> >
>> >> > - User-defined resource types
>> >> > - Native GPU support as a schedulable resource type
>> >> > - Consistent reads from standby node
>> >> > - Namenode port based selective encryption
>> >> > - Improvements related to rolling upgrade support from 2.x to 3.x
>> >> >
>> >> > The RC0 artifacts are at:
>> >> http://home.apache.org/~jhung/hadoop-2.10.0-RC0/
>> >> >
>> >> > RC tag is release-2.10.0-RC0.
>> >> >
>> >> > The maven artifacts are hosted here:
>> >> >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1241/
>> >> >
>> >> > My public key is available here:
>> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> >
>> >> > The vote will run for 5 weekdays, until Wednesday, October 23 at
>> 6:00 pm
>> >> > PDT.
>> >> >
>> >> > Thanks,
>> >> > Jonathan Hung
>> >> >
>> >> > [1]
>> >> >
>> >> >
>> >>
>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0)
>> >> >
>> >>
>> >
>>
>

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/483/

[Oct 22, 2019 6:46:25 PM] (jhung) HDFS-14667. Backport [HDFS-14403] "Cost-based 
FairCallQueue" to
[Oct 22, 2019 11:43:25 PM] (weichiu) HDFS-13901. INode access time is ignored 
because of race between open

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] chimney-lee opened a new pull request #74: HDDS-2348.Remove log4j properties for package org.apache.hadoop.ozone

[GitBox]

chimney-lee opened a new pull request #74: HDDS-2348.Remove log4j properties 
for package org.apache.hadoop.ozone
URL: https://github.com/apache/hadoop-ozone/pull/74
 
 
   ## What changes were proposed in this pull request?
   
   Remove log4j config for package org.apache.hadoop.ozone, as it cause the log 
in this package cannot be written to .log file . For example , the OM 
startup_msg was written to file om.out and ozone.log ; in addition, The OZONE 
and FILE appender is for development purpose.
   
   ## What is the link to the Apache JIRA
   
   
   https://issues.apache.org/jira/browse/HDDS-2348
   
   ## How was this patch tested?
   
   Restart OzoneManager , check the STARTUP_MSG in om.log
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337836283
 
 

 ##
 File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
 ##
 @@ -115,6 +114,12 @@ public SCMPipelineManager(Configuration conf, NodeManager 
nodeManager,
 "SCMPipelineManagerInfo", this);
 initializePipelineState();
 this.grpcTlsConfig = grpcTlsConfig;
+this.pipelineNumberLimit = conf.getInt(
+ScmConfigKeys.OZONE_SCM_PIPELINE_NUMBER_LIMIT,
+ScmConfigKeys.OZONE_SCM_PIPELINE_NUMBER_LIMIT_DEFAULT);
+this.heavyNodeCriteria = conf.getInt(
 
 Review comment:
   NIT: suggest rename heavyNodeCriteria to maxPipelinePerDatanode. 
heavyNodeCriteria can be a function name using maxPipelinePerDatanode.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia commented on issue #74: HDDS-2348.Remove log4j properties for package org.apache.hadoop.ozone

[GitBox]

dineshchitlangia commented on issue #74: HDDS-2348.Remove log4j properties for 
package org.apache.hadoop.ozone
URL: https://github.com/apache/hadoop-ozone/pull/74#issuecomment-545250164
 
 
   @chimney-lee Thank you for filing the jira & opening this PR. I see this is 
your first contribution to Ozone! Welcome!!
   
   You can refer this wiki for guidance on how to contribute to Ozone and other 
details: 
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone
   
   @anuengineer , @elek , @bharatviswa504  - I think it should be fine to avoid 
logging to ozone.log by default and for dev purposes, we can turn this on as 
needed. Your thoughts on this?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337836777
 
 

 ##
 File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
 ##
 @@ -147,10 +152,45 @@ private void initializePipelineState() throws 
IOException {
 }
   }
 
+  private boolean exceedPipelineNumberLimit(ReplicationFactor factor) {
+if (factor != ReplicationFactor.THREE) {
+  // Only put limits for Factor THREE pipelines.
+  return false;
+}
+// Per datanode limit
+if (heavyNodeCriteria > 0) {
+  return (stateManager.getPipelines(ReplicationType.RATIS, factor).size() -
 
 Review comment:
   Or should we consider only OPEN pipeline here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337837128
 
 

 ##
 File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
 ##
 @@ -147,10 +152,45 @@ private void initializePipelineState() throws 
IOException {
 }
   }
 
+  private boolean exceedPipelineNumberLimit(ReplicationFactor factor) {
+if (factor != ReplicationFactor.THREE) {
+  // Only put limits for Factor THREE pipelines.
+  return false;
+}
+// Per datanode limit
+if (heavyNodeCriteria > 0) {
+  return (stateManager.getPipelines(ReplicationType.RATIS, factor).size() -
+  stateManager.getPipelines(ReplicationType.RATIS, factor,
+  Pipeline.PipelineState.CLOSED).size()) > heavyNodeCriteria *
+  nodeManager.getNodeCount(HddsProtos.NodeState.HEALTHY) /
+  factor.getNumber();
+}
+
+// Global limit
+if (pipelineNumberLimit > 0) {
+  return (stateManager.getPipelines(ReplicationType.RATIS,
+  ReplicationFactor.THREE).size() - stateManager.getPipelines(
+  ReplicationType.RATIS, ReplicationFactor.THREE,
+  Pipeline.PipelineState.CLOSED).size()) >
+  (pipelineNumberLimit - stateManager.getPipelines(
 
 Review comment:
   Do we still assume one RATIS.ONE pipeline per DN here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337837808
 
 

 ##
 File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
 ##
 @@ -147,10 +152,45 @@ private void initializePipelineState() throws 
IOException {
 }
   }
 
+  private boolean exceedPipelineNumberLimit(ReplicationFactor factor) {
+if (factor != ReplicationFactor.THREE) {
+  // Only put limits for Factor THREE pipelines.
+  return false;
+}
+// Per datanode limit
+if (heavyNodeCriteria > 0) {
+  return (stateManager.getPipelines(ReplicationType.RATIS, factor).size() -
+  stateManager.getPipelines(ReplicationType.RATIS, factor,
+  Pipeline.PipelineState.CLOSED).size()) > heavyNodeCriteria *
+  nodeManager.getNodeCount(HddsProtos.NodeState.HEALTHY) /
+  factor.getNumber();
+}
+
+// Global limit
+if (pipelineNumberLimit > 0) {
+  return (stateManager.getPipelines(ReplicationType.RATIS,
+  ReplicationFactor.THREE).size() - stateManager.getPipelines(
+  ReplicationType.RATIS, ReplicationFactor.THREE,
+  Pipeline.PipelineState.CLOSED).size()) >
+  (pipelineNumberLimit - stateManager.getPipelines(
+  ReplicationType.RATIS, ReplicationFactor.ONE).size());
+}
+
+return false;
+  }
+
   @Override
   public synchronized Pipeline createPipeline(
   ReplicationType type, ReplicationFactor factor) throws IOException {
 lock.writeLock().lock();
+if (type == ReplicationType.RATIS && exceedPipelineNumberLimit(factor)) {
 
 Review comment:
   Should we move this logic into RatisPipelineProvider so that the exception 
handling will natually fit into the try/final?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #72: HDDS-2341. Validate tar entry path during extraction

[GitBox]

dineshchitlangia commented on a change in pull request #72: HDDS-2341. Validate 
tar entry path during extraction
URL: https://github.com/apache/hadoop-ozone/pull/72#discussion_r337838377
 
 

 ##
 File path: 
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/TarContainerPacker.java
 ##
 @@ -114,12 +116,11 @@
 }
   }
 
+  @SuppressFBWarnings("NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE")
 
 Review comment:
   Can we pls add minimal javadoc to explain why we are suppressing this 
warning? This will ensure that in future anyone changing this code path, is 
cautious.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #71: HDDS-2344. Add immutable entries in to the DoubleBuffer for Volume requests.

[GitBox]

dineshchitlangia commented on a change in pull request #71: HDDS-2344. Add 
immutable entries in to the DoubleBuffer for Volume requests.
URL: https://github.com/apache/hadoop-ozone/pull/71#discussion_r337841329
 
 

 ##
 File path: 
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmOzoneAclMap.java
 ##
 @@ -298,4 +303,27 @@ public static OmOzoneAclMap ozoneAclGetFromProtobuf(
   public List getDefaultAclList() {
 return defaultAclList;
   }
+
+  @Override
+  public Object clone() {
+ArrayList> accessMap = new ArrayList<>();
+
+// Initialize.
+for (OzoneAclType aclType : OzoneAclType.values()) {
+  accessMap.add(aclType.ordinal(), new HashMap<>());
+}
+
+// Add from original accessAclMap to accessMap.
+for (OzoneAclType aclType : OzoneAclType.values()) {
+  final int ordinal = aclType.ordinal();
+  accessAclMap.get(ordinal).forEach((k, v) ->
+  accessMap.get(ordinal).put(k, (BitSet) v.clone()));
+}
+
+// We can do shallow copy here, as OzoneAclInfo is immutable structure.
+ArrayList defaultList = new ArrayList<>();
+defaultList.addAll(defaultAclList);
+
+return new OmOzoneAclMap(defaultList, accessMap);
+  }
 
 Review comment:
   OmOzoneAclMap.clone() does not call super.clone(). Thus the correct object 
type is not guaranteed here and violates the contract for clone(). Most likely 
this is the findbug issue listed in CI run. To fix this, you must call the 
super.clone()


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #71: HDDS-2344. Add immutable entries in to the DoubleBuffer for Volume requests.

[GitBox]

dineshchitlangia commented on a change in pull request #71: HDDS-2344. Add 
immutable entries in to the DoubleBuffer for Volume requests.
URL: https://github.com/apache/hadoop-ozone/pull/71#discussion_r337841711
 
 

 ##
 File path: 
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmVolumeArgs.java
 ##
 @@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.helpers;
+
+import java.util.Collections;
+
+import org.apache.hadoop.ozone.OzoneAcl;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.Time;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.ozone.OzoneAcl.AclScope.ACCESS;
+
+/**
+ *
+ */
+public class TestOmVolumeArgs {
 
 Review comment:
   Empty javadoc, let's use it or scrap it :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #70: HDDS-1643. Send hostName also part of OMRequest.

[GitBox]

dineshchitlangia commented on a change in pull request #70: HDDS-1643. Send 
hostName also part of OMRequest.
URL: https://github.com/apache/hadoop-ozone/pull/70#discussion_r337843721
 
 

 ##
 File path: 
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestOMClientRequestWithUserInfo.java
 ##
 @@ -108,12 +108,14 @@ public void testUserInfo() throws Exception {
 
 InetAddress remoteAddress = omBucketCreateRequest.getRemoteAddress();
 UserGroupInformation ugi = omBucketCreateRequest.createUGI();
+String hostName = omBucketCreateRequest.getHostName();
 
 
-// Now check we have original user info and remote address or not.
-// Here from OMRequest user info, converted to UGI and InetAddress.
+// Now check we have original user info, remote address and hostname or 
not.
+// Here from OMRequest user info, converted to UGI, InetAddress and String.
 Assert.assertEquals(inetAddress.getHostAddress(),
 remoteAddress.getHostAddress());
 Assert.assertEquals(userGroupInformation.getUserName(), ugi.getUserName());
+Assert.assertEquals(inetAddress.getHostName(), hostName);
   }
 }
 
 Review comment:
   Missing new line at EOF


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #70: HDDS-1643. Send hostName also part of OMRequest.

[GitBox]

dineshchitlangia commented on a change in pull request #70: HDDS-1643. Send 
hostName also part of OMRequest.
URL: https://github.com/apache/hadoop-ozone/pull/70#discussion_r337843721
 
 

 ##
 File path: 
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestOMClientRequestWithUserInfo.java
 ##
 @@ -108,12 +108,14 @@ public void testUserInfo() throws Exception {
 
 InetAddress remoteAddress = omBucketCreateRequest.getRemoteAddress();
 UserGroupInformation ugi = omBucketCreateRequest.createUGI();
+String hostName = omBucketCreateRequest.getHostName();
 
 
-// Now check we have original user info and remote address or not.
-// Here from OMRequest user info, converted to UGI and InetAddress.
+// Now check we have original user info, remote address and hostname or 
not.
+// Here from OMRequest user info, converted to UGI, InetAddress and String.
 Assert.assertEquals(inetAddress.getHostAddress(),
 remoteAddress.getHostAddress());
 Assert.assertEquals(userGroupInformation.getUserName(), ugi.getUserName());
+Assert.assertEquals(inetAddress.getHostName(), hostName);
   }
 }
 
 Review comment:
   Missing new line at EOF


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] vivekratnavel commented on issue #23: HDDS-1868. Ozone pipelines should be marked as ready only after the leader election is complete.

[GitBox]

vivekratnavel commented on issue #23: HDDS-1868. Ozone pipelines should be 
marked as ready only after the leader election is complete.
URL: https://github.com/apache/hadoop-ozone/pull/23#issuecomment-545263168
 
 
   /retest


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337848761
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineCreateAndDestroy.java
 ##
 @@ -103,7 +108,9 @@ public void testPipelineCreationOnNodeRestart() throws 
Exception {
 } catch (IOException ioe) {
   // As now all datanodes are shutdown, they move to stale state, there
   // will be no sufficient datanodes to create the pipeline.
-  Assert.assertTrue(ioe instanceof InsufficientDatanodesException);
+  Assert.assertTrue(ioe instanceof SCMException);
+  Assert.assertTrue(((SCMException) ioe).getResult()
 
 Review comment:
   Can we use assertEquals with some message so that when it fails, we can see 
the difference in the ResultCode.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337849370
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
 ##
 @@ -253,10 +256,8 @@ public void testPipelineCreationFailedMetric() throws 
Exception {
   pipelineManager.createPipeline(HddsProtos.ReplicationType.RATIS,
   HddsProtos.ReplicationFactor.THREE);
   Assert.fail();
-} catch (InsufficientDatanodesException idEx) {
-  Assert.assertEquals(
-  "Cannot create pipeline of factor 3 using 1 nodes.",
-  idEx.getMessage());
+} catch (SCMException idEx) {
+  // pipeline creation failed this time.
 
 Review comment:
   Do we have specific exception result code to check here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337852587
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeWithPipelineRules.java
 ##
 @@ -62,8 +63,11 @@ public void setup(int numDatanodes) throws Exception {
 true);
 conf.set(HddsConfigKeys.HDDS_SCM_WAIT_TIME_AFTER_SAFE_MODE_EXIT, "10s");
 conf.set(ScmConfigKeys.OZONE_SCM_PIPELINE_CREATION_INTERVAL, "10s");
+conf.setInt(OZONE_DATANODE_MAX_PIPELINE_ENGAGEMENT, 1000);
 
 Review comment:
   Is there a reason that we need to set a large per DN pipeline limit here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337853065
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java
 ##
 @@ -494,6 +494,9 @@ void initializeConfiguration() throws IOException {
   streamBufferMaxSize.get(), streamBufferSizeUnit.get());
   conf.setStorageSize(OzoneConfigKeys.OZONE_SCM_BLOCK_SIZE, 
blockSize.get(),
   streamBufferSizeUnit.get());
+  // MiniOzoneCluster should have global pipeline upper limit.
+  conf.setInt(ScmConfigKeys.OZONE_SCM_PIPELINE_NUMBER_LIMIT,
+  pipelineNumLimit == 3 ? 2 * numOfDatanodes : pipelineNumLimit);
 
 Review comment:
   Can you add more comments here? If the # of SCM pipeline limit is 3, should 
we set OZONE_SCM_PIPELINE_NUMBER_LIMIT to 3 instead of 2 * numOfDatanodes?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337853227
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerReplicationEndToEnd.java
 ##
 @@ -54,8 +54,7 @@
 import java.util.function.Predicate;
 
 import static 
org.apache.hadoop.hdds.HddsConfigKeys.HDDS_CONTAINER_REPORT_INTERVAL;
-import static 
org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_SCM_PIPELINE_DESTROY_TIMEOUT;
-import static 
org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_SCM_STALENODE_INTERVAL;
+import static org.apache.hadoop.hdds.scm.ScmConfigKeys.*;
 
 Review comment:
   NIT: can we expand the wildcard import?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337853535
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
 ##
 @@ -53,8 +53,7 @@
 import java.util.concurrent.TimeoutException;
 
 import static java.nio.charset.StandardCharsets.UTF_8;
-import static 
org.apache.hadoop.hdds.scm.ScmConfigKeys.HDDS_SCM_WATCHER_TIMEOUT;
-import static 
org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_SCM_STALENODE_INTERVAL;
+import static org.apache.hadoop.hdds.scm.ScmConfigKeys.*;
 
 Review comment:
   NIT: expand wildcard import


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337853755
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerByPipeline.java
 ##
 @@ -73,8 +75,11 @@
   public static void init() throws Exception {
 conf = new OzoneConfiguration();
 conf.set(ScmConfigKeys.OZONE_SCM_PIPELINE_OWNER_CONTAINER_COUNT, "1");
+conf.setInt(OZONE_DATANODE_MAX_PIPELINE_ENGAGEMENT, 2);
 
 Review comment:
   Should we expose a method for MiniOzoneCluster to set per DN pipeline limit?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #71: HDDS-2344. Add immutable entries in to the DoubleBuffer for Volume requests.

[GitBox]

bharatviswa504 commented on a change in pull request #71: HDDS-2344. Add 
immutable entries in to the DoubleBuffer for Volume requests.
URL: https://github.com/apache/hadoop-ozone/pull/71#discussion_r337861523
 
 

 ##
 File path: 
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmOzoneAclMap.java
 ##
 @@ -298,4 +303,27 @@ public static OmOzoneAclMap ozoneAclGetFromProtobuf(
   public List getDefaultAclList() {
 return defaultAclList;
   }
+
+  @Override
+  public Object clone() {
+ArrayList> accessMap = new ArrayList<>();
+
+// Initialize.
+for (OzoneAclType aclType : OzoneAclType.values()) {
+  accessMap.add(aclType.ordinal(), new HashMap<>());
+}
+
+// Add from original accessAclMap to accessMap.
+for (OzoneAclType aclType : OzoneAclType.values()) {
+  final int ordinal = aclType.ordinal();
+  accessAclMap.get(ordinal).forEach((k, v) ->
+  accessMap.get(ordinal).put(k, (BitSet) v.clone()));
+}
+
+// We can do shallow copy here, as OzoneAclInfo is immutable structure.
+ArrayList defaultList = new ArrayList<>();
+defaultList.addAll(defaultAclList);
+
+return new OmOzoneAclMap(defaultList, accessMap);
+  }
 
 Review comment:
   I thought it is the final class and did like that. I think this should be 
marked as final like other classes like OMVolumeArgs. (As these classes 
represent proto structure data). And also as for few parameters, I need a deep 
copy, so I changed like this. Let me know what do you think.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #71: HDDS-2344. Add immutable entries in to the DoubleBuffer for Volume requests.

[GitBox]

bharatviswa504 commented on a change in pull request #71: HDDS-2344. Add 
immutable entries in to the DoubleBuffer for Volume requests.
URL: https://github.com/apache/hadoop-ozone/pull/71#discussion_r337861558
 
 

 ##
 File path: 
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmVolumeArgs.java
 ##
 @@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.helpers;
+
+import java.util.Collections;
+
+import org.apache.hadoop.ozone.OzoneAcl;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.Time;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.ozone.OzoneAcl.AclScope.ACCESS;
+
+/**
+ *
+ */
+public class TestOmVolumeArgs {
 
 Review comment:
   Updated it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #71: HDDS-2344. Add immutable entries in to the DoubleBuffer for Volume requests.

[GitBox]

bharatviswa504 commented on a change in pull request #71: HDDS-2344. Add 
immutable entries in to the DoubleBuffer for Volume requests.
URL: https://github.com/apache/hadoop-ozone/pull/71#discussion_r337861523
 
 

 ##
 File path: 
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmOzoneAclMap.java
 ##
 @@ -298,4 +303,27 @@ public static OmOzoneAclMap ozoneAclGetFromProtobuf(
   public List getDefaultAclList() {
 return defaultAclList;
   }
+
+  @Override
+  public Object clone() {
+ArrayList> accessMap = new ArrayList<>();
+
+// Initialize.
+for (OzoneAclType aclType : OzoneAclType.values()) {
+  accessMap.add(aclType.ordinal(), new HashMap<>());
+}
+
+// Add from original accessAclMap to accessMap.
+for (OzoneAclType aclType : OzoneAclType.values()) {
+  final int ordinal = aclType.ordinal();
+  accessAclMap.get(ordinal).forEach((k, v) ->
+  accessMap.get(ordinal).put(k, (BitSet) v.clone()));
+}
+
+// We can do shallow copy here, as OzoneAclInfo is immutable structure.
+ArrayList defaultList = new ArrayList<>();
+defaultList.addAll(defaultAclList);
+
+return new OmOzoneAclMap(defaultList, accessMap);
+  }
 
 Review comment:
   I thought it is the final class and did like that. I think this should be 
marked as final like other classes like OMVolumeArgs. (As these classes 
represent proto structure data). And also as for few parameters, I need a deep 
copy, so I changed like this(And also I feel, if I use super.clone it will make 
shallow copy, and then I need to change few fields, so I think it will be 
double work in cloning). Let me know what do you think.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

xiaoyuyao commented on a change in pull request #28: HDDS-1569 Support creating 
multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337865795
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneCluster.java
 ##
 @@ -352,6 +353,16 @@ public Builder setNumDatanodes(int val) {
   return this;
 }
 
+/**
+ * Sets the total number of pipelines to create.
+ * @param val number of pipelines
+ * @return MiniOzoneCluster.Builder
+ */
+public Builder setPipelineNumLimit(int val) {
 
 Review comment:
   NIT: rename to setTotalPipelineNumLimit


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

timmylicheng commented on a change in pull request #28: HDDS-1569 Support 
creating multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337866903
 
 

 ##
 File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/states/Node2PipelineMap.java
 ##
 @@ -71,6 +71,10 @@ public synchronized void addPipeline(Pipeline pipeline) {
   UUID dnId = details.getUuid();
   dn2ObjectMap.computeIfAbsent(dnId, k -> ConcurrentHashMap.newKeySet())
   .add(pipeline.getId());
+  dn2ObjectMap.computeIfPresent(dnId, (k, v) -> {
+v.add(pipeline.getId());
 
 Review comment:
   Line 73 is adding v to new keyset when key is absent and line 75 is adding v 
to the existing keyset. Reason why we don't have to have extra if here is 
computeIfAbsent and computeIfPresent are both judging if key is present or not. 
So we don't have to see if Map.get(k) is null.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

timmylicheng commented on a change in pull request #28: HDDS-1569 Support 
creating multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337867554
 
 

 ##
 File path: hadoop-hdds/common/src/main/resources/ozone-default.xml
 ##
 @@ -846,10 +846,17 @@
 
   
   
-ozone.scm.datanode.max.pipeline.engagement
-5
+  ozone.scm.datanode.max.pipeline.engagement
+  0
+  OZONE, SCM, PIPELINE
+  Max number of pipelines per datanode can be engaged in.
 
 Review comment:
   Sure.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

timmylicheng commented on a change in pull request #28: HDDS-1569 Support 
creating multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337867801
 
 

 ##
 File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
 ##
 @@ -115,6 +114,12 @@ public SCMPipelineManager(Configuration conf, NodeManager 
nodeManager,
 "SCMPipelineManagerInfo", this);
 initializePipelineState();
 this.grpcTlsConfig = grpcTlsConfig;
+this.pipelineNumberLimit = conf.getInt(
+ScmConfigKeys.OZONE_SCM_PIPELINE_NUMBER_LIMIT,
+ScmConfigKeys.OZONE_SCM_PIPELINE_NUMBER_LIMIT_DEFAULT);
+this.heavyNodeCriteria = conf.getInt(
 
 Review comment:
   Sure


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #28: HDDS-1569 Support creating multiple pipelines with same datanode

[GitBox]

timmylicheng commented on a change in pull request #28: HDDS-1569 Support 
creating multiple pipelines with same datanode
URL: https://github.com/apache/hadoop-ozone/pull/28#discussion_r337868451
 
 

 ##
 File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
 ##
 @@ -147,10 +152,45 @@ private void initializePipelineState() throws 
IOException {
 }
   }
 
+  private boolean exceedPipelineNumberLimit(ReplicationFactor factor) {
+if (factor != ReplicationFactor.THREE) {
+  // Only put limits for Factor THREE pipelines.
+  return false;
+}
+// Per datanode limit
+if (heavyNodeCriteria > 0) {
+  return (stateManager.getPipelines(ReplicationType.RATIS, factor).size() -
+  stateManager.getPipelines(ReplicationType.RATIS, factor,
+  Pipeline.PipelineState.CLOSED).size()) > heavyNodeCriteria *
+  nodeManager.getNodeCount(HddsProtos.NodeState.HEALTHY) /
+  factor.getNumber();
+}
+
+// Global limit
+if (pipelineNumberLimit > 0) {
+  return (stateManager.getPipelines(ReplicationType.RATIS,
+  ReplicationFactor.THREE).size() - stateManager.getPipelines(
+  ReplicationType.RATIS, ReplicationFactor.THREE,
+  Pipeline.PipelineState.CLOSED).size()) >
+  (pipelineNumberLimit - stateManager.getPipelines(
 
 Review comment:
   Yes. The RATIS ONE pipeline is still using old fashion of picking up nodes. 
Only RATIS THREE pipeline is using the complex new fashion in 
PipelinePlacementPolicy.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org