There is no limitation on the DN side for this. I need to check the SCM
read path to ensure nodes which are DECOMMISSIONING or
ENTERING_MAINTENANCE
are still returned when OM requests the block locations. I agree this
is
important and we need to ensure these nodes can still be read
It would be better we could have the corresponding test case cover this
scenario. We can add this after the merge, : ).
At the moment no. I feel this is something we should control in
Replication
Manager rather than decommissioning. We already have seen issues with
RM
where too many in-flight replication commands are sent to the DNs,
which
cannot complete them in time, and then more get scheduled etc. Each DN
has
a replication limit, so I think we need to enhance RM to hold back the
commands until the DNs have capacity to service them. We may also want
to
give priority to under replicated containers due to a dead node rather
than
decommissioning containers etc.
Agree that we could do this enhancement in RM level.
That is certainly something that can be added, and I would see as one of
the "usability enhancements" I mentioned. What we can do is create a
new
epic Jira for "post branch merge enhancements" and start collecting
these
suggestions there?
Makes sense to me.
Thanks Stephen for the comments, I don't have further comments now.
Thanks,
Yiqun
On 2020/10/27, 5:35 PM, "Stephen O'Donnell" <sodonn...@cloudera.com.INVALID>
wrote:
External Email
Hi Yiqun,
Thanks for taking a look.
> Does the container data can be read by client side when container
node is
in DECOMMISSIONING/ DECOMMISSIONED state? If the container cannot be
accessed, it can lost containers in a short time when multiple nodes
be in
decommissioning.
There is no limitation on the DN side for this. I need to check the SCM
read path to ensure nodes which are DECOMMISSIONING or
ENTERING_MAINTENANCE
are still returned when OM requests the block locations. I agree this
is
important and we need to ensure these nodes can still be read.
> Do we have the rate limitation control for the node decommission?
At the moment no. I feel this is something we should control in
Replication
Manager rather than decommissioning. We already have seen issues with
RM
where too many in-flight replication commands are sent to the DNs,
which
cannot complete them in time, and then more get scheduled etc. Each DN
has
a replication limit, so I think we need to enhance RM to hold back the
commands until the DNs have capacity to service them. We may also want
to
give priority to under replicated containers due to a dead node rather
than
decommissioning containers etc.
> For above command usage, will we support input the node with given a
input node list file, that will be useful for admin users to use this
feature.
That is certainly something that can be added, and I would see as one
of
the "usability enhancements" I mentioned. What we can do is create a
new
epic Jira for "post branch merge enhancements" and start collecting
these
suggestions there?
Thanks,
Stephen.
On Tue, Oct 27, 2020 at 7:09 AM Lin, Yiqun <yiq...@ebay.com.invalid>
wrote:
> Hi Stephen,
>
> I haven't reviewed much of the decommission feature code but have a
look
> for the overview doc you attached.
>
> Just some questions and comments from me:
>
> * Does the container data can be read by client side when container
node
> is in DECOMMISSIONING/ DECOMMISSIONED state? If the container cannot
be
> accessed, it can lost containers in a short time when multiple nodes
be in
> decommissioning.
> * Do we have the rate limitation control for the node decommission?
Large
> number of nodes concurrently decommissioned, lots of closed
containers be
> in replication. And this can impact the performance of SCM I think.
>
> Minor suggestion:
> ozone admin datanode decommission <list of nodes to remove>
> ozone admin datanode maintenance <list of nodes to put to
maintenance >
> ozone admin datanode recommission <list of nodes to recommission>
>
> For above command usage, will we support input the node with given a
input
> node list file, that will be useful for admin users to use this
feature.
>
> Thanks,
> Yiqun
>
> On 2020/10/27, 2:09 AM, "Stephen O'Donnell" <sodonn...@cloudera.com
.INVALID>
> wrote:
>
> External Email
>
> Someone reported that the attachment did not come through -
perhaps the
> mailing strips out attachments?
>
> I have attached it to the HDDS-1880 jia - here is the direct
link:
>
>
>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2Fattachment%2F13014144%2FDecommission%2520and%2520Maintenance%2520Overview.pdf&data=04%7C01%7Cyiqlin%40ebay.com%7C1237e97778984f69cccb08d87a5ba8e0%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637393881378701411%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=y6D5uNvNeOoqlRQysvG%2B7akMzcEgNVwPGxcQFLffdSw%3D&reserved=0
>
> Thanks,
>
> Stephen.
>
> On Mon, Oct 26, 2020 at 5:47 PM Stephen O'Donnell <
> sodonn...@cloudera.com>
> wrote:
>
> > Hi All,
> >
> > I am pleased to announce the Datanode Decommission and
Maintenance
> feature
> > for Ozone -
>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHDDS-1880&data=04%7C01%7Cyiqlin%40ebay.com%7C1237e97778984f69cccb08d87a5ba8e0%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637393881378711404%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Tq%2B4pc8rV8Hovk0GdaI%2FW2SZo0Qd511%2BDdnSzcK%2FMwM%3D&reserved=0
> >
> > The feature is working in Integration tests and also via
> docker-compose.
> > There is still some work to improve monitoring and usability,
but I
> believe
> > the feature is now complete enough to merge into master and
continue
> > development there.
> >
> > I would like to use this thread to discuss the feature and
agree on
> > whether we can merge it into master. To help with the
discussion, I
> have
> > attached a short document describing the major changes.
> >
> > The decommission changes are all on the branch HDDS-1880-Decom.
> >
> > Please reply here with any questions and comments.
> >
> > Thanks,
> >
> > Stephen.
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ozone-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: ozone-dev-h...@hadoop.apache.org
>
>
