[
https://issues.apache.org/jira/browse/HDDS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-12979:
-------------------------------
Description:
Currently, RATIS/THREE write pipeline consists of a single Raft group of three
datanodes.
We found in the previous write tests that the sequential nature of Raft
consensus algorithm is a write bottleneck (instead of the I/O of the datanode
volumes) even for writes that are not related to each other (e.g. unrelated
writes for different containers in the same pipeline or different blocks for
the same container might interfere with each other). When we increased the
number of pipelines per datanode (ozone.scm.datanode.pipeline.limit), we saw
quite a significant increase of the overall write throughput.
We can think about increasing the granularity of the Ratis write pipeline. For
example, we can have such one write pipeline consists of multiple Raft groups
of mutually exclusive datanode volumes. This way, writes can be parallelized
across volumes and the overall throughput can be increased. Additionally, we
can also ensure volume isolation, that is, once a volume is chosen in an active
write pipeline, it will not be chosen for another write pipeline. Therefore,
there won't be issues where one write pipeline write interfere with another.
We can go even more granular where each open container is a Raft group and
closing a container means closing the Ratis group as well.
There are definitely overheads when increasing the number of Ratis group since
each Raft group carries some associated overhead (e.g. direct buffer allocation
for "raft.server.log.write.buffer.size"). We need to do some testing whether
the performance improvements is worth the overhead and overall complexity.
was:
Currently, RATIS/THREE write pipeline consists of a single Raft group of three
datanodes.
We found in the previous write tests that the sequential nature of Raft
consensus algorithm is a write bottleneck (instead of the I/O of the datanode
volumes) even for writes that are not related to each other (e.g. unrelated
writes for different blocks for the same container might interfere with each
other). When we increased the number of pipelines per datanode
(ozone.scm.datanode.pipeline.limit), we saw quite a significant increase of the
overall write throughput.
We can think about increasing the granularity of the Ratis write pipeline. For
example, we can have such one write pipeline consists of multiple Raft groups
of mutually exclusive datanode volumes. This way, writes can be parallelized
across volumes and the overall throughput can be increased. Additionally, we
can also ensure volume isolation, that is, once a volume is chosen in an active
write pipeline, it will not be chosen for another write pipeline. Therefore,
there won't be issues where one write pipeline write interfere with another.
We can go even more granular where each open container is a Raft group and
closing a container means closing the Ratis group as well.
There are definitely overheads when increasing the number of Ratis group since
each Raft group carries some associated overhead (e.g. direct buffer allocation
for "raft.server.log.write.buffer.size"). We need to do some testing whether
the performance improvements is worth the overhead and overall complexity.
> Increasing Ratis Write Pipeline Granularity
> -------------------------------------------
>
> Key: HDDS-12979
> URL: https://issues.apache.org/jira/browse/HDDS-12979
> Project: Apache Ozone
> Issue Type: Wish
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Currently, RATIS/THREE write pipeline consists of a single Raft group of
> three datanodes.
> We found in the previous write tests that the sequential nature of Raft
> consensus algorithm is a write bottleneck (instead of the I/O of the datanode
> volumes) even for writes that are not related to each other (e.g. unrelated
> writes for different containers in the same pipeline or different blocks for
> the same container might interfere with each other). When we increased the
> number of pipelines per datanode (ozone.scm.datanode.pipeline.limit), we saw
> quite a significant increase of the overall write throughput.
> We can think about increasing the granularity of the Ratis write pipeline.
> For example, we can have such one write pipeline consists of multiple Raft
> groups of mutually exclusive datanode volumes. This way, writes can be
> parallelized across volumes and the overall throughput can be increased.
> Additionally, we can also ensure volume isolation, that is, once a volume is
> chosen in an active write pipeline, it will not be chosen for another write
> pipeline. Therefore, there won't be issues where one write pipeline write
> interfere with another.
> We can go even more granular where each open container is a Raft group and
> closing a container means closing the Ratis group as well.
> There are definitely overheads when increasing the number of Ratis group
> since each Raft group carries some associated overhead (e.g. direct buffer
> allocation for "raft.server.log.write.buffer.size"). We need to do some
> testing whether the performance improvements is worth the overhead and
> overall complexity.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
