[
https://issues.apache.org/jira/browse/HDDS-12581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreeja updated HDDS-12581:
--------------------------
Description:
Log files should be parsed using multiple threads, where each thread processes
one file at a time. The number of threads should be dynamically adjusted (can
be given in the command). During the parsing process, batch updates should be
performed on the database. It should be ensured that only one thread performs
the database updates at a time while other threads are blocked during this
operation.
Populates two tables: DatanodeContainerLogTable(acts like a detail log history)
and ContainerLogTable(stores latest state and BCSID for each container-datanode
pair)
CLI command of the form:
{code:java}
ozone debug log container --db=<path to db> parse --path=<log_folder>
--thread-count=<n>{code}
was:
Log files should be parsed using multiple threads, where each thread processes
one file at a time. The number of threads should be dynamically adjusted (can
be given in the command). During the parsing process, batch updates should be
performed on the database. It should be ensured that only one thread performs
the database updates at a time while other threads are blocked during this
operation.
Also create a ozone debug CLI command for the same.
{code:java}
./ozone debug container container_log_parse --parse=<path to logs folder>
--thread-count=<num of threads>{code}
> Multi-threaded Log File Parsing with Batch Updates to DB
> --------------------------------------------------------
>
> Key: HDDS-12581
> URL: https://issues.apache.org/jira/browse/HDDS-12581
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Sreeja
> Assignee: Sreeja
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.1.0
>
>
> Log files should be parsed using multiple threads, where each thread
> processes one file at a time. The number of threads should be dynamically
> adjusted (can be given in the command). During the parsing process, batch
> updates should be performed on the database. It should be ensured that only
> one thread performs the database updates at a time while other threads are
> blocked during this operation.
> Populates two tables: DatanodeContainerLogTable(acts like a detail log
> history) and ContainerLogTable(stores latest state and BCSID for each
> container-datanode pair)
> CLI command of the form:
> {code:java}
> ozone debug log container --db=<path to db> parse --path=<log_folder>
> --thread-count=<n>{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
