[
https://issues.apache.org/jira/browse/HDDS-17183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-17183:
-----------------------------------
Description:
- *Problem:* Each ozone invocation often takes ~1–2s before useful output; no
systematic profiling has been done. Likely contributors: JVM startup, large
classpath from [.classpath
descriptors|https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh]
(ozone_assemble_classpath → ozone_java_exec), and Java init
([GenericCli|https://github.com/apache/ozone/blob/master/hadoop-hdds/cli-common/src/main/java/org/apache/hadoop/hdds/cli/GenericCli.java]
constructs OzoneConfiguration and picocli up front).
- *Context:* Hadoop 2 CLI felt acceptable; Hadoop 3 / Ozone inherited heavier
Hadoop-style shell + classpath model
([ozone|https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/shell/ozone/ozone]
→ artifact names like ozone-cli-shell).
- *Goal:* Measure end-to-end latency, identify dominant phases, then implement
and document mitigations.
- *Out of scope (initially):* OM/SCM RPC time for commands that talk to the
cluster (track separately if profiling shows it dominates for specific
commands).
*Related (link, do not duplicate):* HDDS-11825 — interactive CLI (JLine REPL).
Most HDDS-11825 sub-tasks are resolved; open items (e.g. stack traces) stay on
that ticket. This epic focuses on *cold-start / per-invocation overhead* and
*user-facing docs* for interactive mode.
h2. Alternatives and work tracks
These are *alternatives to GraalVM* called out in the epic so task 5 does not
look like an afterthought:
h2. Other solutions (captured in umbrella description)
These are *alternatives to GraalVM* called out in the umbrella so task 5 does
not look like an afterthought:
||Approach||Notes||
|*Interactive REPL*|Already shipped; amortizes one JVM over many commands
(HDDS-11825). Needs user docs (task 2).|
|*Batch {{--execute}}*|Same JVM, scripted commands
({{{}[Shell.java](hadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/Shell.java){}}}).|
|*CDS / AppCDS*|Ship shared archive with dist; lower class-load cost without
native compile.|
|*Classpath reduction*|Tighter {{*.classpath}} files; avoid loading
admin/repair jars for {{{}ozone sh{}}}.|
|*Lazy CLI initialization*|Defer config/UGI/subcommand trees until command
known.|
|*Client JVM tuning*|{{{}-XX:TieredStopAtLevel=1{}}}, {{{}-Xms{}}}/{{{}-Xmx{}}}
for CLI profile in {{{}OZONE_CLIENT_OPTS{}}}.|
|*Uber-jar / module path*|Fewer class-path entries (I/O during launch);
tradeoffs for Apache release layout.|
|*Long-lived sidecar*|Similar to REPL; usually rejected for ops complexity
unless REPL is insufficient.|
|*CRaC*|Checkpoint/restore for repeated CLI on same host; niche for datacenter
automation.|
h2. Planned tasks
Profile and benchmark Ozone CLI startup latency
Document interactive Ozone CLI for users
Evaluate GraalVM Native Image for Ozone CLI
POC: Native Image or alternate packaging for one CLI entrypoint
Evaluate JVM and classpath mitigations (non-native)
Operational guidance: faster CLI workflows
was:
- *Problem:* Each ozone invocation often takes ~1–2s before useful output; no
systematic profiling has been done. Likely contributors: JVM startup, large
classpath from [.classpath
descriptors|https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh]
(ozone_assemble_classpath → ozone_java_exec), and Java init
([GenericCli|https://github.com/apache/ozone/blob/master/hadoop-hdds/cli-common/src/main/java/org/apache/hadoop/hdds/cli/GenericCli.java]
constructs OzoneConfiguration and picocli up front).
- *Context:* Hadoop 2 CLI felt acceptable; Hadoop 3 / Ozone inherited heavier
Hadoop-style shell + classpath model
([ozone|https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/shell/ozone/ozone]
→ artifact names like ozone-cli-shell).
- *Goal:* Measure end-to-end latency, identify dominant phases, then implement
and document mitigations.
- *Out of scope (initially):* OM/SCM RPC time for commands that talk to the
cluster (track separately if profiling shows it dominates for specific
commands).
*Related (link, do not duplicate):*
[HDDS-11825|https://issues.apache.org/jira/browse/HDDS-11825] — interactive CLI
(JLine REPL). Most HDDS-11825 sub-tasks are resolved; open items (e.g. stack
traces) stay on that ticket. This epic focuses on *cold-start / per-invocation
overhead* and *user-facing docs* for interactive mode.
h2. Alternatives and work tracks
These are *alternatives to GraalVM* called out in the epic so task 5 does not
look like an afterthought:
| Approach | Notes |
|
h2. Planned tasks
Profile and benchmark Ozone CLI startup latency
Document interactive Ozone CLI for users
Evaluate GraalVM Native Image for Ozone CLI
POC: Native Image or alternate packaging for one CLI entrypoint
Evaluate JVM and classpath mitigations (non-native)
Operational guidance: faster CLI workflows
> Speed up Ozone CLI
> ------------------
>
> Key: HDDS-17183
> URL: https://issues.apache.org/jira/browse/HDDS-17183
> Project: Apache Ozone
> Issue Type: Epic
> Components: Ozone CLI
> Reporter: Wei-Chiu Chuang
> Priority: Major
> Labels: cli, performance
>
> - *Problem:* Each ozone invocation often takes ~1–2s before useful output; no
> systematic profiling has been done. Likely contributors: JVM startup, large
> classpath from [.classpath
> descriptors|https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh]
> (ozone_assemble_classpath → ozone_java_exec), and Java init
> ([GenericCli|https://github.com/apache/ozone/blob/master/hadoop-hdds/cli-common/src/main/java/org/apache/hadoop/hdds/cli/GenericCli.java]
> constructs OzoneConfiguration and picocli up front).
> - *Context:* Hadoop 2 CLI felt acceptable; Hadoop 3 / Ozone inherited
> heavier Hadoop-style shell + classpath model
> ([ozone|https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/shell/ozone/ozone]
> → artifact names like ozone-cli-shell).
> - *Goal:* Measure end-to-end latency, identify dominant phases, then
> implement and document mitigations.
> - *Out of scope (initially):* OM/SCM RPC time for commands that talk to the
> cluster (track separately if profiling shows it dominates for specific
> commands).
> *Related (link, do not duplicate):* HDDS-11825 — interactive CLI (JLine
> REPL). Most HDDS-11825 sub-tasks are resolved; open items (e.g. stack traces)
> stay on that ticket. This epic focuses on *cold-start / per-invocation
> overhead* and *user-facing docs* for interactive mode.
> h2. Alternatives and work tracks
> These are *alternatives to GraalVM* called out in the epic so task 5 does not
> look like an afterthought:
>
> h2. Other solutions (captured in umbrella description)
> These are *alternatives to GraalVM* called out in the umbrella so task 5 does
> not look like an afterthought:
> ||Approach||Notes||
> |*Interactive REPL*|Already shipped; amortizes one JVM over many commands
> (HDDS-11825). Needs user docs (task 2).|
> |*Batch {{--execute}}*|Same JVM, scripted commands
> ({{{}[Shell.java](hadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/Shell.java){}}}).|
> |*CDS / AppCDS*|Ship shared archive with dist; lower class-load cost without
> native compile.|
> |*Classpath reduction*|Tighter {{*.classpath}} files; avoid loading
> admin/repair jars for {{{}ozone sh{}}}.|
> |*Lazy CLI initialization*|Defer config/UGI/subcommand trees until command
> known.|
> |*Client JVM tuning*|{{{}-XX:TieredStopAtLevel=1{}}},
> {{{}-Xms{}}}/{{{}-Xmx{}}} for CLI profile in {{{}OZONE_CLIENT_OPTS{}}}.|
> |*Uber-jar / module path*|Fewer class-path entries (I/O during launch);
> tradeoffs for Apache release layout.|
> |*Long-lived sidecar*|Similar to REPL; usually rejected for ops complexity
> unless REPL is insufficient.|
> |*CRaC*|Checkpoint/restore for repeated CLI on same host; niche for
> datacenter automation.|
> h2. Planned tasks
> Profile and benchmark Ozone CLI startup latency
> Document interactive Ozone CLI for users
> Evaluate GraalVM Native Image for Ozone CLI
> POC: Native Image or alternate packaging for one CLI entrypoint
> Evaluate JVM and classpath mitigations (non-native)
> Operational guidance: faster CLI workflows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
