Andrey Yarovoy created HDDS-17716:
-------------------------------------
Summary: Avoid UserGroupInformation.getCurrentUser() per read
request in XceiverClientManager.getPipelineCacheKey
Key: HDDS-17716
URL: https://issues.apache.org/jira/browse/HDDS-17716
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Andrey Yarovoy
Assignee: Andrey Yarovoy
*Description*
Every datanode read acquires an `XceiverClientSpi` through
`XceiverClientManager.acquireClient`. Callers include:
- `ChunkInputStream.acquireClient`;
- `BlockInputStream.acquireClient`;
- `StreamBlockInputStream.acquireGrpcClientForReadData`, called on **every**
positioned read;
- `releaseClient(..., invalidate=true)`.
`acquireClient` builds the cache key with `getPipelineCacheKey`. In secure mode
that appends the user:
{code:java}
if (isSecurityEnabled()) {
key.append('|').append(UserGroupInformation.getCurrentUser().getShortUserName());
}
{code}
`UserGroupInformation.getCurrentUser()` calls `AccessController.getContext()` →
`Subject.getSubject`, which walks the thread stack in the JVM
(`JVM_GetStackAccessControlContext`). HBase calls Ozone from deep RPC-handler
stacks, so the walk is long. The key is also built inside
`synchronized (clientCache)`, a monitor shared by every reader in the process,
so the walk extends that critical section.
### Evidence (HBase YCSB on EC, RegionServer async-profiler JFR, 6×60s per arm)
| | stream read ON | stream read OFF |
|---|---|---|
| `getPipelineCacheKey` → `UGI.getCurrentUser`, % RS CPU | 2.5% | 3.4% |
| share of the EC read path CPU | ~26% (18% pread acquire + 8.5% initialize) |
~25% |
| wall: `acquireClient`, % busy handler time | 0.9% | 1.3% |
`JVM_GetStackAccessControlContext` is among the top self frames (1.1–1.4% of
all RS CPU). At this load the monitor is not contended; at higher reader
concurrency the stack walk inside `synchronized (clientCache)` becomes a convoy.
*Proposed fix*
Resolve the user **once per input stream**, not once per request, and pass it
to the factory:
1. Capture `UserGroupInformation.getCurrentUser().getShortUserName()` (secure
mode only) when the stream is created
(`KeyInputStream` / `BlockInputStreamFactoryImpl`). The stream is opened in the
caller's security context, which is the context that should own its datanode
clients.
2. Add `XceiverClientFactory.acquireClientForReadData(Pipeline, boolean
allowShortCircuit, String user)` (default method delegates to the existing
one), and the matching release. `getPipelineCacheKey` uses the supplied user
when non-null.
3. Compute the key **before** entering `synchronized (clientCache)`; only the
cache get/refcount needs the lock.
Write paths (`BlockOutputStream`) already acquire their client once per block,
so they need no change.
### Rejected alternatives
- **Cache the user per `XceiverClientManager`.** Wrong: the key carries the
user precisely so different users (`doAs`) never share a client.
- **ThreadLocal of the last user.** Wrong under `doAs` on pooled threads.
## Compatibility / side effects
- **Semantics change:** the datanode client is chosen by the user that
**opened** the stream, not the user active at each `read` call. For a stream
shared across `doAs` contexts this differs. Today such sharing silently mixes
users too; tokens are per request and checked on the datanode.
- No wire change. New default method on `XceiverClientFactory`.
## Tests
- Unit: `TestXceiverClientManager` — the same pipeline with two users gives two
clients; the per-stream user is honored; the key is computed without holding
the cache lock (verify with a lock-held assertion hook).
- Unit (Mockito): `getCurrentUser` is invoked once per stream, not once per
read (static mock of `UserGroupInformation`).
- Integration: secure mini-cluster read tests (`TestSecureOzoneRpcClient`, EC
read tests).
- Benchmark: `BlockInputStream`/`ECBlockInputStream` pread loop (4 KB–64 KB
reads, 1/8/32 threads), CPU per read and throughput A/B, n=3 per arm.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
