[
https://issues.apache.org/jira/browse/SOLR-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120885#comment-18120885
]
David Smiley commented on SOLR-18124:
-------------------------------------
🤖 Here's what the three PRs look like together in a trace. The replica was
killed ({{kill -9}}) with uncommitted docs in its tlog, then 5,000 docs (more
than PeerSync can cover) were indexed on the leader, and the replica was
restarted while updates kept trickling in. The result is a single 4.1s
{{recovery}} trace:
!recovery-trace.png|thumbnail!
Exported trace: [^recovery-trace.json]
h3. What it shows
* *{{recovery}}* (4.1s) — {{recovery.afterStartup=true}},
{{recovery.strategy=syncOrReplicate}}, {{recovery.outcome=recovered}},
{{recovery.retries=0}}
** Events, in order: {{leaderFound}}, {{bufferUpdates}}, {{peerSync}},
{{peerSyncDone}} (success=false), {{replicate}}, {{replay}}, {{replayDone}},
{{publishActive}}, {{recovered}}
** Requests to the leader nest as children: {{/admin/ping}}, prep-recovery,
PeerSync {{/get}}, commit on the leader
** *{{replication}}* (639ms) — {{fullCopy=true}}, {{bytesDownloaded}} ≈ 2.0MB,
{{leaderFiles=88}}, result "Fetching latest index is successful"; the 90
{{/replication}} file fetches are its children
** *{{updatelog.replay}}* (343ms) — {{state=APPLYING_BUFFERED}}, 14 adds of
updates buffered during recovery
*** *{{updatelog.replay.log}}* — the buffer tlog file and its size
Before these changes, each of those leader requests showed up as its own
unrelated trace, which is how the {{/admin/ping}} calls with no apparent origin
came up in the first place.
The tlog replay at startup, before recovery, shows up as a separate root
{{updatelog.replay}} trace ({{state=REPLAYING}}, 50 adds). When recovery comes
from {{ADDREPLICA}}, the {{recovery}} → {{replication}} spans nest under the
Collection API's {{AddReplicaCmd}} span.
PRs: #4961 (recovery), #4962 (tlog replay), #4963 (replication)
🤖 Generated with [Claude Code|https://claude.com/claude-code]
> Tracing: Recovery / Replication
> -------------------------------
>
> Key: SOLR-18124
> URL: https://issues.apache.org/jira/browse/SOLR-18124
> Project: Solr
> Issue Type: Improvement
> Components: tracing
> Reporter: David Smiley
> Priority: Minor
> Labels: newdev, pull-request-available
> Attachments: Tempo recover BOTTOM.png, Tempo recovery TOP
> Grafana.png, Trace-7d69d6-2026-09-29 16_01_37.json
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Recovery: deserves a span.
> Replication: deserves a span.
> UpdateLog: deserves a span.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
