This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/doris-skills.git
The following commit(s) were added to refs/heads/main by this push:
new 3151c5c feat: add doris-debug diagnostic skills (#3)
3151c5c is described below
commit 3151c5c54a3f35be79e7d18ae4b941146355b953
Author: ZhenchaoXu <[email protected]>
AuthorDate: Fri Jul 24 16:50:46 2026 +0800
feat: add doris-debug diagnostic skills (#3)
* feat: add doris-debug diagnostic skills
Add a comprehensive diagnostic skill suite for Apache Doris production
troubleshooting, all under a single skills/doris-debug/ directory.
Structure:
skills/doris-debug/
├── SKILL.md # Router with symptom-to-skill mapping
├── query/SKILL.md # Slow/hanging/timeout queries
├── import/SKILL.md # Stream/Broker/Routine Load
├── compaction/SKILL.md # -235 / version accumulation
├── node/SKILL.md # OOM / crash / memory leak
├── materialized-view/ # Sync MV / MTMV
├── tablet/ # Replica / clone / disk skew
├── deployment/ # FE/BE startup / ports
├── data-lake/ # Hive/Iceberg/Paimon/Glue
├── resource-isolation/ # Workload Group / spill
├── cloud/ # Storage-compute separation
└── shared/ # References, guides, case index
Each sub-skill contains SKILL.md with Cause tables, Metric Taxonomy,
decision trees, and code anchors. Includes 16 detailed case files
(Environment → Symptom → Investigation → Root Cause → Fix) and
45 production case patterns in shared/references/03-case-index.md.
Companion to existing doris-profile-reader, doris-best-practices, and
doris-architecture-advisor skills.
* docs: add doris-debug suite to README Skills table
---
README.md | 1 +
skills/doris-debug/SKILL.md | 60 +++
skills/doris-debug/cloud/SKILL.md | 109 +++++
.../references/case-001-warmup-empty-batch.md | 88 ++++
skills/doris-debug/compaction/SKILL.md | 60 +++
.../references/case-001-too-many-versions.md | 22 +
.../case-002-schema-change-base-score.md | 88 ++++
skills/doris-debug/data-lake/SKILL.md | 100 +++++
.../references/case-001-glue-pagination.md | 73 ++++
skills/doris-debug/deployment/SKILL.md | 98 +++++
skills/doris-debug/import/SKILL.md | 40 ++
.../references/case-001-group-commit-wal-pileup.md | 12 +
.../case-002-rpc-timeout-heavy-work-pool.md | 88 ++++
.../references/case-003-insert-job-fe-oom.md | 87 ++++
skills/doris-debug/materialized-view/SKILL.md | 78 ++++
.../references/case-001-mtmv-rewrite-miss.md | 35 ++
skills/doris-debug/node/SKILL.md | 86 ++++
.../node/references/case-001-be-oom-memtracker.md | 26 ++
.../node/references/case-002-filecache-lru-leak.md | 84 ++++
.../case-003-be-scanner-pileup-memory.md | 113 +++++
.../references/case-004-fe-bdbje-corruption.md | 121 ++++++
skills/doris-debug/query/SKILL.md | 87 ++++
.../references/case-001-exchange-waitfordata.md | 12 +
.../references/case-002-plan-time-iceberg-rpc.md | 100 +++++
.../references/case-003-exchange-e11-fanout.md | 139 ++++++
skills/doris-debug/resource-isolation/SKILL.md | 111 +++++
.../references/case-001-workload-policy-cme.md | 106 +++++
.../shared/guides/cascade-import-wal-versions.md | 13 +
.../guides/cascade-query-timeout-exchange.md | 13 +
.../shared/references/01-common-commands.md | 21 +
.../doris-debug/shared/references/02-source-map.md | 37 ++
.../shared/references/03-cir-case-index.md | 478 +++++++++++++++++++++
.../shared/references/04-metrics-guide.md | 137 ++++++
.../shared/references/05-config-quick-ref.md | 113 +++++
skills/doris-debug/tablet/SKILL.md | 100 +++++
35 files changed, 2936 insertions(+)
diff --git a/README.md b/README.md
index e70433d..94282f0 100644
--- a/README.md
+++ b/README.md
@@ -25,6 +25,7 @@ intentionally out of scope here; use your platform's
cluster-management console
|---|---|
| `doris-best-practices` | Apache Doris table design, sizing, and runtime
query investigation (37 rules, 7 use-case templates, 4 sizing guides) |
| `doris-architecture-advisor` | Workload-aware architecture design (8
decision rules, 10 worked industry examples) |
+| `doris-debug` | Production diagnostic suite: symptom router + 10 domain
skills (query, import, compaction, node, MV, tablet, deployment, data-lake,
resource-isolation, cloud), 16 case files, 45 case patterns |
## How to use
diff --git a/skills/doris-debug/SKILL.md b/skills/doris-debug/SKILL.md
new file mode 100644
index 0000000..30ecac9
--- /dev/null
+++ b/skills/doris-debug/SKILL.md
@@ -0,0 +1,60 @@
+---
+name: doris-debug
+description: >
+ Apache Doris production diagnostics router. Use when Apache Doris queries
are slow,
+ imports are failing or timing out, compaction is raising -235 errors, nodes
are OOM
+ or crashing, materialized views are not rewriting, or tablet/replica health
is
+ degraded. Routes to the appropriate doris-debug-* skill. Covers
shared-nothing and
+ cloud (storage-compute separation) deployments.
+version: 0.3.0
+---
+
+# Doris Debug Router
+
+## How to use
+
+1. Identify the symptom domain from the table below.
+2. Read `shared/references/01-common-commands.md` for diagnostic SQL and API
commands.
+3. Read `shared/references/02-source-map.md` to map symptoms to Apache Doris
source code locations.
+4. Open the target skill (`<domain>/SKILL.md`) and follow its cause table and
decision tree.
+5. Check `shared/references/03-case-index.md` for matching production case
patterns.
+
+## Skill routing
+
+| Symptom | Skill |
+|---------|-------|
+| Slow / timeout / hanging query, Profile analysis, Exchange WaitForData |
`query/SKILL.md` |
+| Stream Load / Broker Load / Routine Load failure, Group Commit WAL pile-up |
`import/SKILL.md` |
+| -235 / too many versions, compaction score high, compaction lag |
`compaction/SKILL.md` |
+| BE/FE OOM, crash, false Alive, memory leak | `node/SKILL.md` |
+| Sync MV miss, async MTMV refresh/rewrite issues |
`materialized-view/SKILL.md` |
+| Tablet replica missing, clone backlog, disk skew, tablet repair |
`tablet/SKILL.md` |
+| FE/BE startup failure, port conflict, priority_networks misconfig |
`deployment/SKILL.md` |
+| Hive/Iceberg/Paimon/Glue catalog query failure, metadata refresh, S3/HDFS
connectivity | `data-lake/SKILL.md` |
+| Workload Group queue starvation, CPU/memory isolation leaks, spill issues |
`resource-isolation/SKILL.md` |
+| Storage-compute separation, meta-service latency, file cache, warmup |
`cloud/SKILL.md` |
+
+## Shared references
+
+| Document | Content |
+|----------|---------|
+| `shared/references/01-common-commands.md` | Diagnostic SQL, HTTP API, and
CLI commands |
+| `shared/references/02-source-map.md` | Symptom → Doris source code mapping
(with version annotations) |
+| `shared/references/03-case-index.md` | 45 production case summaries across 8
domains |
+| `shared/references/04-metrics-guide.md` | Per-domain key metrics with
thresholds |
+| `shared/references/05-config-quick-ref.md` | Per-domain key configuration
parameters with tuning guidance |
+| `shared/guides/` | Cross-module cascade troubleshooting |
+
+## Companion skills
+
+- `doris-profile-reader`: Deep Profile analysis for query bottlenecks
(referenced by `query/SKILL.md` Cause C — BE compute bottleneck).
+- `doris-best-practices`: Preventive table design and configuration guidance.
+- `doris-architecture-advisor`: Workload-aware architecture decisions.
+
+## Methodology
+
+1. Client → FE → BE → Disk/Network top-down investigation
+2. Logs / Metrics / Profile / `SHOW PROC` evidence first
+3. Stop the bleeding (throttle, kill, `reset_rpc_channel`) before root cause
+4. Verify hypotheses against Doris source code (`02-source-map.md`)
+5. Match known case patterns (`03-case-index.md`) to avoid rediscovery
diff --git a/skills/doris-debug/cloud/SKILL.md
b/skills/doris-debug/cloud/SKILL.md
new file mode 100644
index 0000000..ec3364f
--- /dev/null
+++ b/skills/doris-debug/cloud/SKILL.md
@@ -0,0 +1,109 @@
+---
+name: doris-debug-cloud
+description: >
+ Use for Doris storage-compute separation (cloud mode) issues: meta-service
+ latency, cache miss storms, object store throughput, and shared-nothing
config
+ conflicts in compute groups.
+version: 0.2.0
+category: cloud
+---
+
+# Cloud (Storage-Compute Separation)
+
+## Before proceeding
+
+Confirm cloud mode. Shared-nothing cluster commands (disk rebalance, clone,
local tablet repair)
+are meaningless in cloud mode. Check:
+
+```sql
+SHOW FRONTENDS\G -- IsCloudMode field (Doris 3.0+)
+```
+
+Or inspect `fe.conf` for `cloud_unique_id` or `meta_service_endpoint`.
+
+## Causes
+
+| ID | Cause | Evidence | Source anchor |
+|----|-------|----------|---------------|
+| A | Meta-service latency | High P99 on `SHOW PROC` / DDL; meta-service RPC
spikes | `MetaServiceClient.java` |
+| B | Cache miss storm | `local_cache_hit_ratio` near 0; high S3 GET rate
post-ingest | `FileCache.cpp` |
+| C | Object store throughput saturation | S3 throttle errors (503 SlowDown);
`Throughput limit exceeded` | `S3FileSystem.cpp` |
+| D | Compute group imbalance | One compute group idle, another overloaded |
`ComputeGroupMgr.java` |
+| E | Wrong config for cloud mode | BE OOM from shared-nothing compaction
knobs applied in cloud | cloud vs local config divergence |
+
+## 10 min triage
+
+```sql
+-- Compute group status
+SHOW COMPUTE GROUPS\G
+
+-- File cache hit ratio (Doris 3.0+)
+SELECT * FROM information_schema.file_cache_stats;
+```
+
+```bash
+# BE cache metrics
+./scripts/doris-debug be-metrics --be http://$BE:8040 --grep
"file_cache|local_cache"
+
+# Object store errors in BE log
+./scripts/doris-debug log-grep be/log --query-id "$QID"
+grep -r "SlowDown\|503\|Throughput.*exceed" be/log/
+```
+
+## Cause A — Meta-service latency
+
+In cloud mode, `SHOW PROC` and DDL go through the meta-service layer, not
direct BDBJE. Latency spikes often come from:
+
+1. Meta-service RPC overload — check `meta_service_rpc_timeout_ms` in fe.conf
+2. `SHOW PROC '/cluster_health'` on a large cluster — walks every tablet via
meta-service
+
+```bash
+# Check meta-service health
+curl -s "http://$MS_HOST:$MS_PORT/api/health";
+```
+
+## Cause B — Cache miss storm
+
+After bulk ingest (INSERT INTO SELECT or Stream Load), the file cache on
compute BEs is cold. First query after ingest hits S3 directly — expect 3-10×
latency.
+
+```sql
+-- Pre-warm cache after bulk ingest
+SELECT COUNT(*) FROM new_table; -- full scan populates file cache
+```
+
+Tune cache capacity:
+```properties
+# be.conf — file cache in cloud mode
+file_cache_path =
[{"path":"/nvme0/file_cache","total_size":536870912000,"query_limit":107374182400}]
+file_cache_type = whole_file_cache # or sub_file_cache for column-level
+```
+
+## Cause C — Object store throttling
+
+S3 has per-prefix request rate limits (~3500 PUT/GET per second per prefix).
For high-concurrency cloud workloads:
+
+```properties
+# be.conf
+s3_max_connections = 256 # increase parallel connections
+s3_request_timeout_ms = 30000 # increase timeout
+```
+
+Consider `sub_file_cache` for cold tables with large Parquet files — reduces
S3 list/get calls.
+
+## Key differences from shared-nothing
+
+| Shared-nothing config | Cloud mode equivalent | Notes |
+|----------------------|----------------------|-------|
+| `storage_root_path` | `file_cache_path` | Local disk is cache only |
+| `SHOW PROC '/backends'` disk | File cache usage | Data durability is in
object store |
+| Clone / balance | N/A | No tablet migration |
+| `max_tablet_version_num` | Still relevant | Compaction runs on compute nodes
|
+| `compaction_*` knobs | Still relevant | Tune per compute group capacity |
+
+## Source
+
+- `fe/.../cloud/ComputeGroupMgr.java`
+- `fe/.../cloud/MetaServiceClient.java`
+- `be/src/io/fs/s3/S3FileSystem.cpp`
+- `be/src/io/cache/whole_file_cache.cpp`
+- `be/src/io/cache/sub_file_cache.cpp`
diff --git a/skills/doris-debug/cloud/references/case-001-warmup-empty-batch.md
b/skills/doris-debug/cloud/references/case-001-warmup-empty-batch.md
new file mode 100644
index 0000000..bb8b0e3
--- /dev/null
+++ b/skills/doris-debug/cloud/references/case-001-warmup-empty-batch.md
@@ -0,0 +1,88 @@
+---
+type: reference
+category: cloud
+keywords: [warmup, file_cache, cache miss, route snapshot, tablet mapping,
empty batch]
+---
+
+# Case-001: Warmup 后查询仍全量读 S3 — FE 下发空 Batch
+
+## Environment
+
+- Doris version: 4.0.4.9 (cloud)
+- Architecture: storage-compute separation, 3 BE in compute group `batch_data`
+
+## Symptom
+
+执行 `WARM UP TABLE ods.ods_event_ri` 后，`SHOW WARM UP JOB` 显示所有 91 个 job 都是
`FINISHED`、`AllBatch=0`，耗时仅秒级。但后续查询仍从 S3 全量读取，本地 file_cache 命中率为 0。
+
+用户反复执行 warmup 和 `force` warmup，结果相同。
+
+## Key evidence
+
+### FE 日志
+
+```
+send warm up request. job_id=..., batch_id=0, job_sizes=0, request_type=SET_JOB
+```
+共 273 次（91 个 job × 3 台 BE），全部 `job_sizes=0`。
+
+FE 计算了非零 warm-up size（每个 job 约 0.2~0.5GB），但下发给 BE 时 batch 为空。
+
+### BE 日志
+
+```
+SET_BATCH ... jobs size=0
+pending_job_size=0|finish_job_size=0
+CLEAR_JOB
+```
+
+每次都是 `jobs size=0` → 状态查询 pending=0 → 立即 `CLEAR_JOB`。
+
+BE 日志中没有出现
`download_segment_file`、`warm_up_cache_async`、`FileCacheBlockDownloader`
等实际下载/预热关键词。
+
+## Investigation
+
+### Step 1: 确认 warmup job 状态
+
+`SHOW WARM UP JOB` 显示 `FINISHED` + `AllBatch=0`。排除下载失败——根本没有触发下载。
+
+### Step 2: 代码路径核对（4.0.4.9）
+
+```
+CacheHotspotManager.warmUpNewClusterByTable()
+ 1. 按分区累加 partition.getDataSize(true)
+ → 打印非零 warm up size
+ 2. 将分区 tablet 与 CloudTabletRebalancer.getSnapshotTabletsInPrimaryByBeId()
+ 做交集
+ → 只有落在该 BE primary snapshot 里的 tablet 才加入 beToWarmUpTablets
+ 3. splitBatch() 对空 tablet 列表 → 空 batch
+ 4. CloudWarmUpJob.buildJobMetas() batch 为空 → 空 job_metas
+ 5. BE CloudBackendService::warm_up_tablets() 收到空 job_metas → 无 pending 任务
+ 6. FE 看到 pending=0 → 标为 FINISHED 并清理
+```
+
+### Step 3: 根因确认
+
+FE route/rebalancer snapshot 中没有目标分区的 tablet 映射到这 3 台 BE。
+可能原因：route snapshot 刚初始化、BE 替换/扩缩容后未稳定、或 snapshot 与当前实际查询路由不一致。
+
+## Root Cause
+
+不是 BE 下载失败，而是 FE 生成的 per-BE warm-up tablet batch 为空。原因是 cloud tablet
rebalancer snapshot 中没有目标表分区 tablet 到目标 BE 的映射。
+
+`force` 参数无法解决此问题（4.0.4.9 代码中 `force` 只绕过 cache capacity 检查，不修复空 tablet 映射）。
+
+## Fix
+
+- **验证 route snapshot**: 在重跑 warmup 前确认
`CloudTabletRebalancer.getSnapshotTabletsInPrimaryByBeId` 对目标 BE 返回非空且包含目标表格分区
tablet
+- **等待 route 稳定**: 如果是刚扩缩容，等 cloud tablet rebalancer 完成一轮稳定 route 后再跑 warmup
+- **用 FE/BE 日志验证**: 重跑时确认 `job_sizes > 0`，BE 日志 `jobs size >
0`、`pending_job_size` 曾经非 0
+- **代码改进**: 当 `warm up size > 0` 但所有 BE batch 为空时，不应返回 `FINISHED`，应报错/取消并提示
route snapshot 无目标 tablet
+
+## Key diagnostic actions
+
+1. `SHOW WARM UP JOB` 检查 `AllBatch` — 如果为 0，问题不在下载
+2. FE 日志搜索 `send warm up request` → 确认 `job_sizes` 是否为 0
+3. BE 日志搜索 `SET_BATCH` → 确认 `jobs size` 是否为 0
+4. BE 日志搜索 `download_segment_file` → 如果完全不存在，确认下载从未触发
+5. 检查 `CloudTabletRebalancer` snapshot 状态
diff --git a/skills/doris-debug/compaction/SKILL.md
b/skills/doris-debug/compaction/SKILL.md
new file mode 100644
index 0000000..068549f
--- /dev/null
+++ b/skills/doris-debug/compaction/SKILL.md
@@ -0,0 +1,60 @@
+---
+name: doris-debug-compaction
+description: >
+ Use for Doris -235 / too many versions and compaction lag. Error raised in
+ RowsetBuilder::check_tablet_version_count when version_count >
max_tablet_version_num
+ (default 2000) or meta serialize size limit.
+version: 0.2.0
+category: compaction
+---
+
+# Compaction / Versions
+
+## Code
+
+`be/src/storage/rowset_builder.cpp`:
+
+- `version_count > max_version_config` → `Status::Error<TOO_MANY_VERSION>(...)`
+- Suggests reduce load frequency or raise `max_tablet_version_num` /
+ `time_series_max_tablet_version_num` in `be.conf`
+- Near limit triggers cumulative compaction submit
+
+Defaults (`config.cpp`): `max_tablet_version_num=2000`,
`time_series_max_tablet_version_num=20000`.
+
+## Causes
+
+| ID | Cause | Evidence | Fix direction |
+|----|-------|----------|---------------|
+| A | Ingest rate > compaction throughput | version_count rising; compaction
score growing | Reduce ingest frequency or increase compaction threads |
+| B | Disk IO bottleneck | `iostat` shows disk util > 90% | Move WAL to
separate disk; add compaction disks |
+| C | max_tablet_version_num too low for workload | -235 on normal ingest
rates | Raise limit (temporary), then fix root cause |
+| D | Single-tablet hotspot | One tablet has 10× versions of peers | Partition
/ bucket key skew |
+
+## Actions
+
+1. Confirm with `./scripts/doris-debug log-grep be/log --pack versions`
+2. Check compaction throughput: `./scripts/doris-debug be-metrics --be
http://$BE:8040 --grep compaction`
+3. Check disk IO: `iostat -x 1`
+4. Slow ingest / enlarge group commit batches
+5. Raising `max_tablet_version_num` is a temporary valve — fix ingest vs
compaction balance
+
+```bash
+./scripts/doris-debug be-metrics --be http://$BE:8040 --grep compaction
+```
+
+## Compaction knobs
+
+```properties
+# be.conf
+max_tablet_version_num = 2000 # hard cap, raise only as stopgap
+time_series_max_tablet_version_num = 20000 # separate cap for time-series
tables
+max_cumulative_compaction_threads = -1 # -1 = auto based on CPU cores
+compaction_task_num_per_disk = 4 # per-disk concurrent compaction
tasks
+compaction_tablet_size_threshold = 107374182400 # 100GB — above this, skip
base compaction
+```
+
+## Source
+
+- `be/src/storage/rowset_builder.cpp` — `-235` error
+- `be/src/common/config.cpp` — all compaction config defaults
+- `be/src/olap/cumulative_compaction_policy.cpp` — score calculation
diff --git
a/skills/doris-debug/compaction/references/case-001-too-many-versions.md
b/skills/doris-debug/compaction/references/case-001-too-many-versions.md
new file mode 100644
index 0000000..effdb44
--- /dev/null
+++ b/skills/doris-debug/compaction/references/case-001-too-many-versions.md
@@ -0,0 +1,22 @@
+# Case: Cumulative compaction can't keep up with high-frequency ingest
+
+## Symptom
+
+`-235` (TOO_MANY_VERSION) on Stream Load / INSERT. `SHOW PROC
'/cluster_health/tablet_health'` shows many tablets near
`max_tablet_version_num` (default 2000). Compaction metrics show cumulative
compaction queued but score rising.
+
+## Root cause
+
+Ingest rate (version creation) > cumulative compaction throughput. This is NOT
a compaction speed problem — it's a compaction trigger / scheduling problem.
Compaction_score grows until tablet version_count hits the hard cap.
+
+## Fix direction
+
+1. Check `be/log/be.INFO` for `"reach version limit"` lines
+2. `log-grep --pack versions` to confirm affected tablets
+3. Raise `max_tablet_version_num` to **unblock writes immediately** (symptom
relief)
+4. Then fix the root cause:
+ - Reduce ingest frequency: increase group commit `interval_ms` /
`data_bytes`
+ - Increase compaction thread pool: `max_cumulative_compaction_threads` /
`compaction_task_num_per_disk`
+ - Check disk IO: `iostat -x 1` — if disk util > 90%, compaction can't catch
up regardless of thread count
+5. For time-series workloads, set `time_series_max_tablet_version_num`
separately
+
+Source: `rowset_builder.cpp` check_tablet_version_count,
`compaction_policy.cpp` cumulative score.
diff --git
a/skills/doris-debug/compaction/references/case-002-schema-change-base-score.md
b/skills/doris-debug/compaction/references/case-002-schema-change-base-score.md
new file mode 100644
index 0000000..fc85f91
--- /dev/null
+++
b/skills/doris-debug/compaction/references/case-002-schema-change-base-score.md
@@ -0,0 +1,88 @@
+---
+type: reference
+category: compaction
+keywords: [base compaction, schema change, SC residue, tablet state,
compaction score ghost]
+---
+
+# Case-002: Schema Change 残留导致 Base Compaction Score 虚高
+
+## Environment
+
+- Doris version: cloud 4.0+
+- Architecture: storage-compute separation
+
+## Symptom
+
+Grafana 监控显示某 BE 的 base compaction score 持续偏高：
+- base score: max 1490, avg 1370
+- cumu score: max 79, avg 32.5（正常）
+
+告警触发但集群无查询异常。用户怀疑 compaction 资源不足。
+
+## Key evidence
+
+```
+be/log:
+get_topn_compaction_score ... type=1 → score 高（base）
+get_topn_compaction_score ... type=2 → score 正常（cumu）
+```
+
+但 `get_topn_tablets_to_compact()` 实际提交的 compaction 任务数与 score 不成比例。
+
+Grafana 截图时间范围 2026-07-01~07-07，base score 缓慢上升而非突发。
+
+历史检查：同一集群有 Schema Change 操作记录。
+
+## Investigation
+
+### Step 1: 区分 Base vs Cumulative
+
+Doris compaction 枚举：
+```
+BASE_COMPACTION = 1 → base score
+CUMULATIVE_COMPACTION = 2 → cumu score
+```
+
+`get_topn_compaction_score ... type=1` 才是 base 分数，`type=2` 是 cumu。不能混着看。
+
+当前 base high / cumu normal → 是 base compaction 被阻塞，不是全局 compaction 资源不足。
+
+### Step 2: 代码核对
+
+cloud 调度调用 `get_topn_tablets_to_compact()` 计算最高分 tablet 并把值写入：
+```
+tablet_base_max_compaction_score → Grafana 取值源
+tablet_cumulative_max_compaction_score
+```
+
+但 score 更新 ≠ 实际 pick tablet 执行。调度器在 pick 前会检查：
+- tablet_state 是否允许 compaction
+- 是否有 slot 可用
+- SC（Schema Change）上下文是否允许
+- 其他过滤条件
+
+### Step 3: 过滤条件排查
+
+SC 操作在 tablet 上创建了 base compaction 候选（因为 SC 会生成新的 rowset 需要合并），但 SC
上下文未清理时，`tablet_state` 或 SC 状态会阻止 compaction 执行。
+
+结果：base score 持续计算并写入 metric → Grafana 告警，但实际 compaction 不执行 → score 得不到缓解。
+
+## Root Cause
+
+Schema Change 操作产生的残留 SC 上下文导致 base compaction 候选 tablet 无法被 compaction 调度器执行。
+
+这不是 "缺少 compaction 资源" 或 "cumu point 太多"，而是 "有候选但不允许执行"。
+
+## Fix
+
+1. 清理残留 SC 上下文（`SHOW ALTER TABLE` 确认 SC 状态 → `CANCEL ALTER` 清理）
+2. 检查 compaction 调度过滤逻辑中 SC 相关的条件判断
+3. 不要盲目增加 compaction 线程或磁盘并发——当前不是线程/IO 不足，是候选被过滤
+
+## Key diagnostic actions
+
+1. 区分 base(type=1) vs cumu(type=2) score（不同问题方向）
+2. `SHOW ALTER TABLE` 检查是否有 pending/failed SC 操作
+3. be/log 搜索 `get_topn_tablets_to_compact` → 确认过滤逻辑
+4. 如果 base score 高但找不到高分 tablet（C5 场景），检查 cloud 调度 slot/状态/过滤条件
+5. `SHOW PROC '/cluster_health/tablet_health'` 确认是否有 tablet 版本异常
diff --git a/skills/doris-debug/data-lake/SKILL.md
b/skills/doris-debug/data-lake/SKILL.md
new file mode 100644
index 0000000..38b9e53
--- /dev/null
+++ b/skills/doris-debug/data-lake/SKILL.md
@@ -0,0 +1,100 @@
+---
+name: doris-debug-data-lake
+description: >
+ Use for Doris external catalog issues: Hive/Iceberg/Paimon/Hudi query
failures,
+ metadata refresh, filesystem S3/HDFS connectivity, external MV rewrite
misses.
+version: 0.2.0
+category: data-lake
+---
+
+# Data Lake (Multi-Catalog)
+
+## Causes
+
+| ID | Cause | Evidence | Source anchor |
+|----|-------|----------|---------------|
+| A | Metadata stale | Query returns old data / missing partitions after
upstream write | `HMSClient.java`, `CachedHMSClient.java` |
+| B | S3/HDFS connectivity | `Access Denied`, `Timeout`, `NoSuchBucket` |
`S3FileSystem.java`, `HdfsResource.java` |
+| C | External MV rewrite miss | EXPLAIN shows no MATERIALIZED_REWRITE |
`MTMVService.java` rewrite rules |
+| D | Schema mismatch | Parquet/ORC schema ≠ Hive Metastore schema |
`ParquetReader.cpp`, schema merge |
+| E | Credential / IAM expiry | STS / IAM token rotated; 403 on S3 |
`S3FileSystem.java` credential refresh |
+| F | Too many open files | FD exhaustion listing large table directories | BE
`ulimit -n`, `FileSystem.listFiles()` |
+
+## 10 min triage
+
+```sql
+-- Verify catalog is alive
+SHOW CATALOGS;
+
+-- Refresh metadata (hive / iceberg / paimon)
+REFRESH CATALOG hive_catalog;
+REFRESH DATABASE hive_catalog.db_name;
+REFRESH TABLE hive_catalog.db_name.tbl_name;
+
+-- Check if external MV rewrite is enabled
+SET materialized_view_rewrite_enable_contain_external_table = true;
+
+-- Analyze external table for CBO stats
+ANALYZE TABLE hive_catalog.db_name.tbl_name;
+```
+
+```bash
+# BE logs for S3/HDFS errors
+./scripts/doris-debug log-grep be/log --query-id "$QID"
+grep -r "403\|Access Denied\|Timeout\|NoSuchBucket\|Token.*expired" be/log/
+
+# Check BE file descriptor limit
+cat /proc/$BE_PID/limits | grep "open files"
+```
+
+## Cause A — Metadata staleness
+
+Hive Metastore metadata is cached; `REFRESH CATALOG` invalidates the cache.
For large catalogs, prefer:
+
+```sql
+-- Refresh only the changed partition
+REFRESH TABLE catalog.db.tbl PARTITION (dt='2026-07-15');
+```
+
+If refresh is slow, check HMSClient cache settings (`fe.conf`):
+```properties
+hive_metastore_client_timeout_second = 10
+```
+
+## Cause B — S3/HDFS connectivity
+
+```properties
+# be.conf — S3 credential chain
+aws_access_key_id = ...
+aws_secret_access_key = ...
+aws_region = us-east-1
+```
+
+```bash
+# Test S3 reachability from BE host
+curl -I "https://<bucket>.s3.<region>.amazonaws.com"
+```
+
+```properties
+# fe.conf for HDFS catalog
+hadoop_conf_dir = /path/to/hadoop/conf # must contain core-site.xml +
hdfs-site.xml
+```
+
+## Cause E — Credential rotation
+
+For STS/AssumeRole-based S3 access, the cached credential may expire:
+```sql
+-- Force credential refresh (Doris 2.1+ / 3.0)
+REFRESH CATALOG hive_catalog;
+```
+
+For long-running queries against external tables, ensure credential TTL >
query timeout.
+
+## Source
+
+- `fe/.../datasource/hive/HMSExternalCatalog.java`
+- `fe/.../datasource/hive/HMSClient.java`
+- `be/src/io/fs/s3/S3FileSystem.cpp` — S3 connector
+- `be/src/io/fs/hdfs/HdfsFileSystem.cpp` — HDFS connector
+- `be/src/vec/exec/format/parquet/vparquet_reader.cpp` — parquet reader
+- `fe/.../mtmv/MTMVService.java` — external MV rewrite
diff --git
a/skills/doris-debug/data-lake/references/case-001-glue-pagination.md
b/skills/doris-debug/data-lake/references/case-001-glue-pagination.md
new file mode 100644
index 0000000..86ebc8d
--- /dev/null
+++ b/skills/doris-debug/data-lake/references/case-001-glue-pagination.md
@@ -0,0 +1,73 @@
+---
+type: reference
+category: data-lake
+keywords: [glue, catalog, pagination, NextToken, SHOW DATABASES, AWS]
+---
+
+# Case-001: Glue Catalog 只展示部分 Database — Pagination 丢失
+
+## Environment
+
+- Doris version: 26.0.3 (cloud)
+- Architecture: storage-compute separation, BYOC
+- Catalog type: AWS Glue
+
+## Symptom
+
+`SHOW DATABASES FROM glue_catalog` 只返回前几页的 database，大量 database 未显示。
+
+直接在 AWS Glue API 调用返回完整列表（100+ databases），但 Doris 只展示前 20 个左右。
+
+## Investigation
+
+### Step 1: 对比 Glue API vs Doris
+
+```bash
+# AWS Glue API 直接调用
+aws glue get-databases --region us-east-1 | jq '.DatabaseList | length'
+# 返回: 100+
+
+# Doris 查询
+mysql> SHOW DATABASES FROM glue_catalog;
+# 返回: 20 rows
+```
+
+### Step 2: 代码核对
+
+Glue `GetDatabases` API 的默认 page size 通常是 20 或 50，返回结果包含 `NextToken` 用于请求下一页。
+
+Doris `GlueCatalog.getAllDatabases()` 中：
+- 如果未正确处理 `NextToken`
+- 或仅调用一次 API 不进行 pagination 循环
+- 结果：只返回第一页的 database
+
+### Step 3: 确认 fe.log
+
+```
+fe/log/fe.log:
+grep -i "GlueCatalog" fe.log
+```
+
+如果有 `getAllDatabases` 相关日志，对比返回的 database 数量与 AWS API 实际数量。
+
+## Root Cause
+
+Doris Glue Catalog 的 `getAllDatabases()` 实现未正确处理 AWS Glue API 的 `NextToken`
pagination 机制，只获取到第一页数据。
+
+## Fix
+
+- 修复 `GlueCatalog.getAllDatabases()` 增加 pagination 循环：
+ ```java
+ do {
+ GetDatabasesResult result = glueClient.getDatabases(request);
+ databases.addAll(result.getDatabaseList());
+ request.setNextToken(result.getNextToken());
+ } while (request.getNextToken() != null);
+ ```
+
+## Key diagnostic actions
+
+1. 对比 AWS Glue API 直接调用结果 vs Doris `SHOW DATABASES` 返回
+2. fe.log 搜索 `GlueCatalog` 或 `getAllDatabases` 错误
+3. 确认 Glue API 返回的 `NextToken`
+4. 如果适用，同样检查 Hive Metastore / Iceberg REST catalog 的 pagination 处理
diff --git a/skills/doris-debug/deployment/SKILL.md
b/skills/doris-debug/deployment/SKILL.md
new file mode 100644
index 0000000..5c5f5a0
--- /dev/null
+++ b/skills/doris-debug/deployment/SKILL.md
@@ -0,0 +1,98 @@
+---
+name: doris-debug-deployment
+description: >
+ Use for Doris FE/BE startup failures, port conflicts, priority_networks
misrouting,
+ meta_dir corruption, and ADD/DROP BACKEND issues.
+version: 0.2.0
+category: deployment
+---
+
+# Deployment & Startup
+
+## Causes
+
+| ID | Cause | Evidence | Source anchor |
+|----|-------|----------|---------------|
+| A | Port conflict | BE/FE fail to bind; `Address already in use` |
`brpc/server.cpp`, `doris_main.cpp` |
+| B | `priority_networks` misconfig | brpc connection refused / wrong
interface | `config.cpp` |
+| C | meta_dir / storage_root_path not writable | `Permission denied` / `No
such file` at startup | `olap/olap_meta.cpp` |
+| D | FE metadata corruption | FE fails to start after crash; `meta/image.*`
errors | `Image.java`, `BDBJEJournal.java` |
+| E | Mixed version cluster | Protocol mismatch in thrift/brpc between FE-BE |
`be/src/agent/`, `heartbeat` |
+| F | DNS / hostname resolution | `UnknownHostException`, BE heartbeat to
wrong address | `Env.java` frontend host check |
+
+## 10 min triage
+
+```bash
+# FE status
+curl -s "http://$FE:8030/api/bootstrap";
+curl -s "http://$FE:8030/api/health";
+
+# BE status
+curl -s "http://$BE:8040/api/health";
+
+# Port check
+ss -tlnp | grep -E "8030|8040|8060|9030"
+```
+
+```sql
+-- Cluster membership
+SHOW FRONTENDS\G
+SHOW BACKENDS\G
+```
+
+## Cause A — Port conflict
+
+Default ports (from `fe/conf/fe.conf` and `be/conf/be.conf`):
+
+| Service | Port | Config key |
+|---------|------|------------|
+| FE HTTP | 8030 | `http_port` |
+| BE HTTP | 8040 | `webserver_port` |
+| BE brpc | 8060 | `brpc_port` |
+| FE MySQL | 9030 | `query_port` |
+| FE Edit log | 9010 | `edit_log_port` |
+
+```bash
+# Kill stale process on target port
+fuser -k 8060/tcp # BE brpc
+# Then restart BE
+```
+
+## Cause B — priority_networks
+
+The most common cause of "BE shows Alive=true but brpc fails."
`priority_networks` tells BE which network interface to bind brpc to.
+
+```properties
+# be.conf — match ONE subnet exactly
+priority_networks = 10.0.0.0/24
+```
+
+Check what interface BE actually bound:
+```bash
+curl -s "http://$BE:8040/api/health"; | grep -o '"ip":"[^"]*"'
+```
+
+Validation: the IP in `SHOW BACKENDS` must be reachable from every FE and
every other BE on port 8060.
+
+## Cause D — FE metadata recovery
+
+```bash
+# Check FE meta directory
+ls -la fe/doris-meta/image/
+ls -la fe/doris-meta/bdb/
+
+# If bdb is corrupt and this is a follower/observer:
+# 1. Remove the bad FE from the cluster
+# 2. Wipe doris-meta/
+# 3. Re-add as a new follower: ALTER SYSTEM ADD FOLLOWER "host:9010"
+```
+
+Never remove `doris-meta/` on the **last healthy Master** FE.
+
+## Source
+
+- `fe/conf/fe.conf` — port defaults
+- `be/src/common/config.cpp` — `priority_networks`
+- `be/src/service/doris_main.cpp` — BE startup
+- `fe/.../master/Env.java` — FE bootstrap
+- `fe/.../journal/bdbje/BDBJEJournal.java`
diff --git a/skills/doris-debug/import/SKILL.md
b/skills/doris-debug/import/SKILL.md
new file mode 100644
index 0000000..1eaaa10
--- /dev/null
+++ b/skills/doris-debug/import/SKILL.md
@@ -0,0 +1,40 @@
+---
+name: doris-debug-import
+description: >
+ Use for Doris Stream/Broker/Routine Load issues and Group Commit async_mode
WAL
+ pile-up. Tunables: table group_commit_interval_ms / group_commit_data_bytes;
+ BE group_commit_insert_threads, group_commit_wal_path,
group_commit_wal_max_disk_limit.
+version: 0.1.0
+category: import
+---
+
+# Import / Group Commit
+
+## async_mode WAL semantics
+
+Write WAL → RPC returns → background group commit → **delete_wal** on success
+(`group_commit_mgr.cpp`). Drain rate = commit finish rate ≠ compaction GB/s.
+
+## Knobs (config.cpp)
+
+| Config | Default | Role |
+|--------|---------|------|
+| `group_commit_insert_threads` | 10 | commit worker pool |
+| `group_commit_relay_wal_threads` | 10 | WAL replay |
+| `group_commit_wal_path` | under storage_root | WAL disks |
+| `group_commit_wal_max_disk_limit` | 10% | WAL quota |
+| table `group_commit_data_bytes` | often 64MB | size flush |
+| table `group_commit_interval_ms` | often 10000 | time flush |
+
+## High MB/s guidance
+
+1. Split load across BEs (WAL is local to coordinator BE).
+2. `wal-du` on each BE storage path.
+3. Prefer dedicated NVMe for `group_commit_wal_path`.
+4. A/B `group_commit_data_bytes` (128MB vs 512MB) — size threshold dominates
when ingest ≫ interval.
+5. `sync_mode` or client throttle for backpressure.
+
+```bash
+./scripts/doris-debug wal-du /path/to/be/storage
+./scripts/doris-debug log-grep be/log --pack group_commit
+```
diff --git
a/skills/doris-debug/import/references/case-001-group-commit-wal-pileup.md
b/skills/doris-debug/import/references/case-001-group-commit-wal-pileup.md
new file mode 100644
index 0000000..7b7f560
--- /dev/null
+++ b/skills/doris-debug/import/references/case-001-group-commit-wal-pileup.md
@@ -0,0 +1,12 @@
+# Case: Group Commit WAL pile-up under ~800MB/s Stream Load
+
+## Symptom
+async_mode WAL grows; drain lag. Props: interval_ms=2000, data_bytes=256MB.
Compaction throughput looks high.
+
+## Root cause
+Per-BE WAL ingest > group commit delete rate; size threshold fires ~0.3s;
compaction GB/s ≠ WAL drain.
+
+## Fix direction
+Spread BEs; dedicated WAL disk; tune data_bytes; `doris-debug wal-du`;
optional sync_mode backpressure.
+
+Source: `group_commit_mgr.cpp` delete_wal after successful commit.
diff --git
a/skills/doris-debug/import/references/case-002-rpc-timeout-heavy-work-pool.md
b/skills/doris-debug/import/references/case-002-rpc-timeout-heavy-work-pool.md
new file mode 100644
index 0000000..611c037
--- /dev/null
+++
b/skills/doris-debug/import/references/case-002-rpc-timeout-heavy-work-pool.md
@@ -0,0 +1,88 @@
+---
+type: reference
+category: import
+keywords: [tablet writer, RPC timeout, heavy work pool, brpc_heavy, import
stuck]
+---
+
+# Case-002: 导入 RPC Timed Out — BE Heavy Work Pool 打满
+
+## Environment
+
+- Doris version: 4.0.9 (cloud)
+- Architecture: storage-compute separation
+
+## Symptom
+
+导入偶发失败，日志显示：
+```
+VNodeChannel[...] node=172.31.36.239:8061 open failed ...
+RPC call is timed out ... Reached timeout=60000ms
+```
+随后 coordinator cancel 整条 load。后续日志：
+```
+failed to prepare rowset: txn is not in 1 state, txn_status=4
+```
+重试后成功，说明是瞬时拥塞而非数据问题。
+
+## Key evidence
+
+```
+2026-07-08 00:58:33.541 开始: query_id=8456cc28099544e9, query type: LOAD
+2026-07-08 00:58:33.551 open tablets channel: tablets num=29, senders=74
+2026-07-08 00:59:57.677 open failed: RPC call is timed out (60s)
+ → cancel other node channels
+2026-07-08 01:00:57.649 txn_status=4 (load channel 已被 cancel)
+```
+
+同一窗口（00:47~01:53）内 `Reached timeout=60000ms @172.31.36.239:8061` 成簇出现，不是孤立超时。
+
+日志中未出现 `fail to offer request to the work pool`（queue offer 失败签名）。
+
+## Investigation
+
+### Step 1: 代码路径确认
+
+```
+VNodeChannel::_open_internal
+ → PBackendService_Stub::tablet_writer_open(timeout=60s)
+ → PInternalService::tablet_writer_open
+ → _heavy_work_pool.try_offer(LoadChannelMgr::open)
+ → LoadChannel::open
+ → TabletsChannel::open / _open_all_writers
+```
+
+`tablet_writer_open` 在 BE 侧进入 `brpc_heavy` work pool 后执行 load channel/open
writer。超时=60s 由 `tablet_writer_open_rpc_timeout_sec` 控制。
+
+### Step 2: 区分 offer 失败 vs 执行阻塞
+
+| 日志签名 | 含义 |
+|---------|------|
+| `fail to offer request to the work pool` | heavy pool 队列满，请求被拒绝 |
+| `RPC call is timed out`（无 offer 失败日志） | 请求进入 heavy pool 后执行过慢或排队超时 |
+
+当前是后者：请求进入了 heavy pool 但执行超时。
+
+### Step 3: 历史比对
+
+历史有同一模式：目标 BE `brpc_heavy_work_pool_threads=256` 在高并发导入时全部 active thread 被
long-running open/write 占满，后续请求排队等待 → 超时。
+
+临时缓解方式：`brpc_heavy_work_pool_threads 256→384`，但没闭环到具体卡点（故障时未保留 pstack）。
+
+## Root Cause
+
+目标 BE 的 `brpc_heavy` work pool 在导入高并发时被 `tablet_writer_open` /
`_open_all_writers` 占满，active heavy 线程长期被占导致后续请求排队超时。
+
+## Fix
+
+- **短期**: 降低/错峰同一 compute group 上的大 insert/load 并发；重试即可恢复
+- **中期**: 调大 `brpc_heavy_work_pool_threads`（256→384），但注意这只是增加排队容量，不是根治
+- **故障时务必保留现场**: 在重启 BE 之前先 `pstack` 卡住的进程，才能定位具体卡点（writer/open、IO、cgroup
stall 等）
+- **长期**: 分析 heavy pool 中 long-running 操作的耗时分布，考虑拆分或限流
+
+## Key diagnostic actions
+
+1. be/log 搜索 `RPC call is timed out` → 确认目标 BE 和时间窗口
+2. 确认同一窗口是否成簇出现（单点瞬时拥塞 vs 持续问题）
+3. 检查 `fail to offer request to the work pool` 签名（区分 offer 失败 vs 执行阻塞）
+4. 故障窗口保留 pstack 再重启 BE
+5. 检查 `tablet_writer_open_rpc_timeout_sec` 配置
diff --git a/skills/doris-debug/import/references/case-003-insert-job-fe-oom.md
b/skills/doris-debug/import/references/case-003-insert-job-fe-oom.md
new file mode 100644
index 0000000..9d8e3c3
--- /dev/null
+++ b/skills/doris-debug/import/references/case-003-insert-job-fe-oom.md
@@ -0,0 +1,87 @@
+---
+type: reference
+category: import
+keywords: [INSERT INTO, OOM, JobManager, job leak, FE memory, jmap, mem_leak]
+---
+
+# Case-003: INSERT INTO Job 不清理导致 FE 内存打爆反复 OOM
+
+## Environment
+
+- Doris version: 26.0.3 (cloud)
+- Architecture: storage-compute separation
+
+## Symptom
+
+FE 内存持续增长直至 OOM，重启后恢复但数小时/数天后再次 OOM。
+
+`jmap -histo` 显示大量 `InsertJob` 对象存活。
+
+`SHOW LOAD` / `SHOW INSERT` 显示数千个历史 job（状态均为 FINISHED 或 CANCELLED），远超正常范围。
+
+## Investigation
+
+### Step 1: 确认 OOM 类型
+
+```bash
+jmap -heap <fe_pid> | head -20
+# Old Gen 使用率接近 100%，频繁 Full GC
+
+jmap -histo <fe_pid> | head -30
+# InsertJob / AbstractJob 对象数量异常高
+```
+
+### Step 2: 确认 job 数量
+
+```sql
+SHOW LOAD ORDER BY CreateTime DESC LIMIT 100;
+-- 大量 FINISHED/CANCELLED 状态的 INSERT INTO job
+
+SELECT COUNT(*) FROM information_schema.loads WHERE Type='INSERT';
+-- 返回数字异常大（数千甚至上万）
+```
+
+### Step 3: 代码核对
+
+INSERT INTO job 流程：
+1. INSERT INTO 语句被注册为 job（包含 `InsertJob` 对象）
+2. Job 执行完成后状态变为 FINISHED
+3. `JobManager` 应将 FINISHED job 从内存中移除
+4. Bug: `JobManager` 未清理 FINISHED job
+5. FE restart 时 `JobManager.reload()` 从元数据加载所有历史 job → 反复 OOM
+
+```
+FE start
+ → JobManager.reload()
+ → 从 doris-meta 加载所有历史 INSERT job
+ → InsertJob 对象全部进入 heap
+ → 累积数千个 → Old Gen 满 → Full GC → OOM
+```
+
+## Root Cause
+
+INSERT INTO job 完成（FINISHED）后 `JobManager` 未从内存中清理对应的 `InsertJob` 对象。每次 FE
重启都会重新加载全部历史 job，导致内存持续增长直至 OOM。
+
+## Fix
+
+- **临时止血（紧急）**:
+ ```sql
+ -- 找到最老的 FINISHED job
+ SHOW LOAD WHERE State='FINISHED' ORDER BY CreateTime LIMIT 1;
+ ```
+ 如果版本支持 `DROP LOAD` 清理历史 job（谨慎操作，仅对不再需要审计的 job）
+
+- **临时止血（代码层面）**: FE 代码增加 `JobManager.cleanFinishedJobs()` 清理逻辑，定期清理 memory
中的已完成 job
+
+- **根治**:
+ 1. `JobManager` 对 FINISHED job 执行即时清理
+ 2. Reload 时增加 max job count 保护，超出阈值时只加载最近 N 个
+ 3. 为历史 job 增加 TTL，超时自动清理元数据和内存
+
+## Key diagnostic actions
+
+1. `SHOW LOAD` / `SHOW INSERT` 统计历史 job 总数
+2. `jmap -histo` 确认 InsertJob/AbstractJob 对象数量
+3. `jmap -heap` 确认 Old Gen 使用率
+4. fe.log 搜索 `OOM` / `GC overhead` / `heap space` 确认 OOM 时间线
+5. 确认 FE restart 后 job 数量是否同样上升（验证 reload 逻辑）
diff --git a/skills/doris-debug/materialized-view/SKILL.md
b/skills/doris-debug/materialized-view/SKILL.md
new file mode 100644
index 0000000..244a5bf
--- /dev/null
+++ b/skills/doris-debug/materialized-view/SKILL.md
@@ -0,0 +1,78 @@
+---
+name: doris-debug-materialized-view
+description: >
+ Use for Doris sync MV (rollup) miss or async MTMV refresh/rewrite issues.
+ Commands: SHOW ALTER TABLE MATERIALIZED VIEW, mv_infos(), EXPLAIN, REFRESH
MATERIALIZED VIEW.
+version: 0.2.0
+category: materialized-view
+---
+
+# Materialized View
+
+## Kind comparison
+
+| Kind | Code area | Check | Rewrite mechanism |
+|------|-----------|-------|-------------------|
+| Sync | `MaterializedViewHandler` / rollup | `SHOW ALTER TABLE MATERIALIZED
VIEW` | Transparent — planner auto-selects |
+| Async MTMV | `fe/.../mtmv/` | `mv_infos()` RefreshState | CBO-based —
EXPLAIN shows if matched |
+
+## Causes
+
+| ID | Cause | Evidence | Kind |
+|----|-------|----------|------|
+| A | Sync MV build pending/stuck | `SHOW ALTER TABLE MATERIALIZED VIEW`
State≠FINISHED | Sync |
+| B | MTMV refresh failed | `mv_infos()` RefreshState=FAIL | Async |
+| C | MTMV rewrite miss | EXPLAIN shows no MATERIALIZED_REWRITE | Async |
+| D | MTMV data stale | `mv_infos()` LastRefreshTime old | Async |
+| E | MV not selected by planner | EXPLAIN shows TABLE SCAN not MATERIALIZED |
Either |
+
+## 10 min triage
+
+```sql
+-- Sync MV status
+SHOW ALTER TABLE MATERIALIZED VIEW FROM db;
+
+-- Async MTMV status
+SELECT * FROM mv_infos('database'='db') WHERE Name='mv'\G
+
+-- Check rewrite eligibility
+EXPLAIN VERBOSE SELECT ...;
+
+-- Force refresh
+REFRESH MATERIALIZED VIEW mv;
+
+-- Enable external MV rewrite (for catalog queries)
+SET materialized_view_rewrite_enable_contain_external_table = true;
+```
+
+## Cause C — MTMV rewrite miss
+
+Common reasons (see `fe/.../mtmv/MTMVService.java` rewrite matcher):
+
+1. **Partition misalignment**: MV has `partition_sync=false` and query
partition doesn't match → `ALTER MATERIALIZED VIEW mv SET ("partition_sync" =
"true")`
+2. **Aggregation granularity**: MV groups by `(a, b)` but query groups by
`(a)` only → MV is not a valid rewrite target
+3. **Filter mismatch**: MV WHERE clause doesn't cover query WHERE clause →
check MV definition
+4. **Session var off**: `SET enable_materialized_view_rewrite = true`
+
+```sql
+-- Debug: force the optimizer to log MV matching
+SET enable_materialized_view_rewrite_debug_log = true;
+EXPLAIN VERBOSE SELECT ...; -- check fe/log/fe.audit.log for MV match details
+```
+
+## Cause A — Sync MV stuck
+
+```sql
+-- Cancel stuck build
+CANCEL ALTER TABLE MATERIALIZED VIEW FROM db ON table;
+
+-- Check BE compaction queue (MV build competes with compaction)
+./scripts/doris-debug be-metrics --be http://$BE:8040 --grep compaction
+```
+
+## Source
+
+- `fe/.../mtmv/MTMVService.java` — MTMV lifecycle
+- `fe/.../alter/MaterializedViewHandler.java` — sync MV alter
+- `fe/.../materializedview/MaterializedViewRewriter.java` — rewrite logic
+- `SessionVariable.java` — `enable_materialized_view_rewrite`,
`materialized_view_rewrite_enable_contain_external_table`
diff --git
a/skills/doris-debug/materialized-view/references/case-001-mtmv-rewrite-miss.md
b/skills/doris-debug/materialized-view/references/case-001-mtmv-rewrite-miss.md
new file mode 100644
index 0000000..36fc0a8
--- /dev/null
+++
b/skills/doris-debug/materialized-view/references/case-001-mtmv-rewrite-miss.md
@@ -0,0 +1,35 @@
+# Case: Async MTMV refresh succeeds but query rewrite never matches
+
+## Symptom
+
+`mv_infos()` shows `RefreshState = SUCCESS`, but `EXPLAIN SELECT` on the base
table never shows `MATERIALIZED_REWRITE`. Query always scans base table.
+
+## Root cause
+
+MTMV rewrite has strict preconditions that aren't surfaced well:
+1. `mv.properties("partition_sync")` must be enabled for partitioned tables,
or the MV partitions don't align
+2. The SELECT must be an exact match or a CBO-provable subset of the MV
definition
+3. If the MV uses aggregation, the query must select at the MV's aggregation
granularity
+4. `enable_materialized_view_rewrite` session var must be true (default true,
but check)
+
+## Fix direction
+
+```sql
+-- 1. Check MV state (not just RefreshState — check all columns)
+SELECT * FROM mv_infos('database'='db') WHERE Name='mv'\G
+
+-- 2. Check rewrite eligibility
+EXPLAIN VERBOSE SELECT ...; -- look for "mv=... matched=false" in output
+
+-- 3. Ensure rewrite switch is on
+SET enable_materialized_view_rewrite = true;
+
+-- 4. For partition alignment
+ALTER MATERIALIZED VIEW mv SET ("partition_sync" = "true");
+REFRESH MATERIALIZED VIEW mv COMPLETE;
+
+-- 5. Check if the query's WHERE clause can be satisfied
+-- If MV filters on col_a but query filters on col_b, rewrite won't match
+```
+
+Source: `fe/.../mtmv/MTMVService.java` rewrite rules,
`fe/.../materializedview/MaterializedViewRewriter.java`.
diff --git a/skills/doris-debug/node/SKILL.md b/skills/doris-debug/node/SKILL.md
new file mode 100644
index 0000000..3cecf91
--- /dev/null
+++ b/skills/doris-debug/node/SKILL.md
@@ -0,0 +1,86 @@
+---
+name: doris-debug-node
+description: >
+ Use for Doris FE/BE OOM, crash, or false Alive. BE: MemTrackerLimiter lines
in be.WARNING;
+ FE: JVM heap / jstack. Distinguish heartbeat Alive from query/brpc serving
health.
+version: 0.2.0
+category: node
+---
+
+# Node Health
+
+## Causes
+
+| ID | Cause | Evidence | Source anchor |
+|----|-------|----------|---------------|
+| A | BE OOM / killed | `dmesg` shows `Killed`; `be.WARNING` MemTrackerLimiter
| `mem_tracker_limiter.cpp` |
+| B | FE JVM OOM | GC overhead / heap dump; `fe.out` shows `OutOfMemoryError`
| `fe.conf` JVM opts |
+| C | BE crash (SIGSEGV/SIGABRT) | Core dump; `be.out` stacktrace |
`doris_main.cpp` signal handler |
+| D | False Alive | `SHOW BACKENDS` Alive=true but queries fail brpc |
`heartbeat` vs brpc health |
+| E | FD exhaustion | `Too many open files` in be.WARNING | `ulimit -n` |
+
+## 10 min triage
+
+```bash
+# Memory pressure signals
+./scripts/doris-debug log-grep be/log --pack memory
+dmesg -T | grep -i "killed process" | tail -5
+
+# FE JVM
+jstack $(cat /path/to/fe.pid) > /tmp/fe.jstack
+jmap -heap $(cat /path/to/fe.pid) 2>/dev/null | head -30
+
+# FD count
+ls /proc/$(cat /path/to/be.pid)/fd | wc -l
+cat /proc/$(cat /path/to/be.pid)/limits | grep "open files"
+
+# Resource snapshot
+top -b -n 1 -p $(cat /path/to/be.pid) | head -5
+free -h
+```
+
+## Cause A — BE OOM
+
+Doris tracks memory via MemTracker, but process RSS exceeds tracked memory due
to:
+- jemalloc arena fragmentation (10-20% overhead)
+- brpc buffer pools (outside MemTracker)
+- Thread stacks (~8MB each × thread count)
+
+```bash
+# RSS vs tracked
+ps -o rss= -p $BE_PID | awk '{printf "RSS: %.1f GB\n", $1/1024/1024}'
+grep "MemTrackerLimiter.*total" be/log/be.WARNING | tail -1
+```
+
+```properties
+# be.conf — defensive settings
+mem_limit = 70% # Leave 30% for overhead
+max_segment_cache_size = 0 # Or cap explicitly
+enable_je_purge = true # jemalloc dirty page cleanup
+```
+
+If `ProcessMemoryReachedCancel` didn't fire but OOM killer did: the gap
between mem_limit and system RAM wasn't enough for jemalloc + brpc overhead.
+
+## Cause B — FE JVM
+
+```bash
+# FE JVM opts (from fe.conf or start script)
+ps aux | grep "PaloFe" | grep -o "\-Xmx[^ ]*"
+```
+
+Common: FE heap grows with catalog metadata (many tables/partitions). Raise
`-Xmx` or reduce `hive_meta_cache_ttl_sec` / catalog refresh frequency.
+
+## Cause D — False Alive
+
+Alive=true means the heartbeat RPC succeeds. This does NOT mean the brpc port
(8060) is healthy for data transfer. If queries fail with E1008 while
Alive=true:
+
+→ Route to **doris-debug-query** (Cause D — Exchange / brpc)
+→ Check `priority_networks`, `enable_brpc_connection_check`,
`reset_rpc_channel`
+→ Do not trust `SHOW BACKENDS` Alive alone
+
+## Source
+
+- `be/src/runtime/memory/mem_tracker_limiter.cpp`
+- `be/src/service/doris_main.cpp` — jemalloc init, signal handler
+- `fe/.../master/HeartbeatHandler.java`
+- `be/src/common/config.cpp` — `mem_limit`, `enable_je_purge`,
`max_segment_cache_size`
diff --git a/skills/doris-debug/node/references/case-001-be-oom-memtracker.md
b/skills/doris-debug/node/references/case-001-be-oom-memtracker.md
new file mode 100644
index 0000000..b1de162
--- /dev/null
+++ b/skills/doris-debug/node/references/case-001-be-oom-memtracker.md
@@ -0,0 +1,26 @@
+# Case: BE OOM from MemTracker — large query spills but limit doesn't fire
+
+## Symptom
+
+BE process killed by OOM killer (`dmesg | grep "Killed"`), but `be.WARNING`
shows `MemTrackerLimiter` lines with memory near but not exceeding `mem_limit`.
`ProcessMemoryReachedCancel` not triggered.
+
+## Root cause
+
+`mem_limit` controls the Doris-tracked memory, but process RSS includes:
+- Jemalloc arena fragmentation (can add 10-20% overhead)
+- brpc buffers (outside MemTracker)
+- Thread stacks
+- Page cache / mmap regions
+
+When `mem_limit = 80%` of system RAM and jemalloc fragments to 95%, the OOM
killer fires even though Doris's own tracking thinks it's within limit.
+
+## Fix direction
+
+1. Check actual RSS: `ps -o rss= -p $BE_PID` → convert to GB
+2. `dmesg -T | grep -i "killed process"` to confirm OOM kill
+3. `log-grep be/log --pack memory` to see last MemTracker snapshot before crash
+4. Lower `mem_limit` to 60-70% of system RAM to leave room for overhead
+5. Check `max_segment_cache_size` / `chunk_reserved_bytes_limit` — these are
outside MemTracker
+6. Consider `enable_je_purge = true` (be.conf) for jemalloc dirty page cleanup
+
+Source: `be/src/runtime/memory/mem_tracker_limiter.cpp`,
`be/src/service/doris_main.cpp` jemalloc init.
diff --git a/skills/doris-debug/node/references/case-002-filecache-lru-leak.md
b/skills/doris-debug/node/references/case-002-filecache-lru-leak.md
new file mode 100644
index 0000000..94daec7
--- /dev/null
+++ b/skills/doris-debug/node/references/case-002-filecache-lru-leak.md
@@ -0,0 +1,84 @@
+---
+type: reference
+category: node
+keywords: [file_cache, LRU, memory leak, heap dump, jeprof, MEM_LIMIT_EXCEEDED]
+---
+
+# Case-002: File Cache LRU Recorder 队列堆积导致 BE 内存泄漏
+
+## Environment
+
+- Doris version: 4.1.2 (cloud)
+- Architecture: storage-compute separation
+
+## Symptom
+
+BE RSS 持续线性增长，无查询时也不释放。最终触发 `MEM_LIMIT_EXCEEDED`。
+CPU 在内存上涨期间也升高。DataPageCache 和 SegmentCache 监控显示正常（低水位）。
+
+## Key evidence
+
+heap dump 分析显示内存增量主要落在：
+```
+LRUQueueRecorder::record_queue_event
+ → moodycamel::ConcurrentQueue<CacheLRULog>
+```
+
+`/profile` 中：
+```
+VmRSS: 228.56 GB
+DataPageCache Current: 410.65 MB ← 不是它
+SegmentCache Current: 0 ← 不是它
+QueryCache@cache Current: 45.56 GB ← 占大头但不是泄漏源
+```
+
+`CacheMemory: 45.95 GB` 主要来自 QueryCache，而 heap dump 增量大头是 FileCache LRU
recorder 队列。
+
+## Investigation
+
+### Step 1: 排除 DataPageCache/SegmentCache
+
+监控显示异常 BE 的 DataPageCache（410MB）和 SegmentCache（0）远低于其他 BE，排除。
+
+### Step 2: heap dump 定位
+
+`jeprof --text` 显示热点在：
+```
+LRUQueueRecorder::record_queue_event
+moodycamel::ConcurrentQueue<CacheLRULog>::enqueue
+```
+
+这是 FileCache 的 LRU 淘汰记录队列（不是 DataPageCache 的数据缓存）。
+
+### Step 3: 代码核对
+
+4.1.2 版本中，`file_cache_background_lru_log_replay_interval_ms` 默认为 `1000`（1
秒消费一次），但没有 hard cap 限制 LRU recorder queue 的大小。
+
+当 FileCache 访问频率极高时，LRU log 的生产速度 > 消费速度（1000ms 一次），队列无限堆积 → heap 持续上涨。
+
+### Step 4: 修复版本确认
+
+4.1.8 版本已引入修复（cherry-pick apache/doris PR #64381）：
+- 新增 `file_cache_background_lru_log_queue_max_size=500000`
+- `file_cache_background_lru_log_replay_interval_ms` 默认从 `1000` 改为 `1`
+- 增加 LRU recorder queue size 的计数/释放逻辑
+
+## Root Cause
+
+FileCache 的 LRU recorder queue 在高频访问下生产速度超过消费速度，且 4.1.2 版本中没有 queue size hard
cap，导致 queue 无限堆积 → BE heap 持续增长。
+
+## Fix
+
+- **临时规避**: 调小 `file_cache_background_lru_log_replay_interval_ms`（1000→1 或
100），提高消费频率。但已堆积的内存不会立即归还
+ ```bash
+ curl -X POST
"http://<be>:8040/api/update_config?file_cache_background_lru_log_replay_interval_ms=1&persist=true"
+ ```
+- **止血**: 重启受影响 BE 释放已堆积的 recorder queue 内存
+- **根治**: 升级到 4.1.8 或 backport PR #64381（增加 queue size hard cap）
+
+## Key diagnostic actions
+
+1. `jeprof --text` heap dump → 确认热点在 `LRUQueueRecorder`
+2. `be-metrics --grep file_cache` → 观察
`file_cache_need_update_lru_blocks_length`
+3. 排除 DataPageCache/SegmentCache/QueryCache 后仍高 RSS → FileCache LRU
+4. 检查 `file_cache_background_lru_log_replay_interval_ms` 当前值
diff --git
a/skills/doris-debug/node/references/case-003-be-scanner-pileup-memory.md
b/skills/doris-debug/node/references/case-003-be-scanner-pileup-memory.md
new file mode 100644
index 0000000..9cb7405
--- /dev/null
+++ b/skills/doris-debug/node/references/case-003-be-scanner-pileup-memory.md
@@ -0,0 +1,113 @@
+---
+type: reference
+category: node
+keywords: [memory leak, scanner pileup, pipeline fragment, RSS growth, jeprof,
QueryCache, OLAP_SCAN_OPERATOR_FILTER_DEPENDENCY]
+---
+
+# Case-003: BE 内存持续增长 + Scanner 堆积（Pipeline Fragment 未释放）
+
+## Environment
+
+- Doris version: 4.1.3 (cloud)
+- Architecture: storage-compute separation, multiple BE
+
+## Symptom
+
+单台 BE RSS 持续线性增长至 228GB（limit 226GB），CPU 也随之升高，其他 BE 正常。
+
+`/profile` 显示：
+```
+VmRSS: 228.56 GB
+Pipeline fragment contexts still running: 3,641
+_num_running_scanners: 1 → 出现 4,686 次（另一台正常 BE 只有 1 次）
+QueryCache@cache Current: 45.56 GB
+DataPageCache Current: 410.65 MB ← 正常
+SegmentCache Current: 0 ← 正常
+```
+
+`MemoryGC` 扫描到 1000+ running query/load 但无大任务可取消。
+
+## Key evidence
+
+### Pipeline dump 分析
+
+异常 BE 上有 3,641 个 pipeline fragment context 仍在运行（正常 BE 远低于此）。
+
+Top 10 滞留 query_id（按 elapsed 排序）：
+
+| Rank | Query ID | Max Elapsed | 卡点 |
+|------|----------|-------------|------|
+| 1 | 1de87894... | 19.34min | OLAP_SCAN_OPERATOR_FILTER_DEPENDENCY |
+| 2 | 9dac5480... | 19.04min | OLAP_SCAN_OPERATOR_FILTER_DEPENDENCY |
+| 3-10 | ... | 15~19min | 同上 |
+
+所有滞留 query 的共同特征：
+- `_num_running_scanners=0`（scanner 已停止）
+- `query_timeout_second=1200`, `is_timeout=false`
+- Query tracker `Current=64B, Peak=13.60MB`（单 query 内存占用很小）
+- 共同卡点：`OLAP_SCAN_OPERATOR_FILTER_DEPENDENCY` / runtime filter NOT_READY
+
+### FE 日志（同时段）
+
+```
+StmtExecutor.execute begin to execute query: ~183,002 条
+LoadAction.streamLoad: ~215,684 条
+Use query cache: ~8,988 条
+```
+
+同时段有极高并发的查询和 Stream Load。
+
+异常 BE `10.20.80.195` 接收到的 Stream Load 重定向次数最高（27,152 次）。
+
+### 内存分配热点
+
+QueryCache 占 45.56GB（最大单类占用），但 DataPageCache/SegmentCache 很低。
+
+## Investigation
+
+### Step 1: 排除 DataPageCache/SegmentCache
+
+监控显示两者远低于其他 BE（DataPageCache 410MB vs 正常 BE 数 GB），排除。
+
+### Step 2: 聚焦 Scanner/Pipeline 堆积
+
+3,641 个 pipeline fragment contexts 仍在运行，_num_running_scanners 出现 4,686 次（对比正常
BE 仅 1 次）。说明大量 fragment 已经停止 scanner 但未退出。
+
+### Step 3: 卡点分析
+
+所有滞留 query 卡在 `OLAP_SCAN_OPERATOR_FILTER_DEPENDENCY`（runtime filter
NOT_READY），不是 active scanner。说明这些 fragment 在等待其他节点广播 runtime filter，但由于某种原因
filter 永远不会到达 → fragment 永远不退出 → context 堆积。
+
+### Step 4: QueryCache 调查
+
+QueryCache 45.56GB 是显著占用。高频查询命中 cache（FE 日志 8,988 次 `Use query cache`），但
QueryCache 的内存占用不会被 MemTracker 精确追踪，可能在被 prune 后留下 native 内存碎片。
+
+### Step 5: Stream Load 路由不均
+
+异常 BE 接收了最多 Stream Load（27,152 次），比其他 BE 高约 17%。虽不是数量级差异，但叠加 scanner pileup
可能加剧内存压力。
+
+## Root Cause（多因素）
+
+1. **Pipeline fragment 滞留**: 大量 fragment 卡在
`OLAP_SCAN_OPERATOR_FILTER_DEPENDENCY`（runtime filter 等待超时或永不满足），fragment
context 不释放
+2. **QueryCache 内存占用**: 45GB+ QueryCache 在扫描压力下未被充分 prune
+3. **Stream Load 路由偏斜**: 异常 BE 承担更多导入流量
+
+主要是因素 1：Scanner 虽已停止，但 fragment 等待 runtime filter → context 堆积 → 内存和
CPU（memory GC 压力）持续增长。
+
+## Fix
+
+- **短期止血**: 重启异常 BE 清除堆积 fragment（重启前保留 pstack + heap dump 用于后续分析）
+- **中期**:
+ - 排查 runtime filter 超时配置（`runtime_filter_wait_time_ms`），避免 fragment 无限等待
+ - 评估临时降低/关闭 QueryCache 验证 RSS 是否下降
+ - 均衡 Stream Load 路由
+- **长期**: 修复 runtime filter 等待超时后 fragment context 的清理机制
+
+## Key diagnostic actions
+
+1. `/profile` 或 pipeline dump → pipeline fragment contexts still running 数量
+2. 按 elapsed 排序找 Top query_id → 确认共同卡点
+3. be/log `MemoryGC` 扫描结果（确认是否有可取消的大任务）
+4. `jeprof --inuse_space` 确认分配热点
+5. `be-metrics --grep cache` 分解各类 cache 占用
+6. FE 日志统计同时段查询/导入量
+7. 异常 BE 上 `top -H` / `pstack` 保留现场
diff --git a/skills/doris-debug/node/references/case-004-fe-bdbje-corruption.md
b/skills/doris-debug/node/references/case-004-fe-bdbje-corruption.md
new file mode 100644
index 0000000..32e8d60
--- /dev/null
+++ b/skills/doris-debug/node/references/case-004-fe-bdbje-corruption.md
@@ -0,0 +1,121 @@
+---
+type: reference
+category: node
+keywords: [FE, BDBJE, startup failure, metadata corruption, .jdb, doris-meta,
recovery]
+---
+
+# Case-004: FE BDBJE 元数据损坏导致无法启动
+
+## Environment
+
+- Doris version: all versions (BDBJE-based FE)
+- Architecture: shared-nothing / cloud
+
+## Symptom
+
+FE 启动时退出，fe.log 显示：
+```
+java.io.FileNotFoundException: doris-meta/bdb/0000014d.jdb
+ (No such file or directory)
+```
+或：
+```
+com.sleepycat.je.EnvironmentFailureException: ... Environment must be closed
...
+java.lang.IllegalStateException: Environment is invalid
+```
+
+FE 进程启动后几秒自动退出。`SHOW FRONTENDS` 在其他节点上显示该 FE 状态为 `UNKNOWN` 或心跳超时。
+
+## Investigation
+
+### Step 1: 确认文件系统状态
+
+```bash
+ls -la fe/doris-meta/bdb/
+# 查看 .jdb 文件列表，确认缺失的文件名
+
+du -sh fe/doris-meta/bdb/
+# 确认目录大小是否异常小（< 正常大小 = 可能被清空或损坏）
+
+df -h fe/doris-meta/
+# 确认磁盘是否曾经写满
+```
+
+常见场景：
+- 磁盘写满 → BDBJE write 失败 → 日志文件不完整或缺失
+- 异常关机（kill -9 / 断电）→ BDBJE 未完成 checkpoint → 日志文件损坏
+- 人为误删 `doris-meta/bdb/` 中的文件
+
+### Step 2: 确认集群状态
+
+```sql
+-- 在其他健康 FE 上执行
+SHOW FRONTENDS\G
+```
+
+确认：
+- 当前是否还有健康的 Master/Follower？
+- 出问题的 FE 角色是什么（Master / Follower / Observer）？
+
+**关键原则：永远不要删除最后一个健康 Master 的 doris-meta/ 目录。**
+
+### Step 3: 恢复策略
+
+#### 场景 A: 出问题的是 Follower 或 Observer
+
+```
+1. 从集群中移除该 FE:
+ ALTER SYSTEM DROP FOLLOWER "bad_host:9010"
+ 或 ALTER SYSTEM DROP OBSERVER "bad_host:9010"
+
+2. 停止该 FE 进程
+
+3. 清空 doris-meta/:
+ rm -rf fe/doris-meta/*
+
+4. 重新加入集群:
+ ALTER SYSTEM ADD FOLLOWER "bad_host:9010"
+ # FE 会从 Master 自动同步元数据
+```
+
+#### 场景 B: 出问题的是 Master 但还有其他健康 Follower
+
+```
+1. 从集群中移除该 Master（集群会自动重新选主）
+2. 清空该 FE 的 doris-meta/
+3. 以 Follower 身份重新加入
+```
+
+#### 场景 C: 出问题的是唯一的 Master 且没有 Follower
+
+```
+⚠️ 最高风险场景
+1. 先备份整个 fe/doris-meta/ 目录
+2. 尝试 BDBJE recovery:
+ java -jar je.jar DbRecover -h fe/doris-meta/bdb
+3. 如果 recovery 失败，使用最近的元数据镜像:
+ fe/doris-meta/image/ 中最新的 image 文件
+4. 联系专业支持
+```
+
+## Root Cause
+
+BDBJE 日志文件（.jdb）因磁盘满、异常关机、或人为误操作导致损坏或丢失。
+
+## Fix
+
+- **从健康副本恢复**: 如果有健康的 Follower/Observer，从其 rsync `doris-meta/`
+- **BDBJE recovery**: 使用 Berkeley DB JE 的 recovery 工具（成功率取决于损坏程度）
+- **预防**:
+ 1. 监控 FE 磁盘使用率，预留充足空间
+ 2. 至少配置 1 Follower + 1 Observer 实现高可用
+ 3. 定期备份 `fe/doris-meta/`
+ 4. 使用优雅关机（`kill -15` 而非 `kill -9`）
+
+## Key diagnostic actions
+
+1. `ls -la fe/doris-meta/bdb/` 确认文件完整性
+2. `SHOW FRONTENDS` 确认集群还有哪些健康节点
+3. 确认损坏 FE 的角色（Master/Follower/Observer）
+4. 永远不要删除最后一个健康 Master 的 `doris-meta/`
+5. 恢复前先备份现有的 `doris-meta/`
diff --git a/skills/doris-debug/query/SKILL.md
b/skills/doris-debug/query/SKILL.md
new file mode 100644
index 0000000..b7b4d4d
--- /dev/null
+++ b/skills/doris-debug/query/SKILL.md
@@ -0,0 +1,87 @@
+---
+name: doris-debug-query
+description: >
+ Use for Apache Doris slow/hanging/timeout queries. Covers FE planner
(Nereids timeout),
+ BE Profile bottlenecks, and Exchange WaitForData / brpc E1008 on port 8060.
+ Session vars: enable_profile, query_timeout, nereids_timeout_second.
+version: 0.2.0
+category: query
+---
+
+# Query Troubleshooting
+
+## Causes
+
+| ID | Cause | Evidence |
+|----|-------|----------|
+| A | Client timeout too small | `query_timeout` vs Profile wall |
+| B | Planner | `Nereids cost too much time` / `nereids_timeout_second` |
+| C | BE compute (scan/join/spill) | Profile active ExecTime |
+| D | Exchange / brpc | WaitForData ≈ timeout; `failed to send brpc when
exchange`; `[E1008]` on `:8060` |
+
+## 10 min
+
+```sql
+SET enable_profile = true;
+-- reproduce
+```
+
+```bash
+./scripts/doris-debug log-grep fe/log be/log --query-id "$QID" --pack planner
+./scripts/doris-debug log-grep be/log --query-id "$QID" --pack exchange
+./scripts/doris-debug audit-topk fe/log/fe.audit.log -k 20 --min-ms 5000
+```
+
+Planner mitigation (session only): `SET disable_join_reorder=true;` for fat
ETL joins; fix stats with `ANALYZE`.
+
+## Cause C — BE compute (scan/join/spill)
+
+When Profile shows high active ExecTime (not Wait\*), route to
**doris-profile-reader** for deep Profile analysis:
+
+> **Skill routing**: If `doris-profile-reader` is available, invoke it with
the query's Profile to identify which operator (scan/join/agg/spill) dominates.
Key indicators to pass:
+> - `ActiveTime` per operator
+> - `RowsRead` / `ScanBytes` for scan operators
+> - `SpillDataSize` if disk spill occurred
+> - `JoinType` + `JoinRows` for join operators
+
+Without profile-reader, check directly:
+```bash
+# Get the Profile text
+curl -s "http://$FE:8030/api/query_profile?query_id=$QID";
+```
+
+Key Profile counters for Cause C:
+| Counter | Meaning | Threshold |
+|---------|---------|-----------|
+| `ScannerGetBlockTime` | Time reading from storage | > 50% of ExecTime → IO
bound |
+| `SpillDataSize` | Data spilled to disk | > 0 → memory pressure, check
`enable_spill` |
+| `JoinProbeTime` | Hash join probe wall time | > 30% → check join order /
type |
+| `RowsRead` / `ScanBytes` | Scan volume | > 1B rows → check partition pruning
|
+
+Profile interpretation: prefer active time over Wait\*.
+
+## Cause D — Exchange / brpc
+
+Code facts (apache/doris):
+
+1. Error text in `be/src/exec/operator/exchange_sink_buffer.h`.
+2. `GET /api/reset_rpc_channel/{endpoints}` clears
**`brpc_internal_client_cache` only** (`reset_rpc_channel_action.cpp`).
+3. `enable_brpc_connection_check` (default false) →
`FragmentMgr::_check_brpc_available` hand_shake; consecutive failures erase
internal cache.
+
+```bash
+curl -s "http://$BE:8040/api/reset_rpc_channel/all";
+```
+
+```properties
+# be.conf — restart for static DEFINE_Bool
+enable_brpc_connection_check = true
+brpc_connection_check_timeout_ms = 10000
+```
+
+Do not trust ICMP alone. Check `priority_networks`.
+
+## Source
+
+- `SessionVariable.java` — `QUERY_TIMEOUT`, `NEREIDS_TIMEOUT_SECOND`
+- Profile WaitForData — exchange source operator
+- `exchange_sink_buffer.h`, `reset_rpc_channel_action.cpp`, `fragment_mgr.cpp`
diff --git
a/skills/doris-debug/query/references/case-001-exchange-waitfordata.md
b/skills/doris-debug/query/references/case-001-exchange-waitfordata.md
new file mode 100644
index 0000000..afac6c0
--- /dev/null
+++ b/skills/doris-debug/query/references/case-001-exchange-waitfordata.md
@@ -0,0 +1,12 @@
+# Case: Timeout = EXCHANGE WaitForData + E1008
+
+## Symptom
+Receiver WaitForData≈timeout, sender scan ms-level, RpcCount=0. Alive=true.
+
+## Root cause
+brpc `:8060` path stuck; not SQL/CPU.
+
+## Fix direction
+`log-grep --pack exchange`; `reset_rpc_channel/all`;
`enable_brpc_connection_check`; verify `priority_networks`.
+
+Source: `exchange_sink_buffer.h`, `reset_rpc_channel_action.cpp`.
diff --git
a/skills/doris-debug/query/references/case-002-plan-time-iceberg-rpc.md
b/skills/doris-debug/query/references/case-002-plan-time-iceberg-rpc.md
new file mode 100644
index 0000000..d5ea491
--- /dev/null
+++ b/skills/doris-debug/query/references/case-002-plan-time-iceberg-rpc.md
@@ -0,0 +1,100 @@
+---
+type: reference
+category: query
+keywords: [plan time, nereids, iceberg, create scan range, catalog rpc, slow
query]
+---
+
+# Case-002: FE Plan Time 10 分钟（Iceberg 外表 Create Scan Range RPC 爆炸）
+
+## Environment
+
+- Doris version: 4.1.3 (Nereids optimizer)
+- Architecture: shared-nothing
+- Catalog type: Iceberg external catalog
+
+## Symptom
+
+SQL `SELECT * FROM iceberg.db.table WHERE lower(name) NOT LIKE '%xxx%'` 耗时 10
分 8 秒。
+同一查询多次执行，每次都是 ~10 分钟。用户怀疑 BE 执行慢或 S3 读慢。
+
+## Key evidence (Profile)
+
+```
+Profile ID: a6e9d417718d4cf6-88857f81140da87f
+
+Total: 10min8sec
+ Plan Time: 10min8sec ← 占 100%
+ Nereids Translate Time: 10min8sec
+ Finalize Scan Node Time: 10min8sec
+ Create Scan Range Time: 10min1sec ← 核心耗时
+ Get Splits Time: 6sec323ms
+
+ Schedule Time: 4ms
+ Wait and Fetch Result Time: 214ms
+ Fetch Result Time: 213ms
+
+FILE_SCAN_OPERATOR:
+ ExecTime: 211.882ms ← BE 执行只有 200ms
+ RowsProduced: 540
+ ScanRows: 593
+ ScanBytes: 1021.08 KB
+ FileNumber: 64
+ partitions: 63
+
+Is Nereads: Yes
+Is Cached: No
+```
+
+**结论：BE 执行只有 214ms/1MB，问题完全在 FE planning 阶段。**
+
+## Investigation
+
+### Step 1: 确认 Plan Time 占比
+
+`Plan Time=10min8sec` 等于 Total，BE `ExecTime=211ms`，排除 BE compute / IO 瓶颈。
+
+### Step 2: 拆解 Nereids planning 阶段
+
+`Nereids Translate Time` = 10min8sec → 不是 FE lock / GC（`Garbage Collect During
Plan Time=52sec` 只占一小部分）。
+
+`Finalize Scan Node Time` = 10min8sec → 问题在最终化 scan node 阶段。
+
+`Create Scan Range Time` = 10min1sec → 创建 scan range 是核心耗时。
+
+`Get Splits Time` = 6sec → 获取文件分片只花了 6 秒。
+
+### Step 3: 代码核对
+
+4.1.3 代码中 Iceberg catalog 在 Create Scan Range 阶段会：
+
+1. 遍历每个 partition（64 个文件 / 63 个分区）
+2. 每次 `getManifestFiles()` → 调用 `UpdateRunningStatus()` → 触发
`msClient.updateInstance()` RPC 到 meta-service
+3. 每次 RPC 约 200ms
+4. 反复调用累积 → 10min+
+
+```
+CreateScanRange
+ → for each partition:
+ → getManifestFiles()
+ → UpdateRunningStatus()
+ → msClient.updateInstance() ← 每次 ~200ms RPC
+```
+
+这与 BE S3 list 或 scan 无关，纯粹是 FE-meta-service 之间的 RPC 次数过多。
+
+## Root Cause
+
+Nereids 在 Create Scan Range 阶段对 Iceberg 外表的处理中，每次获取 manifest 文件时调用了不必要的
meta-service RPC（`updateInstance`），且这个 RPC 对每个 partition 都执行一次。64 个文件 × 每次
~200ms → 10 分钟。
+
+## Fix
+
+- **短期**: 调大 `iceberg_manifest_cache_refresh_interval_s`，减少 manifest 重新获取频率
+- **长期**: 改进 Create Scan Range 阶段，减少对 meta-service 的重复 RPC 调用（合并批量获取，或移除不必要的
`updateInstance`）
+
+## Key diagnostic actions
+
+1. Profile → 确认 `Plan Time` 占比（而不是 BE ExecTime）
+2. 确认 `Is Nereids=Yes`
+3. 拆解 Plan Time 子阶段：`Nereids Translate Time` → `Finalize Scan Node Time` →
`Create Scan Range Time`
+4. 对比 `Get Splits Time`（正常）和 `Create Scan Range Time`（异常），确认问题在创建 scan range
而非获取文件列表
+5. 如果是 Iceberg/Hive 外表 → 检查 manifest/catalog RPC 耗时
diff --git
a/skills/doris-debug/query/references/case-003-exchange-e11-fanout.md
b/skills/doris-debug/query/references/case-003-exchange-e11-fanout.md
new file mode 100644
index 0000000..e1b37a3
--- /dev/null
+++ b/skills/doris-debug/query/references/case-003-exchange-e11-fanout.md
@@ -0,0 +1,139 @@
+---
+type: reference
+category: query
+keywords: [E11, exchange, brpc, fanout, Resource temporarily unavailable,
parallel_exchange_instance_num]
+---
+
+# Case-003: 高峰期 Exchange brpc E11 — 并发 fragment 过多导致 RPC 发送侧打满
+
+## Environment
+
+- Doris version: 4.0.7 (cloud)
+- Architecture: storage-compute separation, 3 BE in compute group
+
+## Symptom
+
+高峰期 10:00~10:30 大量查询和导入报错：
+```
+failed to send brpc when exchange, error=Resource temporarily unavailable,
+error_text=[E11]Resource temporarily unavailable @10.244.0.29:8060
+```
+伴随 `RPC meet failed: [E11]`、`pipeline_fragment_context.cpp:171] cancel`、导入
`cancel node channel`。
+
+## Key evidence
+
+### 时间分布（E11 命中统计）
+
+| 时间 | E11 日志行数 | 去重 query_id |
+|------|-------------|-------------|
+| 10:00 | 7,282 | 46 |
+| 10:03 | 35,639 | 58 |
+| 10:10 | 248 | 2 |
+| 10:25 | 79,391 | 107 |
+| 10:26 | 47,766 | 125 |
+| 10:27 | 29,522 | 68 |
+
+高峰集中在 10:25~10:27，全窗口去重 query_id 共 406 个。
+
+### 目标端分布
+
+```
+10.244.0.29:8060 → 144,181 行
+10.244.0.20:8060 → 55,419 行
+10.244.0.10:8060 → 248 行
+```
+
+E11 集中在两台 BE，不是全集群均匀分布。发送端样例：
+```
+10.244.0.10 → 10.244.0.29/20
+10.244.0.20 → 10.244.0.29
+```
+
+### Fragment 并发量
+
+```
+10:25:01 fragment_num: 1575 / 1556 / 1326 (三台BE)
+10:25:21 fragment_num: 2056 (10.244.0.10)
+10:26:42 fragment_num: 2108
+10:27:12 fragment_num: 2444
+```
+
+BE `fragment_mgr.cpp` 每分钟启动 fragment 行数：
+```
+10:17: 15,296
+10:20: 9,020
+10:25: 7,994
+10:26: 6,902
+10:27: 8,046
+```
+
+### 并行度配置
+
+```sql
+parallel_exchange_instance_num = 100 ← 关键
+parallel_fragment_exec_instance_num = 8
+parallel_pipeline_task_num = 0
+```
+
+### 排除项
+
+日志中未检索到以下信号：
+- `pthread_create failed` / `failed to create thread` → 不是线程池耗尽
+- `queue is full` / `EOVERCROWDED` → 不是队列溢出
+- `too many open files` → 不是 FD 耗尽
+- `MEM_LIMIT_EXCEEDED` → 不是内存限制
+
+## Investigation
+
+### Step 1: 确认错误来源
+
+代码路径（4.0.7）：
+```
+be/src/pipeline/exec/exchange_sink_buffer.h
+ ExchangeSendCallback::call()
+ → brpc::Controller::Failed()
+ → "failed to send brpc when exchange, error={}, error_text={}"
+```
+
+这是 BE-BE exchange RPC 发送路径失败，不是 SQL 语义错误，也不是 FE thrift/mysql 线程池拒绝。
+
+### Step 2: 区分 E11 brpc vs pthread_create EAGAIN
+
+两者都显示 "Resource temporarily unavailable"，但含义完全不同：
+
+| 信号 | 含义 |
+|------|------|
+| `[E11]` + `failed to send brpc when exchange` | brpc socket 发送侧
EAGAIN，socket send buffer 满或对端接收慢 |
+| `pthread_create failed (EAGAIN)` + `cgroup: fork rejected by pids
controller` | 线程/pids 限制打满 |
+
+当前日志是前者，不应按线程池扩容处理。
+
+### Step 3: 判断故障范围
+
+E11 集中在两台 BE（29 和 20），BE 10 几乎不受影响。且同窗口 FE load job dispatch 每分钟 398~1360
次，导入与查询同时高并发。
+
+## Root Cause
+
+高并发 LOAD + SELECT 叠加，`parallel_exchange_instance_num=100` + 默认 pipeline
并行度，导致真实同时执行的 fragment/exchange RPC 数远超 brpc socket 处理能力。发送侧 socket send buffer
满 → brpc controller 返回 E11 → query/load cancel。
+
+这是已知问题族，历史同类 case 均通过降低 exchange/pipeline 并行度缓解。
+
+## Fix
+
+- **短期**: 调小 `parallel_exchange_instance_num`（100 → 50 或 32），控制 exchange fanout
+- **中期**: 对高峰期 workload 加 Workload Group 并发/排队限制
+- **长期**: backport apache/doris PR #50113（one rpc send multi blocks，减少 RPC 次数）
+
+调参顺序：
+1. 先只调 `parallel_exchange_instance_num`: 100 → 50 观察
+2. 如仍复现 → 继续降到 32 或配合降低 `parallel_pipeline_task_num` 到 1
+3. 不要盲目增大线程池参数（当前没有线程池 reject 证据，增线程只会放大 E11）
+
+## Key diagnostic actions
+
+1. 确认错误前缀是 `failed to send brpc when exchange`（能确认是 exchange brpc 路径）
+2. 统计 E11 目标 BE 分布：单点 vs 全集群（不同处理方向）
+3. 提取窗口内 fragment_num / fragment 启动量（确认 fanout 规模）
+4. 检索排除 pthread_create / pids / FD 耗尽信号
+5. 确认 `parallel_exchange_instance_num` / `parallel_pipeline_task_num` 当前值
+6. 历史同类案例检索：`failed to send brpc when exchange, [E11]` 问题族
diff --git a/skills/doris-debug/resource-isolation/SKILL.md
b/skills/doris-debug/resource-isolation/SKILL.md
new file mode 100644
index 0000000..69031df
--- /dev/null
+++ b/skills/doris-debug/resource-isolation/SKILL.md
@@ -0,0 +1,111 @@
+---
+name: doris-debug-resource-isolation
+description: >
+ Use for Doris Workload Group / resource tag queue starvation, CPU/memory
isolation
+ leaks, and workload policy debugging. Commands: SHOW WORKLOAD GROUPS,
EXPLAIN resource.
+version: 0.2.0
+category: resource-isolation
+---
+
+# Resource Isolation (Workload Group)
+
+## Causes
+
+| ID | Cause | Evidence | Source anchor |
+|----|-------|----------|---------------|
+| A | Queue starvation | Queries queued in `SHOW WORKLOAD GROUPS`; `QueueTime`
growing | `WorkloadGroupMgr.java` |
+| B | CPU oversubscription | One WG consuming >90% CPU despite limits |
`CgroupCpuCtl.cpp` |
+| C | Memory leak across groups | MemTracker shows one WG exceeding
`memory_limit` | `WorkloadGroupMemMgr.cpp` |
+| D | Tag routing wrong | Query hits wrong BE set; ResourceTag mismatch |
`ResourceTag.java`, `Tag.java` |
+| E | WG not active / cgroup missing | `CREATE WORKLOAD GROUP` had no effect |
`WorkloadGroupMgr.cpp` cgroup init |
+
+## 10 min triage
+
+```sql
+-- Workload group status
+SHOW WORKLOAD GROUPS\G
+-- Check columns: Id, Name, ActiveQueries, QueuedQueries, CpuShares,
MemoryLimit
+
+-- Which WG is my query in?
+SELECT * FROM information_schema.workload_group_resource_usage;
+
+-- Workload policies (routing rules)
+SHOW WORKLOAD POLICY\G
+```
+
+```bash
+# BE cgroup inspection
+cat /sys/fs/cgroup/cpu/doris/<wg_id>/cpu.shares
+cat /sys/fs/cgroup/memory/doris/<wg_id>/memory.limit_in_bytes
+
+./scripts/doris-debug be-metrics --be http://$BE:8040 --grep "workload_group"
+```
+
+## Cause A — Queue starvation
+
+```sql
+-- See who's queued
+SHOW WORKLOAD GROUPS\G
+
+-- Raise queue concurrency
+ALTER WORKLOAD GROUP etl_wg SET (
+ "max_concurrency" = "8",
+ "max_queue_size" = "100",
+ "queue_timeout" = "300"
+);
+```
+
+Check `be/src/pipeline/task_scheduler.cpp` — the pipeline scheduler respects
cgroup shares; if a high-share WG starves a low-share one, the low-share WG's
queue backs up.
+
+## Cause B — CPU oversubscription
+
+```sql
+-- Create a hard CPU cap
+ALTER WORKLOAD GROUP report_wg SET (
+ "cpu_share" = "1024",
+ "cpu_hard_limit" = "200%" -- 2 cores max
+);
+
+-- Force query into specific WG
+SET workload_group = 'report_wg';
+```
+
+cgroup v1 vs v2 behavior differs. Confirm which is active on BE hosts:
+```bash
+mount | grep cgroup
+ls /sys/fs/cgroup/cpu/doris/ # cgroup v1
+ls /sys/fs/cgroup/doris/ # cgroup v2
+```
+
+## Cause D — Tag routing
+
+```sql
+-- Check BE tags
+SHOW BACKENDS\G -- look at Tag column
+
+-- Check WG tag constraint
+SHOW WORKLOAD GROUPS\G -- look at ResourceTag
+```
+
+If a workload group specifies a tag but no BEs carry that tag, queries queue
forever. Fix:
+```sql
+ALTER SYSTEM MODIFY BACKEND "host:9050" SET ("tag.tag.location" = "high_mem");
+```
+
+## Enabling cgroup on BE
+
+```properties
+# be.conf
+doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris
+enable_workload_group = true
+```
+
+Without cgroup configured, `cpu_share` is a cooperative hint, not a hard
guarantee.
+
+## Source
+
+- `be/src/common/config.cpp` — `enable_workload_group`, cgroup paths
+- `be/src/pipeline/task_scheduler.cpp` — pipeline scheduling
+- `fe/.../workloadgroup/WorkloadGroupMgr.java` — WG lifecycle
+- `fe/.../resource/Tag.java` — resource tags
+- `be/src/util/cgroup_util.cpp` — cgroup v1/v2 adapter
diff --git
a/skills/doris-debug/resource-isolation/references/case-001-workload-policy-cme.md
b/skills/doris-debug/resource-isolation/references/case-001-workload-policy-cme.md
new file mode 100644
index 0000000..77881ca
--- /dev/null
+++
b/skills/doris-debug/resource-isolation/references/case-001-workload-policy-cme.md
@@ -0,0 +1,106 @@
+---
+type: reference
+category: resource-isolation
+keywords: [ConcurrentModificationException, session variable, one-shot,
setVarOnce, workload policy, HashMap concurrent]
+---
+
+# Case-001: 查询偶发 ConcurrentModificationException — Session Variable One-Shot
回滚异常
+
+## Environment
+
+- Doris version: 4.1.7 (cloud)
+- Architecture: storage-compute separation
+- Nereids optimizer enabled
+
+## Symptom
+
+查询偶发 `java.util.ConcurrentModificationException`，概率低但影响面大。
+
+关键现象：SQL 实际已执行完成（`Query finished` 已打印，`ReturnRows=1`），但 audit State 被置为
`ERR`，客户端看到异常。
+
+## Key evidence
+
+### FE 日志
+
+```
+StatsCalculator.disableJoinReorderIfStatsInvalid():
+ disable join reorder since row count not available:
+ internal.data_warehouse_dws.mv_freshness_bet_overview_account_10m
+```
+
+表统计信息 row_count=0，触发 Nereids `setVarOnce(disable_join_reorder=true)`。
+
+```
+Query ... finished. ReturnRows=1, Time(ms)=6, ScanRows=0
+```
+
+SQL 已执行完成。
+
+```
+java.util.ConcurrentModificationException
+ at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java)
+ at java.base/java.util.HashMap$EntryIterator.next(HashMap.java)
+ at java.base/java.util.HashMap$EntryIterator.next(HashMap.java)
+ at VariableMgr.revertSessionValue()
+```
+
+在 finally 回滚 session variable 时 HashMap 并发修改报错。
+
+## Investigation
+
+### Step 1: 确认 CME 触发路径
+
+1. 查询扫描的表 `row_count=-1`（统计信息不可用）
+2. `StatsCalculator.disableJoinReorderIfStatsInvalid()` 调用
`SessionVariable.setVarOnce(disable_join_reorder=true)`
+3. `setVarOnce()` 把原始值写入 `sessionOriginValue`（`HashMap`）
+4. SQL 执行完成后，`StmtExecutor.execute()` finally 调用
`VariableMgr.revertSessionValue()`
+5. `revertSessionValue()` 直接遍历 `sessionOriginValue.keySet()` 的 live keySet
+6. 同一 `SessionVariable` 的 `sessionOriginValue` 在遍历期间被其他路径并发写入/清理 → `HashMap`
fail-fast → CME
+
+### Step 2: 代码核对（4.1.7）
+
+```java
+// SessionVariable.java
+public Map<SessionVariableField, String> sessionOriginValue = new HashMap<>();
+
+// VariableMgr.java
+public static void revertSessionValue(SessionVariable obj) {
+ Map<SessionVariableField, String> sessionOriginValue =
obj.getSessionOriginValue();
+ if (!sessionOriginValue.isEmpty()) {
+ for (SessionVariableField field : sessionOriginValue.keySet()) {
+ // 直接遍历 live HashMap keySet，无 snapshot，无锁
+ setValue(obj, field, sessionOriginValue.get(field));
+ }
+ }
+}
+```
+
+### Step 3: 影响范围
+
+不只是 Nereids 统计信息路径。所有 `setVarOnce` 路径（runtime filter wait time、auto analyze
等）都共享同一风险面。
+
+### Step 4: 历史相关
+
+历史有 `apache/doris#48239`（ExportTaskExecutor clone SessionVariable），但只覆盖 Export
场景，不覆盖普通 SELECT 的 `StmtExecutor → revertSessionValue` 路径。
+
+## Root Cause
+
+`VariableMgr.revertSessionValue()` 直接遍历 live `HashMap.keySet()`（无
snapshot、无锁）。当 `sessionOriginValue` 在遍历期间被其他路径并发修改时，`HashMap` iterator
fail-fast 抛出 CME。
+
+触发条件：Nereids 优化器因统计信息不可用而 `setVarOnce(disable_join_reorder=true)`，随后在 finally
回滚时 `sessionOriginValue` 被并发修改。
+
+## Fix
+
+- **短期规避**: 手动对涉及表执行 `ANALYZE TABLE` 收集统计信息，避免触发 `row_count=-1 → setVarOnce`
路径。但这只能规避当前触发点，不修复底层 bug。
+- **代码修复**:
+ 1. `VariableMgr.revertSessionValue()` 先 snapshot `entrySet` 再遍历（`new
ArrayList<>(sessionOriginValue.entrySet())`）
+ 2. 对 `sessionOriginValue` 的读写加同步边界
+ 3. `setIsSingleSetVar(false) / clearSessionOriginValue()` 在 finally
中执行，避免异常后残留
+- **状态修复**: `StmtExecutor` finally 当前只 catch `DdlException`，应保护 cleanup
异常避免覆盖已成功完成的查询状态
+
+## Key diagnostic actions
+
+1. fe.log 搜索 `ConcurrentModificationException` → 确认栈在 `revertSessionValue`
+2. 确认同一连接是否多线程并发使用（加速 CME 暴露）
+3. 查看 `SHOW TABLE STATS` 确认 row_count 是否正常
+4. `ANALYZE TABLE` 作为临时规避（仅规避统计信息路径，不修复根因）
diff --git a/skills/doris-debug/shared/guides/cascade-import-wal-versions.md
b/skills/doris-debug/shared/guides/cascade-import-wal-versions.md
new file mode 100644
index 0000000..0409534
--- /dev/null
+++ b/skills/doris-debug/shared/guides/cascade-import-wal-versions.md
@@ -0,0 +1,13 @@
+# 级联：高吞吐 async Group Commit ↔ WAL ↔ 版本
+
+```
+group_commit:async_mode 高 MB/s
+ → 接收 BE 本地 WAL 上涨
+ → group commit 吞吐 < 写入
+ → version_count → max_tablet_version_num → TOO_MANY_VERSION
+ → Compaction / 查询恶化
+```
+
+Skill：`import` → `compaction`
+代码：`group_commit_mgr.cpp`、`rowset_builder.cpp`
+工具：`./scripts/doris-debug wal-du <storage_root>`
diff --git a/skills/doris-debug/shared/guides/cascade-query-timeout-exchange.md
b/skills/doris-debug/shared/guides/cascade-query-timeout-exchange.md
new file mode 100644
index 0000000..6b6bba5
--- /dev/null
+++ b/skills/doris-debug/shared/guides/cascade-query-timeout-exchange.md
@@ -0,0 +1,13 @@
+# 级联：查询超时 ↔ Exchange / brpc
+
+```
+query_timeout
+ → Profile EXCHANGE WaitForData ≈ 超时
+ → SINK PendingFinish / RpcCount=0
+ → be.WARNING: failed to send brpc when exchange | E1008 @:8060
+ → SHOW BACKENDS Alive=true（易误判）
+```
+
+Skill：`query`（Cause D）
+代码：`exchange_sink_buffer.h`、`reset_rpc_channel_action.cpp`
+止血：`curl http://be:8040/api/reset_rpc_channel/all`（仅 internal cache）
diff --git a/skills/doris-debug/shared/references/01-common-commands.md
b/skills/doris-debug/shared/references/01-common-commands.md
new file mode 100644
index 0000000..a70597a
--- /dev/null
+++ b/skills/doris-debug/shared/references/01-common-commands.md
@@ -0,0 +1,21 @@
+# Doris 公共诊断命令
+
+端口默认（以 conf 为准）：FE HTTP `8030` / MySQL `9030`；BE HTTP `8040` / brpc `8060`。
+
+```sql
+SHOW FRONTENDS\G
+SHOW BACKENDS\G
+SHOW PROC '/backends';
+SHOW PROC '/cluster_health/tablet_health';
+SHOW LOAD ORDER BY CreateTime DESC LIMIT 30;
+SHOW TABLETS FROM db.tbl;
+ADMIN SHOW REPLICA STATUS FROM db.tbl;
+SET enable_profile = true;
+```
+
+```bash
+# CLI helpers from this repo
+./scripts/doris-debug be-metrics --be http://$BE:8040
+./scripts/doris-debug log-grep $BE_LOG_DIR --pack exchange --query-id "$QID"
+curl -s "http://$BE:8040/api/reset_rpc_channel/all";
+```
diff --git a/skills/doris-debug/shared/references/02-source-map.md
b/skills/doris-debug/shared/references/02-source-map.md
new file mode 100644
index 0000000..59edea5
--- /dev/null
+++ b/skills/doris-debug/shared/references/02-source-map.md
@@ -0,0 +1,37 @@
+# Doris Source Map（排障 ↔ 源码）
+
+> **Baseline version**: Apache Doris 2.1 / 3.0 (2025). Source paths verified
against `apache/doris` main branch circa 2025-07.
+> For older versions (1.2, 2.0), paths may differ — check your deployed
branch's `be/src/common/config.cpp` and `fe/.../SessionVariable.java`.
+
+| 现象 | 源码位置 | 版本提示 | Skill |
+|------|----------|----------|-------|
+| `failed to send brpc when exchange` |
`be/src/exec/operator/exchange_sink_buffer.h` | 2.1+ pipeline engine | query |
+| `reset_rpc_channel` 只清 internal cache |
`be/src/service/http/action/reset_rpc_channel_action.cpp` | all versions |
query |
+| brpc stub hand_shake / erase cache | `be/src/runtime/fragment_mgr.cpp`
(`_check_brpc_available`) | 2.0+ | query |
+| `enable_brpc_connection_check` | `be/src/common/config.cpp` | 2.1+ | query |
+| Group Commit finish + `delete_wal` |
`be/src/load/group_commit/group_commit_mgr.cpp` | 2.1+ (group_commit is 2.1
feature) | import |
+| `group_commit_*` BE 配置 | `be/src/common/config.cpp` (~1407+) | 2.1+ | import
|
+| TOO_MANY_VERSION / `-235` 文案 | `be/src/storage/rowset_builder.cpp`
`check_tablet_version_count` | all versions | compaction / import |
+| `max_tablet_version_num` default 2000 | `be/src/common/config.cpp` | all
versions | compaction |
+| WaitForData counter | exchange source operator Profile | pipeline engine
(2.0+) | query |
+| Nereids timeout session | `fe/.../qzone/SessionVariable.java`
`nereids_timeout_second` | 2.1+ Nereids | query |
+| MTMV 服务 | `fe/.../mtmv/MTMVService.java` | 2.1+ | materialized-view |
+| Sync MV alter | `fe/.../alter/MaterializedViewHandler.java` | all versions |
materialized-view |
+| Clone worker pool | `be/src/agent/task_worker_pool.cpp` | all versions |
tablet |
+| Tablet health proc | `fe/.../master/TabletHealth.java` | all versions |
tablet |
+| cgroup CPU control | `be/src/util/cgroup_util.cpp` | 2.1+ |
resource-isolation |
+| Workload Group lifecycle | `fe/.../workloadgroup/WorkloadGroupMgr.java` |
2.1+ | resource-isolation |
+| File cache (cloud) | `be/src/io/cache/whole_file_cache.cpp` | 3.0+ (cloud) |
cloud |
+| Compute Group | `fe/.../cloud/ComputeGroupMgr.java` | 3.0+ (cloud) | cloud |
+| S3 connector | `be/src/io/fs/s3/S3FileSystem.cpp` | all versions | data-lake
/ cloud |
+| MemTrackerLimiter | `be/src/runtime/memory/mem_tracker_limiter.cpp` | all
versions | node |
+| BE startup / port bind | `be/src/service/doris_main.cpp` | all versions |
deployment |
+| FE bootstrap | `fe/.../master/Env.java` | all versions | deployment |
+
+### Upgrade caveats
+
+- **1.2 → 2.0**: Pipeline engine became default. Old non-pipeline profiles
lack `WaitForData`.
+- **2.0 → 2.1**: Group Commit async_mode introduced. `group_commit_*` config
keys not present in 2.0.
+- **2.1 → 3.0**: Cloud mode / compute groups introduced.
`meta_service_endpoint` and `file_cache_path` replace some shared-nothing
config.
+
+排障结论应能在源码中找到对应错误字符串或配置项；如果找不到，先确认 Doris 版本和分支。
diff --git a/skills/doris-debug/shared/references/03-cir-case-index.md
b/skills/doris-debug/shared/references/03-cir-case-index.md
new file mode 100644
index 0000000..5fcc270
--- /dev/null
+++ b/skills/doris-debug/shared/references/03-cir-case-index.md
@@ -0,0 +1,478 @@
+# Doris 生产案例索引（Production Case Index）
+
+> 从 20,000+ 线上排障记录中筛选 43 个代表性案例，按 doris-debug skill 分类。
+> 每个案例含 Symptom / Root cause / Fix / 关键诊断动作。
+
+## 统计概览
+
+| 诊断域 | 案例数 | 关键模式 |
+|--------|--------|----------|
+| Query | 7 | Plan Time / Scan 瓶颈 / Exchange E11 / 结果跳变 / Overwrite 冲突 / CPU
异常 / 计算组差异 |
+| Import | 6 | 307 重定向 / FE OOM 泄露 / S3 load 慢 / RPC timeout / Arrow 格式 /
meta-service 慢 |
+| Compaction | 5 | SC 残留 base score / cumu segfault / 版本 compacted / cumu
score 高 / inverted index 重建 |
+| Node | 7 | Auto analyze OOM / file_cache 泄漏 / BE coredump / FE bdbje 损坏 / BE
内存持续泄漏 / CPU 异常 / 启动 200GB |
+| Tablet | 3 | 统计异常 / S3 冷热分离 / 写入超时 |
+| Deployment | 2 | FE NPE 启动 / BE 配置无法启动 |
+| Data Lake | 5 | Glue pagination / Iceberg 分区表 / Plan Time Iceberg / 谓词下推缺失 /
View struct |
+| Resource Isolation | 5 | Spill disk / cgroup memory / Workload policy CME /
Compute group 差异 / WG 绑定 |
+| Cloud | 5 | Plan 竞态 / BE 资源不释放 / Warmup cache miss / File cache 内存泄漏 / FDB
堆积 |
+
+---
+
+## Query（查询诊断）— 7 个案例
+
+### Case Q1: FE Nereids Plan Time 10min（Iceberg 外表 RPC 爆炸）
+
+- **Severity**: Medium
+- **Doris 版本**: 4.1.3
+- **Symptom**: 查询总耗时 10min8s，`Plan Time=10min8sec`，BE 实际执行只有 214ms/1MB 数据
+- **Root cause**: Nereids `Finalize Scan Node → Create Scan Range` 阶段每次
`getManifestFiles` 调用 `UpdateRunningStatus` 触发 `msClient.updateInstance` RPC，64
文件 × 63 分区反复调用，每次 ~200ms → 10min+
+- **Fix**: 短期：调大 `iceberg_manifest_cache_refresh_interval_s`；长期：改进 scan range
创建减少重复 RPC
+- **Evidence**: Profile `Create Scan Range Time=10min1sec`, `Get Splits
Time=6sec`, `Is Nereids=Yes`
+- **关键诊断动作**: Profile → Plan Time 占比 → 拆解 Nereids Translate / Finalize Scan
Node / Create Scan Range
+
+### Case Q2: CPU 使用率飙升（Inverted Index 重建）
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1→3.0 升级
+- **Symptom**: 无查询无导入时所有 BE CPU 100%
+- **Root cause**: Doris 2.1→3.0 升级后复合倒排索引格式不兼容，BE 启动后对所有 tablet 重建索引
+- **Fix**: 新版本跳过不兼容的复合索引重建；临时 `SET disable_auto_compaction = true`
+- **Evidence**: be/log `begin to write inverted index`, CPU profile
`FSWriteTime + CompressTime > 80%`
+- **关键诊断动作**: `top -H` → be/log `inverted index` → `SHOW PROC '/backends'`
CompactionScore
+
+### Case Q3: [E11] Resource temporarily unavailable（Exchange brpc fanout）
+
+- **Severity**: High
+- **Doris 版本**: 4.0.7
+- **Symptom**: 高峰期查询报 `failed to send brpc when exchange, error=[E11]Resource
temporarily unavailable`
+- **Root cause**: `parallel_exchange_instance_num=100` 在高并发/高 fanout 查询下
exchange RPC 发送侧打到 EAGAIN，brpc socket 发送队列满
+- **Fix**: 调小 `parallel_exchange_instance_num`；长期 backport apache/doris PR
#50113 (one rpc send multi blocks)
+- **Evidence**: be/log `RPC meet failed: [E11]`、`failed to send brpc when
exchange @<target-be>:8060`
+- **关键诊断动作**: 1) 区分 E11 brpc vs pthread_create EAGAIN（后者才是线程池打满）2) 统计目标 BE
分布（单点 vs 全集群）3) 确认 `parallel_exchange_instance_num` 配置
+
+### Case Q4: 同一查询在不同计算组耗时差异大（Scan Open/Init 阶段）
+
+- **Severity**: Medium
+- **Doris 版本**: 4.0.8
+- **Symptom**: 同一 SQL 在快组 284ms，慢组 3sec257ms，两边 rows/bytes 相同（均返回 0 行）
+- **Root cause**: 慢计算组的 OLAP scan `OpenTime=2sec647ms`，在真正读数据前需要同步更多远端 rowset
metadata / delete bitmap（cloud 模式同步开销）
+- **Evidence**: 慢 Profile `ScannerInitTime=2sec647ms`, 快 Profile
`ScannerInitTime=<1ms`
+- **关键诊断动作**: 1) Profile 对比 scan open/init 阶段 2) 确认 ScannerInitTime/OpenTime
差异 3) 检查计算组缓存状态
+
+### Case Q5: Overwrite 报错分区冲突
+
+- **Severity**: High
+- **Doris 版本**: 4.0.4
+- **Symptom**: INSERT OVERWRITE t PARTITION(p1) SELECT ... WHERE p2=... →
"partitions conflict"
+- **Root cause**: PARTITION 子句取 p1 但 WHERE 条件匹配到 p2，代码未统一分区来源
+- **Fix**: 统一分区来源（只用 PARTITION 子句或只用 WHERE 条件），4.0.5+ 修复
+- **关键诊断动作**: EXPLAIN 确认实际扫描/写入分区 → 对比 PARTITION 子句 vs WHERE 条件
+
+### Case Q6: 查询结果跳变
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1.7
+- **Symptom**: 同一查询多次执行返回不同结果
+- **Root cause**: tablet 统计信息不正确，CBO 在不同执行间选了不同的 scan 路径
+- **关键诊断动作**: EXPLAIN 对比两次 plan → `ADMIN CHECK TABLET`
+
+### Case Q7: Match Phrase vs Like 结果不一致
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1.12
+- **Symptom**: `MATCH_PHRASE_PREFIX('xx')` 和 `LIKE 'xx%'` 对相同数据返回不同结果
+- **Root cause**: Inverted Index 分词器与 LIKE 的字符匹配语义差异
+- **关键诊断动作**: 1) EXPLAIN 确认是否命中 inverted index 2) `SHOW INDEX` 确认索引定义
+
+---
+
+## Import（导入诊断）— 6 个案例
+
+### Case I1: Stream Load HTTP 307 重定向失败
+
+- **Severity**: High
+- **Doris 版本**: 4.0.8
+- **Symptom**: Stream Load 返回 307，Location header 指向的 BE IP 客户端不可达
+- **Root cause**: FE 返回 redirect BE 内网 IP，客户端在外网
+- **Fix**: `enable_redirect_strict_check = false` + 确保 BE IP 客户端可达
+- **关键诊断动作**: `curl -v` 查看 307 Location → 确认 IP 可达性 → `SHOW BACKENDS`
+
+### Case I2: INSERT INTO job 不清理 → FE OOM
+
+- **Severity**: Critical
+- **Doris 版本**: 26.0.3
+- **Symptom**: FE 内存持续增长直至 OOM，重启后 reload 历史 INSERT job 再次 OOM
+- **Root cause**: INSERT INTO job 标记 FINISHED 后 `JobManager` 未清理，累积数千个 job
+- **Fix**: FE 代码改进 job 清理；临时 `jmap -histo` 确认 InsertJob 堆积
+- **关键诊断动作**: `SHOW LOAD` / `SHOW INSERT` → `jmap -histo` → `jmap -dump:live`
+
+### Case I3: Broker Load S3 list 慢
+
+- **Severity**: High
+- **Doris 版本**: 26.0.3
+- **Symptom**: Broker Load 从 S3 导入耗时远超预期，list 操作耗时 > 下载
+- **Root cause**: S3 prefix 文件数过多 + `s3_max_connections` 过小
+- **Fix**: 增加 `s3_max_connections`，合并小文件后再导入
+- **关键诊断动作**: S3 list 耗时 → BE log s3 → `s3_max_connections`
+
+### Case I4: 导入 RPC timed out（heavy work pool 打满）
+
+- **Severity**: Critical
+- **Doris 版本**: 4.0.9
+- **Symptom**: 导入 open tablet writer 阶段等待目标 BE 返回 RPC 超过 60s，随后 coordinator
cancel 整条 load
+- **Root cause**: 目标 BE `brpc_heavy` work pool 在该时段被打满/卡住，`tablet_writer_open`
进入 heavy pool 后执行过慢或排队超时
+- **Code path**: `VNodeChannel::_open_internal` →
`PBackendService_Stub::tablet_writer_open(timeout=60s)` →
`PInternalService::tablet_writer_open` → `_heavy_work_pool.try_offer`
+- **Fix**: 短期降低/错峰 load 并发；中期 `brpc_heavy_work_pool_threads 256→384`；故障时保留
pstack 再重启
+- **关键诊断动作**: 1) be/log `RPC call is timed out` 目标 BE 2) 确认同一窗口是否成簇出现 3) 检查
`fail to offer request to the work pool` 签名 4) pstack 卡点
+
+### Case I5: Arrow 格式 Stream Load 报错
+
+- **Severity**: Critical
+- **Doris 版本**: 2.1+
+- **Symptom**: Arrow 格式 Stream Load 报 schema 解析错误
+- **Root cause**: Arrow IPC 格式与 Doris 预期的 Arrow schema 字段类型映射不一致
+- **关键诊断动作**: Arrow schema 验证 → BE log arrow → 对比支持的 Arrow 类型映射
+
+### Case I6: meta-service 导致导入慢
+
+- **Severity**: High
+- **Doris 版本**: cloud 4.0+
+- **Symptom**: 导入耗时显著增长，与数据量不成比例
+- **Root cause**: meta-service 在导入 publish 阶段延迟高
+- **关键诊断动作**: fe.log publish 阶段耗时 → meta-service latency 监控
+
+---
+
+## Compaction（合并诊断）— 5 个案例
+
+### Case C1: Schema Change 残留 → Base Score 高
+
+- **Severity**: Medium
+- **Doris 版本**: cloud 4.0+
+- **Symptom**: base compaction score max 1490 / avg 1370，cumu score 正常 (max 79)
+- **Root cause**: SC 操作创建大量 base compaction tablet 但不清理 SC 上下文，`tablet_state`
或 SC 状态阻止执行
+- **Evidence**: `get_topn_compaction_score ... type=1` → score 高，但 `type=2` 正常
+- **Fix**: 清理残留 SC 上下文 + 检查 compaction 过滤条件
+- **关键诊断动作**: 区分 base(type=1) vs cumu(type=2) → `SHOW ALTER TABLE` → be/log
过滤逻辑
+
+### Case C2: Cumu Compaction Segfault
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1.10
+- **Symptom**: BE 执行 cumulative compaction 时 SIGSEGV core dump
+- **Root cause**: 特定 rowset 合并场景下 segment 元数据读取越界 → 空指针
+- **Fix**: 修复 rowset reader 边界检查（patch 已合入）
+- **关键诊断动作**: `dmesg` → GDB backtrace `compaction.cpp:XXX` → 提取触发 tablet →
rowset 列表
+
+### Case C3: Versions Already Compacted（compaction 竞态）
+
+- **Severity**: High
+- **Doris 版本**: 2.1+
+- **Symptom**: compaction 提交报 `[E-230]versions are already compacted,
version_range=[X-Y]`
+- **Root cause**: 多个 compaction 任务竞争同一 tablet，提交时另一任务已处理相同版本范围
+- **Fix**: 提交前二次检查版本范围有效性
+- **关键诊断动作**: be/log `already compacted` → `SHOW TABLET <id>` 版本列表 → 任务调度重叠检查
+
+### Case C4: Cumu Score 高但 Base 正常
+
+- **Severity**: Medium
+- **Doris 版本**: cloud 4.0+
+- **Symptom**: cumu score 持续升高，base score 正常
+- **Root cause**: 高频写入产生大量 cumulative point，cumu compaction 吞吐不足
+- **Fix**: 调大
`max_cumulative_compaction_threads`、`compaction_task_num_per_disk`
+- **关键诊断动作**: `be-metrics --grep compaction`→ `iostat -x 1` 磁盘 IO → 对比 base vs
cumu score
+
+### Case C5: Base Score 高但找不到对应 Tablet
+
+- **Severity**: Medium
+- **Doris 版本**: 3.0.10
+- **Symptom**: Grafana base score 告警高，但 `get_topn_tablets_to_compact()` 没有高分
tablet
+- **Root cause**: cloud 调度更新了 `tablet_base_max_compaction_score` 指标但不 pick
tablet（slot/状态/过滤条件不满足）
+- **关键诊断动作**: 区分 "score 更新但未调度" vs "有候选但不允许执行" → 检查 slot 和过滤条件
+
+---
+
+## Node（节点健康诊断）— 7 个案例
+
+### Case N1: Auto Analyze → OOM
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1+
+- **Symptom**: `auto_analyze` 开启后 BE OOM kill
+- **Root cause**: auto_analyze 全表扫描收集统计信息时不受 `mem_limit` 约束
+- **Fix**: 限制 `auto_analyze_table_sample_percent` + mem_limit 保护
+- **关键诊断动作**: be/log MemTrackerLimiter → `SHOW ANALYZE STATUS`
+
+### Case N2: File Cache 内存泄漏
+
+- **Severity**: High
+- **Doris 版本**: 4.1.2
+- **Symptom**: BE RSS 持续增长，无查询也不释放，file_cache 占用持续增长
+- **Root cause**: file_cache LRU evict 在高命中率时失效
+- **Fix**: 修复 LRU evict + 设置 `file_cache_query_limit`
+- **关键诊断动作**: `be-metrics --grep file_cache` → RSS vs MemTracker → `jeprof`
+
+### Case N3: BE 持续 Coredump
+
+- **Severity**: Critical
+- **Doris 版本**: 4.0.4
+- **Symptom**: 同一 BE 反复 core dump 重启
+- **Root cause**: 特定查询/导入触发 segment reader 内存越界或 use-after-free
+- **关键诊断动作**: `dmesg` / `coredumpctl` → GDB backtrace → 提取 query_id / load_id
→ 回放触发
+
+### Case N4: FE BDBJE 损坏
+
+- **Severity**: Critical
+- **Doris 版本**: all versions
+- **Symptom**: FE 启动 `java.io.FileNotFoundException:
doris-meta/bdb/0000014d.jdb`
+- **Root cause**: BDBJE 日志文件损坏/丢失（磁盘满或异常关机）
+- **Fix**: 从健康 Follower/Observer rsync doris-meta；Master 无副本 → BDBJE recovery
工具
+- **关键诊断动作**: `ls fe/doris-meta/bdb/` → `SHOW FRONTENDS` → rsync
+
+### Case N5: 单台 BE 内存持续泄漏 + CPU 异常
+
+- **Severity**: High
+- **Doris 版本**: 4.1.3
+- **Symptom**: 单 BE RSS 持续线性增长 + CPU 也随之升高，其他 BE 正常
+- **Root cause**: 疑似特定查询的 fragment 未释放 + 持续重试；cpu 异常是内存回收压力导致
+- **关键诊断动作**: `jeprof` 分配热点 → `SHOW PROC '/current_queries'` → be/log
MemTracker → 隔离节点对比
+
+### Case N6: BE 启动即占 200GB 内存
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1.7
+- **Symptom**: BE 刚启动，无查询，RSS 200GB+
+- **Root cause**: `mem_limit=80%` 但 `storage_page_cache_limit` 等未显式设置，默认分配大量
page cache
+- **Fix**: 显式设置 `storage_page_cache_limit` + `max_segment_cache_size`
+- **关键诊断动作**: RSS vs mem_limit → `be-metrics --grep cache` → be.conf 缓存配置
+
+### Case N7: 压测场景内存泄漏
+
+- **Severity**: Medium
+- **Doris 版本**: 4.1.7
+- **Symptom**: YCSB 高频短查询场景下 BE 内存持续上升不回落
+- **Root cause**: 高频短查询产生的 fragment context 回收不及时
+- **关键诊断动作**: `jeprof --inuse_space` → 对比压测前后 → fragment 对象计数
+
+---
+
+## Tablet（Tablet/副本诊断）— 3 个案例
+
+### Case T1: Tablet 统计信息异常
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1+
+- **Symptom**: `SHOW TABLET` / `SHOW DATA` 显示的 tablet 大小/行数与实际不符
+- **关键诊断动作**: `SHOW TABLET <id>` → `ADMIN DIAGNOSE TABLET <id>` → `SHOW PROC
'/statistic'`
+
+### Case T2: S3 冷热分层数据未清理
+
+- **Severity**: High
+- **Doris 版本**: 2.1.7
+- **Symptom**: cooldown 到 S3 的数据在 S3 侧未被删除，存储成本持续增长
+- **Root cause**: cooldown 策略只迁移不删除，expired 清理任务被后台调度延迟
+- **关键诊断动作**: `SHOW EXPIRED POLICY` → S3 listing vs 本地 tablet → `SHOW PROC
'/trash'`
+
+### Case T3: 写入超时 + EPOLLOUT 失败
+
+- **Severity**: Low（但影响写入）
+- **Doris 版本**: 3.1.3
+- **Symptom**: 写入偶尔 1 分钟+，日志大量 `Fail to wait EPOLLOUT of fd=XXX: Connection
timed out`
+- **Root cause**: brpc socket 发送缓冲区满或对端接收慢 → TCP send buffer 拥塞
+- **关键诊断动作**: `netstat -s` 重传统计 → TCP send queue 检查 → 对端 BE 负载
+
+---
+
+## Deployment（部署诊断）— 2 个案例
+
+### Case D1: FE NPE 无法启动
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1+
+- **Symptom**: FE 启动时 `NullPointerException` 无法完成 bootstrap
+- **关键诊断动作**: fe.log stacktrace → `fe/doris-meta/image/` 文件完整性 → fe.conf 变更历史
+
+### Case D2: 新配置 BE 无法启动
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1.8
+- **Symptom**: 新加入集群的 BE 启动失败
+- **Root cause**: BE 配置中的 `storage_root_path` 权限/路径问题或 `priority_networks` 绑定失败
+- **关键诊断动作**: be.out 启动日志 → 端口占用 `fuser` → `priority_networks` 配置 → 目录权限
+
+---
+
+## Data Lake（数据湖诊断）— 5 个案例
+
+### Case DL1: Glue Catalog Pagination 丢失 Database
+
+- **Severity**: Medium
+- **Doris 版本**: 26.0.3
+- **Symptom**: `SHOW DATABASES FROM glue_catalog` 只返回部分 database
+- **Root cause**: Glue API `getAllDatabases` 未正确处理 `NextToken` pagination
+- **Fix**: 修复 pagination 逻辑
+- **关键诊断动作**: 对比 Glue API 直接调用 vs Doris SHOW DATABASES → fe.log `GlueCatalog`
+
+### Case DL2: Iceberg 分区表无法识别（Temporal Transforms）
+
+- **Severity**: High
+- **Doris 版本**: 26.0.3
+- **Symptom**: `SHOW PARTITIONS FROM iceberg_catalog.db.tbl` 返回空
+- **Root cause**: Iceberg `year/month/day` temporal partition transforms 不被
Doris 支持
+- **Fix**: 增加 temporal partition transform 支持
+- **关键诊断动作**: Iceberg `SHOW CREATE TABLE` → 确认 `partitioning` → fe.log
`IcebergTable`
+
+### Case DL3: Iceberg INSERT 耗时较长
+
+- **Severity**: High
+- **Doris 版本**: 2.1+
+- **Symptom**: INSERT INTO iceberg_catalog.db.tbl 耗时远超预期
+- **Root cause**: Iceberg commit 阶段与 catalog
的多次交互（snapshot/commit/expireSnapshots）
+- **关键诊断动作**: Profile INSERT 各阶段耗时 → Iceberg commit metrics →
`iceberg_commit_batch_size`
+
+### Case DL4: JDBC 外表与内表 JOIN 无谓词下推
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1+
+- **Symptom**: JDBC catalog 外表 JOIN Doris 内表时 WHERE 条件未下推到外表
+- **Root cause**: CBO 未识别 JDBC catalog 的谓词下推能力（`push_down_predicates`
capability）
+- **Fix**: 添加 JDBC catalog 谓词下推适配
+- **关键诊断动作**: EXPLAIN VERBOSE → 确认 scan 阶段无 pushdown → fe.log catalog
capability
+
+### Case DL5: MV 基表结构变更后 EXPLAIN 报错
+
+- **Severity**: Medium
+- **Doris 版本**: 2.1+
+- **Symptom**: `EXPLAIN SELECT ...` 报 "View original struct info is invalid"
+- **Root cause**: MV 定义中引用的基表列已变更（ALTER TABLE DROP/CHANGE COLUMN），MV 元数据未同步
+- **关键诊断动作**: `SHOW CREATE MATERIALIZED VIEW` vs `SHOW CREATE TABLE` → 对比列差异
+
+---
+
+## Resource Isolation（资源隔离诊断）— 5 个案例
+
+### Case R1: Spill Disk 不生效
+
+- **Severity**: High
+- **Doris 版本**: 4.0+
+- **Symptom**: `enable_spill=true` 后查询仍然 OOM
+- **Root cause**: spill 只对 agg/join operator 生效；spill 触发阈值与 mem_limit 关系不清
+- **关键诊断动作**: Profile `SpillDataSize` → `be-metrics --grep spill` → 确认触发
operator
+
+### Case R2: MEM_LIMIT_EXCEEDED 但系统内存充足（cgroup）
+
+- **Severity**: High
+- **Doris 版本**: 2.1+
+- **Symptom**: `mem_limit=80%` 但导入报 MEM_LIMIT_EXCEEDED，`free` 显示还有大量可用内存
+- **Root cause**: `/proc/meminfo MemAvailable` 在 cgroup 容器中包含可回收 page
cache，值不准确
+- **Fix**: 改用 cgroup `memory.limit_in_bytes - memory.usage_in_bytes`
+- **关键诊断动作**: `free -h` vs cgroup memory.limit_in_bytes → be/log
MemTrackerLimiter → `be-metrics --grep memory`
+
+### Case R3: Workload Policy ConcurrentModificationException
+
+- **Severity**: Critical
+- **Doris 版本**: 4.1.7
+- **Symptom**: 查询偶发 `java.util.ConcurrentModificationException`，概率低但影响面大
+- **Root cause**: Workload Policy 的 query routing 阶段并发修改 policy list（未加锁迭代）
+- **Fix**: 加 concurrent collection 或 copy-on-read
+- **关键诊断动作**: fe.log stacktrace → 确认 `WorkloadPolicyMgr` 调用栈 → 并发场景重现
+
+### Case R4: Workload Group 绑定多个 Compute Group 的路由
+
+- **Severity**: Medium
+- **Doris 版本**: cloud 26.0.3
+- **Symptom**: 一个 WG 绑定多个 compute group 时，查询路由行为不明确
+- **Root cause**: WG-compute group 多对多绑定下，查询分发逻辑未文档化
+- **关键诊断动作**: `SHOW WORKLOAD GROUPS` → compute group 绑定关系 → `EXPLAIN` 确认路由
+
+### Case R5: MV Refresh 资源归属问题
+
+- **Severity**: Medium
+- **Doris 版本**: cloud 26.0.3
+- **Symptom**: MTMV refresh 在一个 compute group 执行，但使用创建者的 workload group 资源配额
+- **Root cause**: MV refresh 任务继承创建者 session 的 WG，而非目标 compute group 默认 WG
+- **关键诊断动作**: `mv_infos()` → refresh 任务 Profile → WG 资源使用对比
+
+---
+
+## Cloud Diagnoses（存算分离诊断）— 5 个案例
+
+### Case CL1: Warmup 后查询仍 Cache Miss
+
+- **Severity**: Critical
+- **Doris 版本**: 4.0.4.9
+- **Symptom**: 执行 warmup 后查询仍全量从 S3 读取，file_cache 未命中
+- **Root cause**: warmup 操作未覆盖所有需要的 segment 文件或 delete bitmap，或 cache TTL 过短
+- **Fix**: 确认 warmup 覆盖范围 + 延长 cache TTL
+- **关键诊断动作**: `be-metrics --grep file_cache` hit ratio → S3 GET 量对比 → warmup 范围
+
+### Case CL2: BE 资源不释放但无查询运行
+
+- **Severity**: High
+- **Doris 版本**: 4.0.11
+- **Symptom**: BE RSS 持续 %mem_limit，`SHOW PROC '/current_queries'` 为空
+- **Root cause**: 历史查询 fragment 结束但 brpc channel 未释放 + file_cache 未 evict
+- **Fix**: `reset_rpc_channel` + 缩短 file_cache TTL
+- **关键诊断动作**: `SHOW PROC '/current_queries'` → `be-metrics --grep fragment` →
`be-metrics --grep file_cache` → RSS vs MemTracker
+
+### Case CL3: 查询卡住 + 慢的 Plan 被选中（Plan 竞态）
+
+- **Severity**: High
+- **Doris 版本**: cloud 26.0.4
+- **Symptom**: 同一查询时而 200ms、时而卡住数分钟
+- **Root cause**: CBO cost estimation 在 cloud 模式下 S3 IO 延迟不确定 + plan cache race
+- **关键诊断动作**: EXPLAIN 对比快慢 → `SET enable_plan_cache=false` → Profile 对比
+
+### Case CL4: Meta-service 事务堆积
+
+- **Severity**: High
+- **Doris 版本**: cloud 4.1.7
+- **Symptom**: meta-service 事务堆积，导致 FE 操作延迟
+- **Root cause**: 写入放大或 commit 冲突率升高
+- **关键诊断动作**: meta-service latency metrics → fe.log meta-service RPC 耗时
+
+### Case CL5: BE 异常重启（无 core dump）
+
+- **Severity**: Medium
+- **Doris 版本**: 4.0.4
+- **Symptom**: cloud 模式下 BE 无 core dump 但进程退出重启
+- **Root cause**: 疑似 brpc health check 超时触发 watchdog kill
+- **关键诊断动作**: be.out 退出日志 → dmesg OOM killer → `be-metrics --grep brpc`
+
+---
+
+## 诊断模式总结（跨案例共性）
+
+### 方法论
+1. **Profile 优先**: 超过半数的性能案例第一步都是看 Profile → Plan Time vs ExecTime
+2. **区分 base vs cumu compaction**: `type=1` = base, `type=2` = cumu → 不同策略不同参数
+3. **jemalloc overhead**: RSS 超出 MemTracker 10-20% 是常态 → mem_limit 留足余量
+4. **Catalog scan range 耗时**: S3/HDFS/Glue/Iceberg 的慢不在 IO，在
RPC/metadata/partition enumeration
+5. **brpc 问题不等于网络问题**: E11/EAGAIN 往往不是网络不通，而是线程池/并发度问题
+6. **先止血再根因**: `reset_rpc_channel`、kill query、调小并发参数 →
保留现场证据（pstack/jstack/heap dump）→ 再分析
+
+### 高频根因类别
+
+| 类别 | 涉及案例 | 占比 |
+|------|----------|------|
+| RPC/并发度/线程池 | I4, Q3, CL2 | ~20% |
+| 内存泄漏/溢出 | N1, N2, N5, N7, R1, R2 | ~25% |
+| Compaction 调度/竞态 | C1, C3, C4, C5 | ~15% |
+| Catalog/metadata 交互 | Q1, DL1, DL2, DL3 | ~15% |
+| 配置错误/版本兼容 | Q2, D2, N6 | ~10% |
+| 代码 bug（空指针/并发） | R3, C2, I5 | ~15% |
+
+---
+
+## 使用方式
+
+在 doris-debug skill 的 case 文件中引用本索引的案例：
+
+```markdown
+## Related cases
+- [Q3] Exchange brpc E11 fanout — 高峰期 exchange RPC EAGAIN，调小
parallel_exchange_instance_num
+- [I4] Import RPC timed out — heavy work pool 打满，tablet_writer_open 超时
+```
+
+排障时优先搜索本索引中相似症状的案例，避免重复排查已知问题。
diff --git a/skills/doris-debug/shared/references/04-metrics-guide.md
b/skills/doris-debug/shared/references/04-metrics-guide.md
new file mode 100644
index 0000000..a5a6fa5
--- /dev/null
+++ b/skills/doris-debug/shared/references/04-metrics-guide.md
@@ -0,0 +1,137 @@
+# Doris 关键监控指标速查（Per-Domain Metrics Guide）
+
+> 按诊断域分类的指标含义、获取方式和告警阈值。
+
+## Query 执行指标
+
+### FE 侧
+
+| 指标 | 来源 | 含义 | 告警阈值 |
+|------|------|------|----------|
+| `QueryTime` | `fe.audit.log` | 端到端耗时 (ms) | > 30s |
+| `ScanRows` | `fe.audit.log` | 扫描行数 | > 1B（关注扫描放大） |
+| `ScanBytes` | `fe.audit.log` | 扫描字节数 | > 10GB（全表扫描风险） |
+| `Plan Time` | Profile | FE 规划耗时 | > 1s（关注 Nereids/CBO） |
+| `Nereids Translate Time` | Profile | Nereids 转译耗时 | > 1s |
+| `Create Scan Range Time` | Profile | 创建 scan range | > 5s（关注外表 RPC） |
+| `ExecTime` | Profile per-operator | 每个 operator 的执行耗时 | 单 operator > 50%
total |
+
+### BE 侧
+
+| 指标 | 来源 | 含义 | 告警阈值 |
+|------|------|------|----------|
+| `WaitForData` | Profile exchange operator | 等待上游 exchange 数据 | ≈ query
timeout → brpc 问题 |
+| `ScannerGetBlockTime` | Profile scan operator | 扫描读 block 耗时 | > 50%
ExecTime → IO 瓶颈 |
+| `SpillDataSize` | Profile | 落盘数据量 | > 0 → 内存压力 |
+| `JoinProbeTime` | Profile join operator | Hash join probe 耗时 | > 30%
ExecTime → join 瓶颈 |
+| `FragmentInstanceNum` | Profile / be/log | fragment instance 数 | > 1000 →
并发过高 |
+
+```sql
+SET enable_profile = true;
+-- 执行查询后在 FE HTTP 获取 Profile:
+curl -s "http://$FE:8030/api/query_profile?query_id=$QID";
+```
+
+## Compaction 指标
+
+| 指标 | 来源 | 含义 | 告警阈值 |
+|------|------|------|----------|
+| `tablet_base_max_compaction_score` | BE /metrics | 最大 base score | > 100 |
+| `tablet_cumulative_max_compaction_score` | BE /metrics | 最大 cumu score | >
50 |
+| `compaction_bytes_total` | BE /metrics | compaction 写入 bytes/s | 接近磁盘带宽 →
磁盘瓶颈 |
+| `version_count` | `SHOW TABLET` | tablet 版本数 | > 1800 → 接近 max(2000) |
+| `CloneTaskQueue` | `SHOW PROC '/tasks'` | clone 任务积压 | > 50 |
+
+```bash
+./scripts/doris-debug be-metrics --be http://$BE:8040 --grep compaction --warn
+```
+
+**Base vs Cumulative 区分**：
+- `type=1` = BASE_COMPACTION → base score，问题方向是 base rowset 合并
+- `type=2` = CUMULATIVE_COMPACTION → cumu score，问题方向是写入频率
+
+## Node 内存指标
+
+| 指标 | 来源 | 含义 | 告警阈值 |
+|------|------|------|----------|
+| `process_mem_usage` | BE /metrics 或 `/profile` | BE 进程 RSS | > 85% mem_limit
|
+| `MemTrackerLimiter` | `be.WARNING` | 内存追踪器超限 | 任何出现 → 内存压力 |
+| `jemalloc_retained_bytes` | BE /metrics | jemalloc 保留未归还 OS 的内存 | > 5GB →
碎片严重 |
+| `DataPageCache[size]` | `/profile` | DataPageCache 大小 | 需结合 RSS 判断 |
+| `SegmentCache[size]` | `/profile` | SegmentCache 大小 | 需结合 RSS 判断 |
+| `QueryCache@cache` | `/profile` | QueryCache 大小 | > 20% mem_limit → 考虑降低 |
+| `file_cache_need_update_lru_blocks_length` | BE bvar | FileCache LRU queue
长度 | > 100000 → 可能堆积 |
+
+```bash
+./scripts/doris-debug be-metrics --be http://$BE:8040 --grep
"memory|jemalloc|cache" --warn
+```
+
+**RSS vs MemTracker 差距**：
+- jemalloc arena 碎片：10-20% overhead
+- brpc buffer pools：不在 MemTracker 内
+- Thread stacks：~8MB × thread count
+- 实践：`mem_limit` = 系统 RAM 的 60-70%，留出 overhead 空间
+
+## Import 指标
+
+| 指标 | 来源 | 含义 | 告警阈值 |
+|------|------|------|----------|
+| `LoadTime` | `SHOW LOAD` | 导入总耗时 | > 10min（大表除外） |
+| `publish_timeout` | be/log | 版本发布超时 | 出现 → compaction 或 tablet 数问题 |
+| `wal_size` | `wal-du` 命令 | WAL 磁盘用量 | > 磁盘 50% → 消费跟不上 |
+| `group_commit_insert_threads` | be.conf | commit worker 池大小 | 忙碌率 > 80% |
+| `heavy_work_pool_active` | BE bvar | heavy pool active 线程数 | = max → 可能排队 |
+
+```bash
+./scripts/doris-debug wal-du /path/to/be/storage
+./scripts/doris-debug log-grep be/log --pack group_commit
+```
+
+## Cloud 存算分离指标
+
+| 指标 | 来源 | 含义 | 告警阈值 |
+|------|------|------|----------|
+| `file_cache_hit_ratio` | BE /metrics | 本地缓存命中率 | < 50% → cache 效果差 |
+| `s3_read_bytes_total` | BE /metrics | S3 读字节（缓存未命中时） | 持续高位 → cache miss 或
warmup 不足 |
+| `meta_service_rpc_latency` | FE 日志或监控 | meta-service RPC 耗时 | P99 > 100ms |
+| `compute_group_fragment_num` | FE cloud cluster check | 计算组 fragment 并发数 | >
2000 → 并发过高 |
+| `file_cache_size` | BE /metrics | file cache 当前占用 | 接近 file_cache_path 配置的
total_size → 考虑扩容 |
+
+```bash
+./scripts/doris-debug be-metrics --be http://$BE:8040 --grep "file_cache|s3"
+```
+
+## Resource Isolation 指标
+
+| 指标 | 来源 | 含义 | 告警阈值 |
+|------|------|------|----------|
+| `ActiveQueries` | `SHOW WORKLOAD GROUPS` | 当前活跃查询 | = max_concurrency |
+| `QueuedQueries` | `SHOW WORKLOAD GROUPS` | 排队查询数 | > 0 持续 → 队列饱和 |
+| `QueueTime` | Profile | 查询排队时间 | > 30s |
+| `cpu_hard_limit` | cgroup | CPU 硬限制 | 使用率 > 90% |
+| `memory_limit_bytes` | cgroup | cgroup 内存限制 | RSS > 85% |
+
+```sql
+SHOW WORKLOAD GROUPS\G
+SELECT * FROM information_schema.workload_group_resource_usage;
+```
+
+```bash
+cat /sys/fs/cgroup/cpu/doris/<wg_id>/cpu.shares
+cat /sys/fs/cgroup/memory/doris/<wg_id>/memory.limit_in_bytes
+```
+
+## FE 自身健康
+
+| 指标 | 来源 | 含义 | 告警阈值 |
+|------|------|------|----------|
+| `Old Gen usage` | `jmap -heap` | FE JVM Old Gen 使用率 | > 85% → 接近 OOM |
+| `GC time` | `fe.gc.log` | GC 耗时 | > 10s / 次的 Full GC |
+| `number_of_queries` | `SHOW PROC '/current_queries'` | 当前运行查询数 | > 1000 |
+| `BDBJE log file count` | `ls fe/doris-meta/bdb/*.jdb \| wc -l` | BDBJE 日志文件数
| 增长异常 → 可能磁盘问题 |
+| `Edit log replay gap` | `SHOW FRONTENDS` → `ReplayedJournalId` | 从 Master
的日志回放延迟 | > 1000 |
+
+```bash
+jstack <fe_pid> | grep -A5 "BLOCKED\|WAITING" | head -30
+jmap -heap <fe_pid> 2>/dev/null | grep "Old\\|Eden"
+```
diff --git a/skills/doris-debug/shared/references/05-config-quick-ref.md
b/skills/doris-debug/shared/references/05-config-quick-ref.md
new file mode 100644
index 0000000..084a983
--- /dev/null
+++ b/skills/doris-debug/shared/references/05-config-quick-ref.md
@@ -0,0 +1,113 @@
+# Doris 关键配置速查（Per-Domain Config Quick Reference）
+
+> 按诊断域分类的关键配置参数，含默认值、作用和调参方向。
+
+## Query 执行
+
+### 超时控制
+
+| 参数 | 默认值 | 作用 | 调参方向 |
+|------|--------|------|----------|
+| `query_timeout` | 300(s) | 查询超时 | 慢 ETL 查询增大，交互式减小 |
+| `nereids_timeout_second` | 30(s) | Nereids 优化器超时 | Plan Time 过长时增大到 60-120 |
+| `insert_timeout` | 14400(s) | INSERT 超时 | 大表导入增大 |
+
+### 并发控制
+
+| 参数 | 默认值 | 作用 | 调参方向 |
+|------|--------|------|----------|
+| `parallel_exchange_instance_num` | 100 | exchange instance 数 | E11 时**减小**到
50/32 |
+| `parallel_fragment_exec_instance_num` | 8 | fragment instance 数 | E11 时减小到 4
或 1 |
+| `parallel_pipeline_task_num` | 0 | pipeline task 数（0=auto） | E11 或 CPU 高时设为
1 |
+| `max_instance_num` | 64 | 单 fragment 最大 instance | 减少 fanout |
+| `enable_plan_cache` | true | 查询计划缓存 | Plan 竞态时临时关闭验证 |
+
+### Session Variable（排查用）
+
+```sql
+SET enable_profile = true; -- 启用 Profile
+SET disable_join_reorder = true; -- 跳过 join reorder
+SET enable_materialized_view_rewrite = false; -- 跳过 MV rewrite
+SET enable_spill = true; -- 启用磁盘溢出
+SET materialized_view_rewrite_enable_contain_external_table = true;
+```
+
+## Compaction
+
+| 参数 | 默认值 | 作用 | 调参方向 |
+|------|--------|------|----------|
+| `max_tablet_version_num` | 2000 | tablet 版本数硬上限 | -235 时临时增大到 5000，但必须同时修复
compaction |
+| `time_series_max_tablet_version_num` | 20000 | 时序表版本上限 | 时序场景单独调大 |
+| `max_cumulative_compaction_threads` | -1（auto） | cumu compaction 线程数 | cumu
score 高时显式设置 |
+| `compaction_task_num_per_disk` | 4 | 每盘并发 compaction 数 | 磁盘 IO 未饱和时增大 |
+| `compaction_tablet_size_threshold` | 100GB | 超过此大小跳过 base compaction | 大
tablet 场景调大 |
+| `base_compaction_interval_seconds_since_last_operation` | 86400 | base
compaction 最低间隔 | SC 后调小加速合并 |
+| `disable_auto_compaction` | false | 全局禁用 compaction | 紧急止血（如倒排索引重建） |
+
+```bash
+# 运行时查看
+curl -s "http://$BE:8040/api/show_config"; | grep compaction
+```
+
+## Node 内存
+
+| 参数 | 默认值 | 作用 | 调参方向 |
+|------|--------|------|----------|
+| `mem_limit` | 80% | BE 内存软限制 | 实际 RSS = mem_limit + 10-20%，容器环境建议 60-70% |
+| `max_segment_cache_size` | 0（无限制） | Segment Cache 大小 | 显式设置避免缓存吃满 RSS |
+| `storage_page_cache_limit` | 0（无限制） | Page Cache 大小 | 显式限制 |
+| `enable_je_purge` | false | jemalloc dirty page 回收 | 内存碎片严重时开启 |
+| `chunk_reserved_bytes_limit` | 2GB | chunk 分配器保留内存 | OOM 时减小 |
+| `memory_gc_enable` | true | 内存压力时 GC | 保持开启 |
+| `process_memory_recovery_enable` | false | OOM 前尝试 cancel query | 建议开启 |
+
+## Import
+
+| 参数 | 默认值 | 作用 | 调参方向 |
+|------|--------|------|----------|
+| `group_commit_insert_threads` | 10 | commit worker 线程 | 消费跟不上时增大 |
+| `group_commit_relay_wal_threads` | 10 | WAL relay 线程 | WAL 堆积时增大 |
+| `group_commit_data_bytes` | 64MB（table level） | size flush 阈值 | 高频导入调大增吞吐 |
+| `group_commit_interval_ms` | 10000（table level） | time flush 阈值 |
高频导入调大减少版本数 |
+| `group_commit_wal_max_disk_limit` | 10% | WAL 磁盘上限 | 磁盘大时增大 |
+| `tablet_writer_open_rpc_timeout_sec` | 60 | open tablet writer RPC 超时 |
Heavy pool 排队严重时增大 |
+| `brpc_heavy_work_pool_threads` | 256 | heavy pool 线程数 | 高并发导入时增大（如 384），但需保留
pstack 确认卡点 |
+| `enable_redirect_strict_check` | true | Stream Load redirect IP 校验 |
客户端外网时设为 false |
+
+## Cloud 存算分离
+
+| 参数 | 默认值 | 作用 | 调参方向 |
+|------|--------|------|----------|
+| `file_cache_path` | 无 | file cache 路径和容量 | 按 NVMe 容量设置 |
+| `file_cache_type` | `whole_file_cache` | 缓存粒度（整文件/子文件） | `sub_file_cache` 减少
S3 list |
+| `file_cache_query_limit` | 0 | 单查询 file cache 上限 | 防止单查询吃满 cache |
+| `file_cache_background_lru_log_replay_interval_ms` | 1（4.1.8+） | LRU
recorder replay 间隔 | 内存泄漏时调小（1000→1） |
+| `file_cache_background_lru_log_queue_max_size` | 500000（4.1.8+） | LRU
recorder queue 硬上限 | 防止 queue 无限堆积 |
+| `s3_max_connections` | 256 | S3 并发连接数 | 高并发时增大 |
+| `s3_request_timeout_ms` | 30000 | S3 请求超时 | S3 慢时增大 |
+
+## Resource Isolation
+
+| 参数 | 默认值 | 作用 | 调参方向 |
+|------|--------|------|----------|
+| `enable_workload_group` | true | 启用 Workload Group | cgroup 未配置时 WG 只是建议 |
+| `max_concurrency` | (per WG) | WG 最大并发 | 队列堆积时增大 |
+| `max_queue_size` | (per WG) | WG 最大排队数 | 排队爆满时增大 |
+| `queue_timeout` | (per WG) | 排队超时 | 长时间排队时增大 |
+| `cpu_share` | (per WG) | CPU share 权重 | 饥饿 WG 增大 |
+| `cpu_hard_limit` | (per WG) | CPU 硬限制 | 单 WG 吃满 CPU 时设置 |
+
+```sql
+ALTER WORKLOAD GROUP wg SET ("max_concurrency" = "8", "queue_timeout" = "300");
+```
+
+## FE 高可用
+
+| 参数 | 默认值 | 作用 | 调参方向 |
+|------|--------|------|----------|
+| `-Xmx` / `-Xms` | 取决于部署 | FE JVM heap | OOM 或 GC 频繁时增大 |
+| `hive_metastore_client_timeout_second` | 10 | HMS 客户端超时 | HMS 慢时增大 |
+| `iceberg_manifest_cache_refresh_interval_s` | 3600 | Iceberg manifest 缓存刷新间隔
| Plan Time 长时增大 |
+| `bdbje_cleaner_threads` | 1 | BDBJE 日志清理线程 | .jdb 文件堆积时增大 |
+| `meta_dir` | `fe/doris-meta` | FE 元数据目录 | 放到高性能磁盘，预留 20%+ 空间 |
+| `priority_networks` | 空 | brpc 绑定网段 | 多网卡环境必须设置 |
diff --git a/skills/doris-debug/tablet/SKILL.md
b/skills/doris-debug/tablet/SKILL.md
new file mode 100644
index 0000000..ac481f5
--- /dev/null
+++ b/skills/doris-debug/tablet/SKILL.md
@@ -0,0 +1,100 @@
+---
+name: doris-debug-tablet
+description: >
+ Use for Doris tablet/replica health, version skew, clone backlog, disk
balance,
+ and tablet repair. Commands: SHOW PROC tablet_health, SHOW TABLET, ADMIN
REPAIR.
+version: 0.2.0
+category: tablet
+---
+
+# Tablet & Replica
+
+## Quick scan
+
+```sql
+SHOW PROC '/cluster_health/tablet_health';
+SHOW PROC '/statistic';
+SHOW PROC '/tasks';
+```
+
+## Causes
+
+| ID | Cause | Evidence | Source anchor |
+|----|-------|----------|---------------|
+| A | Replica missing / unhealthy | `SHOW PROC
'/cluster_health/tablet_health'` Health≠TOTAL | `TabletHealth.java` |
+| B | Version skew | tablet version lag > 3 behind quorum |
`TabletInvertedIndex.cpp` |
+| C | Clone backlog | `SHOW PROC '/tasks'` CLONE queue depth > 50 |
`CloneTask.cpp`, BE `clone_worker_count` |
+| D | Disk skew | `SHOW PROC '/statistic'` one disk > 85% while others < 50% |
`DiskBalancer` / balance config |
+| E | Single tablet too large | tablet size >
`compaction_tablet_size_threshold` | `config.cpp` + compaction |
+
+## 10 min triage
+
+```sql
+-- 1. Global health snapshot
+SHOW PROC '/cluster_health/tablet_health';
+
+-- 2. Per-table replica drilldown
+ADMIN SHOW REPLICA STATUS FROM db.table WHERE ReplicaStatus != 'OK';
+
+-- 3. Task queue depth
+SHOW PROC '/tasks';
+```
+
+```bash
+# BE-side tablet count & version lag
+./scripts/doris-debug be-metrics --be http://$BE:8040 --grep
"tablet|version|clone"
+./scripts/doris-debug log-grep be/log --pack versions
+
+# Check specific tablet distribution
+curl -s "http://$FE:8030/api/tablets_distribution?db=db_name&table=tbl_name";
+```
+
+## Cause A — Replica missing
+
+```sql
+-- Identify unhealthy replicas
+ADMIN SHOW REPLICA STATUS FROM db.table WHERE ReplicaStatus != 'OK';
+
+-- Repair with low priority (production safe)
+ADMIN REPAIR TABLE db.table PRIORITY = 'LOW';
+
+-- For urgent single-tablet repair
+ADMIN REPAIR TABLE db.table PARTITION (p202401) PRIORITY = 'HIGH';
+```
+
+Check BE `heartbeat_service_mgr.cpp` — heartbeats from a BE with missing
tablets won't self-heal if compaction or clone threads are exhausted.
+
+## Cause C — Clone backlog
+
+Clone concurrency: `clone_worker_count` (default 3 in `config.cpp`). A high
Clone backlog often means:
+
+1. Tablet migration triggered by `balance_load_disk_safe_threshold` /
`storage_high_watermark_usage_percent`
+2. Clone threads can't keep up with the repair plan
+
+```properties
+# be.conf — raise clone workers cautiously
+clone_worker_count = 6
+```
+
+## Cause D — Disk skew
+
+```sql
+SHOW PROC '/statistic'; -- per-disk usage
+SHOW PROC '/backends'; -- per-BE disk UsedPct
+```
+
+```properties
+# fe.conf
+balance_load_disk_safe_threshold = 0.5
+storage_high_watermark_usage_percent = 80
+```
+
+Manual rebalance: `ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");`
then selective `ADMIN REBALANCE DISK`.
+
+## Source
+
+- `fe/.../master/TabletHealth.java` — cluster_health proc
+- `fe/.../clone/TabletSchedCtx.java`
+- `be/src/agent/task_worker_pool.cpp` — clone worker pool
+- `be/src/common/config.cpp` — `clone_worker_count`, `storage_root_path`
+- `be/src/olap/tablet_manager.cpp` — tablet version tracking
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
