Olrookie opened a new issue, #67858: URL: https://github.com/apache/doris/issues/67858
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version - Apache Doris **4.0.3** (community release, tag `4.0.3-rc03`), **non-cloud mode** (1 FE + 1 BE on one node) - Remote storage: Alibaba Cloud OSS (S3-compatible), registered via `CREATE S3 RESOURCE` + `CREATE STORAGE POLICY` with `cooldown_ttl = 600s` - Affected table: AUTO PARTITION duplicate-key table; ETL runs `INSERT OVERWRITE` on its partitions every minute - The relevant code is unchanged in 4.0.4–4.0.8 / branch-4.0 (verified at source level) ### What's Wrong? ## Summary In **non-cloud mode**, after a **BE restart**, queries that need to open segments of rowsets already cooled down to S3/OSS fail and keep failing with `[E-206] get fs failed`: the BE-side in-memory storage-resource map is empty after restart and is never re-populated, and the read-path self-heal is gated by `config::is_cloud_mode()`. ## Root cause chain (verified in 4.0.3-rc03 source) 1. BE keeps storage resources only in an in-memory global map (`s_storage_resource_mgr` in `be/src/olap/storage_policy.cpp`). It is not persisted. 2. The map is populated **only** by the `PUSH_STORAGE_POLICY` agent task (`push_storage_policy_callback` in `be/src/agent/task_worker_pool.cpp` → `update_s3_resource()` → `put_storage_resource()`). 3. After a BE restart the map is empty. FE does not re-push storage policies/resources to a restarting BE (the tablet report already carries the BE's known resource ids/versions, but no re-push is triggered). 4. When a query opens a cooled segment, `RowsetMeta::remote_storage_resource()` (`be/src/olap/rowset/rowset_meta.cpp:167`) misses the map; its self-heal (`sync_storage_vault()`) is **gated by `config::is_cloud_mode()`** (line 178), so in non-cloud mode it returns `InternalError("cannot find storage resource. resource_id={}")` (line 188) without any re-sync attempt. 5. `RowsetMeta::physical_fs()` (line 113) logs that error and returns `nullptr`; `BetaRowset::load_segment` (`be/src/olap/rowset/beta_rowset.cpp:190`) then fails the query with the bare `[E-206] get fs failed`. 6. Nothing heals it: the query keeps failing on every attempt until something else re-populates the map (a new cooldown triggering a policy push, or a partition drop/rebuild). ## Production evidence (observational — we did not run a controlled restart experiment) Single-node 4.0.3 cluster, one S3-compatible (OSS) resource; minute-level `INSERT OVERWRITE` ETL on local (non-cooled) partitions kept working throughout everything below. - 2026-09-04 15:37 BE restart (upgrade; our be.WARNING window opens 09-07 12:12, already inside the resulting error storm): queries against partitions cooled to OSS failing with `[E-206] get fs failed` (~92K warnings), which only stopped when those partitions were dropped/recreated (09-07 15:55). - 09-07 15:50 the policy was attached to two other partitions and they cooled down successfully — a successful cooldown needs the resource on the BE, so a push must have re-populated the resource map between the restart and that cooldown. - 2026-09-10 10:25 BE restart. The first warning after BE came back (10:25:46.927) is `cannot find storage resource. resource_id=...` (twice), then >150K `[E-206]` warnings within ~5 hours from 6,720 distinct queries, until the BE process was stopped again at 15:22. Between the two storms (multi-day window) there is not a single such warning. ## Observability traps - `SELECT count(*)` on the affected table succeeds (it only reads rowset meta), so health checks look fine. - Point / `LIMIT` queries that stop early can succeed via the segment cache, so the failure looks intermittent. - Deterministic verification needs a real full scan of the cooled tablet (e.g. `ORDER BY` on a non-key column). ### What You Expected? Cooled (remote) rowsets should remain readable after a BE restart in non-cloud mode. Concretely, any of: 1. FE re-pushes storage policies/resources to a BE when it registers/reports after a restart (the report already carries resource ids/versions); 2. or the read path self-heals on a map miss in non-cloud mode too — remove the `config::is_cloud_mode()` gate in `RowsetMeta::remote_storage_resource()` and re-fetch the resource from FE; 3. or BE persists / rebuilds the resource map at startup. Independently, the failure should be actionable instead of `[E-206] get fs failed`: `BetaRowset::load_segment` should include `resource_id` in the message (nearby code already does this, e.g. beta_rowset.cpp:100/157), and the underlying `cannot find storage resource` root cause should not stay hidden. ### How to Reproduce? **We have NOT run a controlled reproduction.** The steps below are derived from the 4.0.3 source analysis above and are consistent with what we observed in production, but they are untested — treat them as a suggested path; corrections welcome. 1. Non-cloud cluster, 1 FE + 1 BE. 2. `CREATE S3 RESOURCE` (any S3-compatible storage), then `CREATE STORAGE POLICY ... PROPERTIES("cooldown_ttl"="60s")`. 3. Create a table with the policy attached, load data once, wait for rowsets to cool down to S3; verify a full scan works (e.g. `SELECT ... ORDER BY <non-key column>`). 4. Restart the BE, and do NOT load new data before step 5 — anything that makes FE push the policy again re-populates the BE map and closes the failure window. 5. Run the same full-scan query; per the code it should fail with `[E-206] get fs failed`, and be.WARNING should log `cannot find storage resource. resource_id=...` on every attempt. Why reproduction may look non-deterministic (matches our production experience): - Only queries that actually open cooled segments fail: `SELECT count(*)` reads meta only and always succeeds; `LIMIT`-style queries can succeed via early termination + segment cache. - The failure window is "from BE restart until the next policy re-push"; after a re-push every query succeeds again. ### Anything Else? - Related open issue: #63669 hits the same `get fs failed` family on a different path (clone). Our case is the plain read path after restart in non-cloud mode. - Environment details: single node, S3-compatible object storage; storage policy attached to AUTO PARTITION table partitions; heavy minute-level `INSERT OVERWRITE` workload (writes unaffected). - We also observed a separate `[E-3115] version not continuous for mow` problem on a mow table the same day; I can file it separately once I have a minimal repro. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
