milenkovicm opened a new pull request, #2431:
URL: https://github.com/apache/datafusion-ballista/pull/2431
# Which issue does this PR close?
Closes #.
# Rationale for this change
The standalone history server kept its job index current by re-walking the
event-log directory on a fixed timer (`--update-interval-seconds`, default
10s). That has two problems:
- **Latency vs. cost trade-off is hard-coded.** A short interval means a
`stat` per file per tick over a directory that can hold thousands of
finished jobs; a long interval means a finished job is invisible for up to
that long.
- **The source is assumed to be a flat local directory.** All listing and
reading was inline `std::fs`, so there was no seam for an alternative
backend (object store, archived snapshot, remote listing) or an
alternative "when to refresh" policy.
This change pulls "where logs come from" and "when to act on them" behind
two traits, and wires in a `notify`-based directory watch so a completed log
is indexed the moment it is renamed into place rather than on the next tick.
# What changes are included in this PR?
**New `history/source.rs` — `EventLogSource` trait**
- `scan_jobs() -> BoxStream<io::Result<PathBuf>>`, `read_job_index`,
`read_completed_job`.
- `LocalDirSource` is the only impl and carries over the previous flat-dir
`std::fs` logic unchanged (including the `.eventlog` vs `.eventlog.running`
extension filter).
**New `history/trigger.rs` — `ScanTrigger` / `EventLogTrigger` traits**
- `ScanTrigger::scan_tick` decides when a full directory pass runs;
`OnceTrigger` fires exactly once then parks, `NoopTrigger` never fires.
- `EventLogTrigger::next_eventlog` reports individual `*.eventlog` files as
they appear; `NotifyTrigger` watches the directory with the `notify`
crate, treating both create and rename-into-place (`Modify(Name(..))`) as
"a new file appeared", filtering by the `eventlog` extension.
- Unit tests for `NotifyTrigger`: create, arrive-by-rename, and
ignore-running-logs-and-unrelated-files.
**`history/mod.rs` — `HistoryStore` rework**
- `HistoryStore::load` + `refresh` + `RefreshStats` + `FileStamp` + the
`seen` map are removed.
- `HistoryStore::new(dir)` sets up the `NotifyTrigger` watch and builds the
initial index synchronously; `dir` must already exist.
- `HistoryStore::new_static(dir)` builds a frozen snapshot (both triggers
noop) for tests and one-shot use over an archived directory.
- `spawn_refresh_task` becomes `spawn_service_tasks(store) -> ServiceTasks`,
which runs a watch loop (`index_one` per file from the trigger) and a
scan loop (full `load_index` pass per `scan_tick`). `ServiceTasks` aborts
both loops on drop.
- Indexing is now **upsert-only**: a log that disappears or is rewritten
under a new job id is no longer reconciled out of the index.
- `read_job` is now `async` and delegates I/O to the source;
`read_job_blocking` is gone, replaced by `impl From<JobReadError> for
SchedulerErrorResponse`.
- The timer/refresh/removal test suite is deleted; remaining tests move to
`new_static` / `.await`.
**`bin/history_server.rs`**
- Drops the `--update-interval-seconds` arg.
- `create_dir_all`s the event-log directory before `HistoryStore::new`
(the watch cannot be placed on a missing path).
**Dependencies**
- Adds `notify = "8"` as a workspace dependency, wired into the scheduler
crate's `rest-api` feature as optional `dep:notify`.
- `Cargo.lock` updated accordingly.
# Are there any user-facing changes?
Yes, all behind the `rest-api` feature:
- **`--update-interval-seconds` is removed** from the `history_server`
binary. Refresh is now a single startup pass plus an event-driven
filesystem watch.
- **Deleted or rewritten event logs are no longer dropped from the listing
at runtime.** Previously a rescan removed them; now a restart is required
to reflect removals.
- The history server now **creates the event-log directory if it does not
exist** instead of treating a missing directory as empty.
- Commit subject is "draft interfaces" — this is WIP; the trait surface may
still change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
