andygrove opened a new issue, #2270: URL: https://github.com/apache/datafusion-ballista/issues/2270
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** `GET /api/jobs` returns every job it knows about in a single response, on both the scheduler (`ballista/scheduler/src/api/handlers.rs`) and the history server added in #2265. On the scheduler this is bounded in practice, because completed jobs are cleaned up after `finished_job_state_clean_up_interval_seconds` and everything is gone on restart. The history server has no such bound: logs accumulate in the event-log directory until somebody prunes them, so the response grows with however long an operator chooses to retain history. A directory holding tens of thousands of jobs produces a response of a size no client wants, and the TUI refreshes the list periodically. Raised by @milenkovicm in review on #2265. **Describe the solution you'd like** Paging on `/api/jobs`, added to the scheduler and the history server together rather than to one of them. Doing it on only one side would give the same endpoint two different contracts, and the TUI (`ballista-cli/src/tui/http_client.rs`) points at either one, so it cannot start passing a limit until both understand it. Sketch, to be settled in the issue rather than assumed: - Query parameters on `GET /api/jobs`, defaulting to the current behaviour so existing clients keep working. - A stable order to page over. #2265 makes the history server return jobs newest first by start time, with the job id as a tiebreaker so the order is total. Job ids are random 7-character strings (`TaskManager::generate_job_id`), so they cannot be paged over on their own. The live scheduler currently applies no ordering at all and would need one. - The TUI asking for a bounded window and requesting more as the user scrolls, instead of re-fetching everything on each refresh. Worth deciding at the same time whether the response stays a bare JSON array or gains an envelope carrying a total count or a continuation token, since that shape is the part existing clients would notice. **Describe alternatives you've considered** - **Leave it unpaged and rely on pruning.** What happens today, and it puts the server's response size at the mercy of an operator's retention policy. It is documented as a limitation on the history server user-guide page, which is worth doing but is not a fix. - **Cap the response server-side without a paging API.** Silently truncating a list is worse than a large one, since a job that ran would simply not be visible with nothing to say so. - **Page in the history server only.** Cheaper, but it is the divergence described above. **Additional context** The history server side is the more pressing of the two, since it is the one with unbounded retention. See #2265 and the review discussion there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
