Fly-Style opened a new pull request, #19378:
URL: https://github.com/apache/druid/pull/19378
This PR updates the seekable-stream cost-based autoscaler to make task-count
decisions more stable and easier to reason about.
The main behavioral change is replacing the previous linear idle cost with a
U-shaped idle cost centered around an ideal idle ratio. This penalizes both
under-provisioning, where tasks have too little idle headroom, and
over-provisioning, where tasks spend too much time idle. The goal is to keep
ingestion tasks near a practical operating point instead of treating all
additional idle time as uniformly bad.
This also separates task-count boundary controls for scale-up and
scale-down. Scale-up remains unbounded by default so the autoscaler can react
aggressively to lag, while scale-down is bounded by default to avoid large
drops in task count. Candidate task counts are still generated from valid
partitions-per-task ratios, but the optimizer can now limit which candidates
are evaluated depending on the configured scale direction boundary.
### What Changed
- Added U-shaped idle cost in WeightedCostFunction, with an ideal idle
ratio and asymmetric penalties for under- and over-provisioning.
- Updated default cost weights to favor a more balanced lag/idle tradeoff.
- Replaced the single useTaskCountBoundaries setting with:
- Added CostResult.INFINITE_COST so skipped candidates can still be
represented safely in cost tables.
- Updated supervisor docs for the cost-based autoscaler behavior and
config options.
- Expanded tests around U-shaped idle cost, config serialization/defaults,
valid task-count generation, and bounded vs unbounded task-count jumps.
This PR has:
- [ ] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
