Fly-Style commented on code in PR #19378: URL: https://github.com/apache/druid/pull/19378#discussion_r3149126022
########## docs/ingestion/supervisor.md: ########## @@ -193,29 +193,30 @@ The following example shows a supervisor spec with `lagBased` autoscaler: ``` </details> -**2. Cost-based autoscaler strategy (experimental)** +**2. Cost-based autoscaler strategy** -An autoscaler which computes the required supervisor task count via cost function based on ingestion lag and poll-to-idle ratio. -Task counts are selected from a bounded range derived from the current partitions-per-task ratio, -not strictly from factors/divisors of the partition count. This bounded partitions-per-task window enables gradual scaling while -voiding large jumps and still allowing non-divisor task counts when needed. +The cost-based autoscaler picks the number of ingestion tasks that minimizes a combined cost score. The score has two components: -**It is experimental and the implementation details as well as cost function parameters are subject to change.** +- **Lag cost** — how long it would take to drain the current backlog at the observed processing rate. More tasks reduce this cost. +- **Idle cost** — how far the predicted idle ratio is from the target of ~25%. Tasks that are too busy (under-provisioned) or too idle (over-provisioned) both drive the score up. +The sweet spot is roughly 25% idle, giving headroom to absorb traffic spikes without wasting resources. + +At every evaluation interval, Druid computes the score for each candidate task count and picks the one with the lowest total cost. Note: Kinesis is not supported yet, support is in progress. Review Comment: I need to verify if anybody have a Kinesis workload with CBA working. If you want, we can remove that part. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
