On 2014-06-17 17:05:05 -0400 (-0400), Sean Dague wrote: [...] > If nodepool (conceptually) filled the longest outstanding requests with > higher priority, I'd be uber happy. This would also help with more fully > using our capacity, because the mix of nodes that we need any given hour > kind of changes. But as jeblair said, this is non trivial to implement. > Ensuring a minimum number of nodes (where that might be 1 or 2) for each > class would have helped this particular situation. We actually had 0 > nodes in use or ready of the type at the time. [...]
Part of the complication here is that nodepool's current demand determination algorithm considers nodes in a "building" state as satisfying the stated demand, so as to avoid it madly overprovisioning when there are nodes on the way to take care of the pending load. Unfortunately this coupled with incidents in cloud providers causing nodepool to think a node is building when it's actually stuck can quickly lead to zero capacity if a lower-demand node type ends up with all its proportion occupied by nodes stuck in that state. Similarly for ready workers which nodepool thinks got added to a Jenkins master but which didn't actually "stick" (we see this sometimes and it probably indicates a Jenkins bug of some kind). -- Jeremy Stanley _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
