I noticed that large jobs that needed to reserve resources, kept having their expected start time drift into the future. This happened even when the large job was the highest priority job in the queue. The jobs just kept waiting and waiting.
Looking at the running jobs on the nodes that the job expected to start on, the ending time for those jobs was sometimes several hours before the expected start time of the pending job. I turned on debug output for SelectType, BackfillMap and Backfill and noticed that the _will_run_test function in plugins/select/cons_res/select_cons_res.c uses a doubling factor for the time_window it uses in selecting jobs ending close together. Because we sometimes have good number of jobs which backfill behind large jobs, this doubling factor caused the time_window to be several hours past when the actual running jobs on those nodes would complete. As users continued to utilize the "flexible" backfill window, the large job kept drifting further into the future. If I change the time_window to have a more linear increase, rather than the doubling factor in the original code, it certainly increases the number of iterations _will_run_test check goes through, but it ends up with a much better estimate of possible starting time for the job. We've been using the cons_res select plugin, because we have had occasion to allocate nodes to multiple jobs and cons_res give a bit more flexibility for that kind of thing. However, the linear select plugin, does not seem to have the same time_window selection as it attempts to find the proper start time for pending jobs. Looking through the documentation, it may be that the linear select plugin would be better, in our case, since we don't run a lot of serial, node-sharing jobs, but it's unclear from the documentation if I can switch to the select plugin on a running system. My guess is that it'll cause problems. Anyone tried this? ----- Gary Skouson
