We have seen on a number of benchmarks that the scheduler does not operate like you (I) would expect. One of the tools we have examined a trace file and showed that, for a "long" period of time, two processes were sharing a single cpu. That, in itself, is not startling, but there were a number of idle processors when this was happening.
I found the article below searching for others who saw the same sort of problem. What made me keep reading was: "With so many rules about when the load balancing does or does not occur, it becomes difficult to reason about how long an idle core would remain idle if there is work to do and how long a task might stay in a run queue waiting for its turn to run when there are idle cores in the system." Exactly the issue we were experiencing! The authors were kind enough to provide patches in their article -- we built a kernel including the patches. We have not had much time to test it, but we have seen a 30% gain on one benchmark.. The assistance I would like is to have an experienced scheduler person look at these changes to see how we can apply them to power. We have smt8, but need to have all the threads working for smt8 to make a difference -- this fix addresses that problem and gives us an advantage. Jim Van Fleet https://blog.acolyer.org/2016/04/26/the-linux-scheduler-a-decade-of-wasted-cores/
