On 01/07/2015 11:29 AM, Clint Byrum wrote: > Excerpts from Derek Higgins's message of 2015-01-07 02:51:41 -0800: >> Hi All, >> I intended to bring this up at this mornings meeting but the train I >> was on had no power sockets (and I had no battery) so sending to the >> list instead. >> >> We currently run our CI with on images built for i386, we took this >> decision a while back to save memory ( at the time is allowed us to move >> the amount of memory required in our VMs from 4G to 2G (exactly where in >> those bands the hard requirements are I don't know) >> >> Since then we have had to move back to 3G for the i386 VM as 2G was no >> longer enough so the saving in memory is no longer as dramatic. >> >> Now that the difference isn't as dramatic, I propose we switch back to >> amd64 (with 4G vms) in order to CI on what would be closer to a >> production deployment and before making the switch wanted to throw the >> idea out there for others to digest. >> >> This obviously would impact our capacity as we will have to reduce the >> number of testenvs per testenv hosts. Our capacity (in RH1 and roughly >> speaking) allows us to run about 1440 ci jobs per day. I believe we can >> make the switch and still keep capacity above 1200 with a few other changes >> 1. Add some more testenv hosts, we have 2 unused hosts at the moment and >> we can probably take 2 of the compute nodes from the overcloud. >> 2. Kill VM's at the end of each CI test (as opposed to leaving them >> running until the next CI test kills them), allowing us to more >> successfully overcommit on RAM >> 3. maybe look into adding swap on the test env hosts, they don't >> currently have any, so over committing RAM is a problem the the OOM >> killer is handling from time to time (I only noticed this yesterday). >> >> The other benefit to doing this is that is we were to ever want to CI >> images build with packages (this has come up in previous meetings) we >> wouldn't need to provide i386 packages just for CI, while the rest of >> the world uses the amd64. > > +1 on all counts. > > It's also important to note that we should actually have a whole new > rack of servers added to capacity soon (I think soon is about 6 months > so far, but we are at least committed to it). So this would be, at worst, > a temporary loss of 240 jobs per day.
Actually it should be sooner than that - hp1 still isn't in the CI rotation yet, so once that infra change merges (the only thing preventing us from using it AFAIK) we'll be getting a bunch more capacity in the much nearer term. Unless Derek is already counting that in his estimates above, of course. I don't feel like we've been all that capacity constrained lately anyway, so as I said in my other (largely unnecessary, as it turns out) email, I'm +1 on doing this. -Ben _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
