I have background processes running Sidekiq that run for several hours each night. They generate a large number of R14 errors (1600 for 2 processes each running about 6-9 hours). But it seems like the excessive memory is slight (not more than 520MB vs the 512MB allocated), so it would not be justified to use jumbo (1024MB) processes.
Threads require pre-loading of the runtime environment, which includes the Rails library. If there are parts of the library that I'm not using (and there are), I ought to be able to fine-tune the pre-load process to exclude them, eventually achieving a point where memory usage comes within 512MB. Determining which parts are used and which aren't can be a tedious experience. Can I not just rely on the paging to eventually page out the parts I'm not using, at which point no more paging will take place because the out-of-memory pages are never requested? I tried lowering the number of concurrent jobs per process (i.e. threads) from 10 to 8, but it seemed like it only made up 4-5MB in excess memory and did not affect the number of R14 errors. In order to make up for the lost concurrency, I added another process. The overall effect was to slow everything down (an additional 2-1/2 hours of processing to get through the workload), so I concluded that the extra concurrency (3x8=24 vs 2x10 = 20) put extra load on the Postgre database which became a processing bottleneck. So my gut feeling is that changing the jobs-per-process count is not the best way to lower memory usage. Any other thoughts on how to solve this (or whether it really needs to be solved) would be appreciated. -- -- You received this message because you are subscribed to the Google Groups "Heroku" group. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "Heroku Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to heroku+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
