I'm running web2py in a production environment that handles about 30 requests per second, using Nginx + uWSGI.
The server is a Linode VPS with 16 cores. uWSGI runs an average of 30 workers (it spawns or recycles workers depending on traffic load) The server has plenty of resources available (CPU usage is always around 25%, used memory is around 50%). My web2py application consists in a bunch of news websites (articles and blog posts). Most of requests are simple HTML pages, and they are generated in a few milliseconds. However, since a couple of months ago, I noticed that a few of those requests take much longer than they should (some of them can take up to 60 seconds, which is nonsense). It isn't always the same request that hangs. Requests hang randomly: the same request that takes 300 milliseconds to complete can take up to 50 seconds later. I checked the health of some related services (like postgresql for databases, pgbouncer for pooling database connections, redis for cache) and there is nothing weird going on there. None of those services is reporting any warning or error, they are all running smoothly. Here is the only interesting thing I noticed: *those requests that hang always correspond to uWSGI workers that have been assigned to core 0*. uWSGI has a config option called "harakiri" which is basically a timeout: if a worker has been working in a requests for more than X seconds, then the worker is killed (recycled). When this happens, uWSGI logs information about the event: the worker number, the request that hanged, and the core number that the worker was assigned to. And the problem is **always** with core 0*. *Here there are some examples of what I see in uWSGI log: Thu May 9 07:37:29 2019 - *** HARAKIRI ON WORKER 2 (pid: 27125, try: 1) *** Thu May 9 07:37:29 2019 - HARAKIRI !!! worker 2 status !!! Thu May 9 07:37:29 2019 - HARAKIRI [core 0] 186.111.149.160 - GET /necrologicas since 1557398188 Thu May 9 07:37:29 2019 - HARAKIRI !!! end of worker 2 status !!! Thu May 9 07:55:44 2019 - *** HARAKIRI ON WORKER 9 (pid: 18405, try: 1) *** Thu May 9 07:55:44 2019 - HARAKIRI !!! worker 9 status !!! Thu May 9 07:55:44 2019 - HARAKIRI [core 0] 186.109.121.239 - GET / since 1557399283 Thu May 9 07:55:44 2019 - HARAKIRI !!! end of worker 9 status !!! Thu May 9 09:02:48 2019 - *** HARAKIRI ON WORKER 8 (pid: 3287, try: 1) *** Thu May 9 09:02:48 2019 - HARAKIRI !!! worker 8 status !!! Thu May 9 09:02:48 2019 - HARAKIRI [core 0] 66.249.79.48 - GET /noticia/ 2557/secuestran-plantas-de-marihuana-y-una-moto since 1557403307 Thu May 9 09:02:48 2019 - HARAKIRI !!! end of worker 8 status !!! Thu May 9 09:15:00 2019 - *** HARAKIRI ON WORKER 10 (pid: 9931, try: 1) *** Thu May 9 09:15:00 2019 - HARAKIRI !!! worker 10 status !!! Thu May 9 09:15:00 2019 - HARAKIRI [core 0] 66.249.65.142 - GET /amp/156013 /prevencion-salud-show-cantando-con-adriana since 1557404039 Thu May 9 09:15:00 2019 - HARAKIRI !!! end of worker 10 status !!! Thu May 9 09:29:15 2019 - *** HARAKIRI ON WORKER 22 (pid: 14688, try: 1) *** Thu May 9 09:29:15 2019 - HARAKIRI !!! worker 22 status !!! Thu May 9 09:29:15 2019 - HARAKIRI [core 0] 181.95.11.146 - GET /noticia/ 73359/santa-fe-un-changarin-ayudo-a-una-mujer-y-ella-con-una-colecta-le- compro-una-bic since 1557404894 Thu May 9 09:29:15 2019 - HARAKIRI !!! end of worker 22 status !!! Thu May 9 11:05:20 2019 - *** HARAKIRI ON WORKER 38 (pid: 521, try: 1) *** Notice *that the problem is always on core 0*. I've searched the entire logs and I haven't found the problem presented in a different core number, it's always the core 0. I've already asked in the uWSGI forum, but it's unlikely uWSGI being the problem (the problem started happening without having updated or changed anything of uWSGI). Could this be an issue within web2py? I personally don't think so, but I would like to hear some opinions from the experts :) IMHO the problem could be related to virtualization (remember this is running on a Linode VPS). What do you think? Any comment or suggestion will be much appreciated. Thanks! Best regards, Lisandro. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/4d74f85a-c01f-4ffa-bbac-6a620c79d037%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
