This is really good to know. I've a similar architecture to you, and am planning to add redis to the stack soon. Knowing about issues to be on the lookout for is very helpful.
On Friday, 24 May 2019 16:26:50 UTC+1, Lisandro wrote: > > I've found the root cause of the issue: the guilty was Redis. > > This is what was happening: Redis has an option for persistance > <https://redis.io/topics/persistence> wich stores the DB to the disk > every certain amount of time. The configuration I had was the one that > comes by default with Redis, that stores the DB every 15 minutes if at > least 1 key changed, every 5 minutes if at least 10 keys changed, and every > 60 seconds if 10000 keys changed. My Redis instance was saving DB to the > disk every minute, and the saving process was taking about 70 seconds. > Apparently, during that time, many of the requests were hanging. What I did > was to simply disable the saving process (I can do it in my case because I > don't need persistance). > > I'm not sure why this happens. I know that Redis is single-threaded, but > its documentation states that many tasks (such as saving the DB) run in a > separate thread that Redis creates. So I'm not sure how is that the process > of saving DB to the disk is making the other Redis operations hang. But > this is what was happening, and I'm able to confirm that, after disabling > the DB saving process, my application response times have decreased to > expected values, no more timeouts :) > > I will continue to investigate this issue with Redis in the proper forum. > I hope this helps anyone facing the same issue. > > Thanks for the help! > > El lunes, 13 de mayo de 2019, 13:49:26 (UTC-3), Lisandro escribió: >> >> After doing a lot of reading about uWSGI, I've discovered that "uWSGI >> cores are not CPU cores" (this was confirmed by unbit developers >> <https://github.com/unbit/uwsgi/issues/233#issuecomment-16456919>, the >> ones that wrote and mantain uWSGI). This makes me think that the issue I'm >> experiencing is due to a misconfiguration of uWSGI. But as I'm a developer >> and not a sysadmin, it's being hard for me to figure out exactly what uWSGI >> options should I tweak. >> >> I know this is out of the scope of this group, but I'll post my uWSGI app >> configuration anyway, in case someone still wants to help: >> >> [uwsgi] >> pythonpath = /var/www/medios/ >> mount = /=wsgihandler:application >> master = true >> workers = 40 >> cpu-affinity = 3 >> lazy-apps = true >> harakiri = 60 >> reload-mercy = 8 >> max-requests = 4000 >> no-orphans = true >> vacuum = true >> buffer-size = 32768 >> disable-logging = true >> ignore-sigpipe = true >> ignore-write-errors = true >> listen = 65535 >> disable-write-exception = true >> >> >> Just to remember, this is running on a machine with 16 CPUs. >> Maybe I should *enable-threads*, set *processes* options and maybe tweak >> *cpu-affinity. * >> My application uses Redis for caching, so I think I can enable threads >> safely. >> What do you think? >> >> >> El jueves, 9 de mayo de 2019, 21:10:57 (UTC-3), Lisandro escribió: >>> >>> I've checked my app's code once again and I can confirm that it doesn't >>> create threads. It only uses subprocess.cal() within functions that are >>> called in the scheduler environment, I understand that's the proper way to >>> do it because those calls don't run in uwsgi environment. >>> >>> In the other hand, I can't disable the master process, I use "lazy-apps" >>> and "touch-chain-reload" options of uwsgi in order to achieve graceful >>> reloading, because acordingly to the documentation about graceful >>> reloading >>> <https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html> >>> : >>> *"All of the described techniques assume a modern (>= 1.4) uWSGI release >>> with the master process enabled."* >>> >>> Graceful reloading allows me to update my app's code and reload uwsgi >>> workers smoothly, without downtime or errors. What can I do if I can't >>> disable master process? >>> >>> You mentioned the original problem seems to be a locking problem due to >>> threads. If my app doesn't open threads, where else could be the cause of >>> the issue? >>> >>> The weirdest thing for me is that the timeouts are always on core 0. I >>> mean, uwsgi runs between 30 and 45 workers over 16 cores, isn't too much of >>> a coincidence that requests that hang correspond to a few workers always >>> assigned on core 0? >>> >>> >>> El jueves, 9 de mayo de 2019, 17:10:19 (UTC-3), Leonel Câmara escribió: >>>> >>>> Yes I meant stuff exactly like that. >>>> >>> -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/1a56441a-1e22-4168-880b-777c3da8b837%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
