Thanks for this.  Let me know if you find a resolution to the 'saving to 
disk' latency issue.  Redis sessions would be an improvement, but I'd want 
to ensure disk-persistence for them (but not for cached things like search 
results).  How many sessions are you storing, and how much RAM does it 
consume?

On Thursday, 6 June 2019 20:33:28 UTC+1, Lisandro wrote:
>
> If you're going to add Redis, let me add a couple of comments about my own 
> experience:
>
>  - Using Redis to store sessions (not only to cache) was a huge 
> improvement in my case. I have public websites, some of them with much 
> traffic, so my app handles many sessions. I was using the database for 
> handling sessions, but when I changed to Redis, the performance improvement 
> was considerable. 
>
>  - Do some tests with the argument "with_lock" available in RedisCache and 
> RedisSessions (from gluon.contrib). In my specific case, using 
> with_lock=False is better, but of course this depends on each specific 
> scenario.
>
>  - An advise: choose proper values for "maxmemory" and "maxmemory-policy" 
> options from Redis configuration. The first one sets the max amount of 
> memory that Redis is allowed to use, and "maxmemory-policy" allows you to 
> choose how Redis should evict keys when it hits the maxmemory: 
> https://redis.io/topics/lru-cache. 
>
>
> El jueves, 6 de junio de 2019, 12:15:38 (UTC-3), Tim Nyborg escribió:
>>
>> This is really good to know.  I've a similar architecture to you, and am 
>> planning to add redis to the stack soon.  Knowing about issues to be on the 
>> lookout for is very helpful.
>>
>> On Friday, 24 May 2019 16:26:50 UTC+1, Lisandro wrote:
>>>
>>> I've found the root cause of the issue: the guilty was Redis.
>>>
>>> This is what was happening: Redis has an option for persistance 
>>> <https://redis.io/topics/persistence> wich stores the DB to the disk 
>>> every certain amount of time. The configuration I had was the one that 
>>> comes by default with Redis, that stores the DB every 15 minutes if at 
>>> least 1 key changed, every 5 minutes if at least 10 keys changed, and every 
>>> 60 seconds if 10000 keys changed. My Redis instance was saving DB to the 
>>> disk every minute, and the saving process was taking about 70 seconds. 
>>> Apparently, during that time, many of the requests were hanging. What I did 
>>> was to simply disable the saving process (I can do it in my case because I 
>>> don't need persistance). 
>>>
>>> I'm not sure why this happens. I know that Redis is single-threaded, but 
>>> its documentation states that many tasks (such as saving the DB) run in a 
>>> separate thread that Redis creates. So I'm not sure how is that the process 
>>> of saving DB to the disk is making the other Redis operations hang. But 
>>> this is what was happening, and I'm able to confirm that, after disabling 
>>> the DB saving process, my application response times have decreased to 
>>> expected values, no more timeouts :)
>>>
>>> I will continue to investigate this issue with Redis in the proper 
>>> forum. 
>>> I hope this helps anyone facing the same issue.
>>>
>>> Thanks for the help!
>>>
>>> El lunes, 13 de mayo de 2019, 13:49:26 (UTC-3), Lisandro escribió:
>>>>
>>>> After doing a lot of reading about uWSGI, I've discovered that "uWSGI 
>>>> cores are not CPU cores" (this was confirmed by unbit developers 
>>>> <https://github.com/unbit/uwsgi/issues/233#issuecomment-16456919>, the 
>>>> ones that wrote and mantain uWSGI). This makes me think that the issue I'm 
>>>> experiencing is due to a misconfiguration of uWSGI. But as I'm a developer 
>>>> and not a sysadmin, it's being hard for me to figure out exactly what 
>>>> uWSGI 
>>>> options should I tweak. 
>>>>
>>>> I know this is out of the scope of this group, but I'll post my uWSGI 
>>>> app configuration anyway, in case someone still wants to help:
>>>>
>>>> [uwsgi]
>>>> pythonpath = /var/www/medios/
>>>> mount = /=wsgihandler:application
>>>> master = true
>>>> workers = 40
>>>> cpu-affinity = 3
>>>> lazy-apps = true
>>>> harakiri = 60
>>>> reload-mercy = 8
>>>> max-requests = 4000
>>>> no-orphans = true
>>>> vacuum = true
>>>> buffer-size = 32768
>>>> disable-logging = true
>>>> ignore-sigpipe = true
>>>> ignore-write-errors = true
>>>> listen = 65535
>>>> disable-write-exception = true
>>>>
>>>>
>>>> Just to remember, this is running on a machine with 16 CPUs.
>>>> Maybe I should *enable-threads*, set *processes* options and maybe 
>>>> tweak *cpu-affinity. *
>>>> My application uses Redis for caching, so I think I can enable threads 
>>>> safely. 
>>>> What do you think?
>>>>
>>>>
>>>> El jueves, 9 de mayo de 2019, 21:10:57 (UTC-3), Lisandro escribió:
>>>>>
>>>>> I've checked my app's code once again and I can confirm that it 
>>>>> doesn't create threads. It only uses subprocess.cal() within functions 
>>>>> that 
>>>>> are called in the scheduler environment, I understand that's the proper 
>>>>> way 
>>>>> to do it because those calls don't run in uwsgi environment.
>>>>>
>>>>> In the other hand, I can't disable the master process, I use 
>>>>> "lazy-apps" and "touch-chain-reload" options of uwsgi in order to achieve 
>>>>> graceful reloading, because acordingly to the documentation about 
>>>>> graceful reloading 
>>>>> <https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html>
>>>>> :
>>>>> *"All of the described techniques assume a modern (>= 1.4) uWSGI 
>>>>> release with the master process enabled."*
>>>>>
>>>>> Graceful reloading allows me to update my app's code and reload uwsgi 
>>>>> workers smoothly, without downtime or errors. What can I do if I can't 
>>>>> disable master process?
>>>>>
>>>>> You mentioned the original problem seems to be a locking problem due 
>>>>> to threads. If my app doesn't open threads, where else could be the cause 
>>>>> of the issue? 
>>>>>
>>>>> The weirdest thing for me is that the timeouts are always on core 0. I 
>>>>> mean, uwsgi runs between 30 and 45 workers over 16 cores, isn't too much 
>>>>> of 
>>>>> a coincidence that requests that hang correspond to a few workers always 
>>>>> assigned on core 0?
>>>>>
>>>>>
>>>>> El jueves, 9 de mayo de 2019, 17:10:19 (UTC-3), Leonel Câmara escribió:
>>>>>>
>>>>>> Yes I meant stuff exactly like that.
>>>>>>
>>>>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/web2py/3445ab99-3d05-4ea5-a504-58eab4b9a12e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to