For sure, in order to support haproxy/redis-sentinel asgi_redis would have to handle redis-py's ReadOnlyError and reopen the connection. Way above my pay grade to inject this into 1.X. It seems 2.X might suffer from the same issue as I don't see it handling this condition either. Still baffles me no one is using some for of fault-tolerant Redis in production yet. Surely others must have been bitten by this already.
I'll have to weigh the merits (and pain) of migrating to Channels 2 this late in the game, we don't have many consumers, but we do some funky things with session data since we are multi-tenant. On Sun, Aug 5, 2018 at 2:53 PM Andrew Godwin <and...@aeracode.org> wrote: > Channels 1 and its Redis bindings have different behaviours to 2, so this > might be improved there (but I can't guarantee it). > > Andrew > > On Sun, Aug 5, 2018 at 11:48 AM Filbert <timn...@gmail.com> wrote: > >> Wow, I thought I had this working using haproxy in front of >> redis-sentinel. haproxy fails over when redis-sentinel fails over within >> 30 seconds. >> >> Problem is Daphne (Channels 1.X), reports over and over: >> >> Error trying to receive messages: Error running script (call to >> f_3640886a0c8901ca9188f5f7a5f7a346145b9c5f): @user_script:3: @user_script: >> 3: -READONLY You can't write against a read only slave. >> >> >> Which I assume means Daphne doesn't detect a problem and just keeps the >> socket open on the (old) master that is now the slave. >> >> >> On Thursday, August 2, 2018 at 8:53:24 PM UTC-4, Andrew Godwin wrote: >>> >>> Yup, any sort of TCP-level proxy with failover will work for this sort >>> of thing, but you'll likely see some errors whenever you failover as >>> Channels loses its blocking connection that it's using to wait for messages >>> - as always, please test a failover before you rely on it! >>> >>> Andrew >>> >>> On Thu, Aug 2, 2018 at 5:50 PM Filbert <tim...@gmail.com> wrote: >>> >>>> Andrew, >>>> Just to be clear, there is no reason I can't use haproxy to front end >>>> redis sentinel to effectively hide all this from channels 1 or 2 right? >>>> Very surprised there isn't anyone running this in production that doesn't >>>> need some form of HA. Buried trying to get a product out the door right >>>> now, but as I get time and more familiar with channels I'll probably dig in >>>> deeper on this. >>>> Thanks. >>>> >>>> On Thursday, August 2, 2018 at 7:32:52 PM UTC-4, Andrew Godwin wrote: >>>>> >>>>> The Sentinel option was removed in the 2.0 rewrite as it didn't have >>>>> someone to help maintain it (I didn't have the time then and don't right >>>>> now), so you are right, there's currently no easy HA option for Redis with >>>>> Channels. >>>>> >>>>> If you're interested in one, I can give you some tips on what code >>>>> needs to be written, but it's not something I really want to maintain >>>>> myself as part of the greater Channels effort as HA solutions tend to be >>>>> quite workload-heavy to develop and test and end up being specific to >>>>> certain deployment scenarios. Contributions of code that could help are >>>>> welcome, though. >>>>> >>>>> Andrew >>>>> >>>>> On Wed, Aug 1, 2018 at 1:22 PM Filbert <tim...@gmail.com> wrote: >>>>> >>>>>> Well in digging through posts I see Sentinel may be an option, does >>>>>> that mean if my CHANNEL_LAYER would have a single host entry? >>>>>> >>>>>> On Wednesday, August 1, 2018 at 9:42:39 AM UTC-4, Filbert wrote: >>>>>>> >>>>>>> Thanks Andrew, >>>>>>> So with a load-balanced ten web server implementation using Channels >>>>>>> (and of course ten instances of redis) do I have a HA solution then? >>>>>>> RIght >>>>>>> now Django settings simply has a list of those servers in >>>>>>> CHANNEL_LAYERS, >>>>>>> what if (say) three of them crash, will Channels continue to try and >>>>>>> hash >>>>>>> to the dead servers? Again, I'll accept a brief set of errors, but if >>>>>>> this >>>>>>> effectively causes Channels to continue to error that is not my >>>>>>> definition >>>>>>> of HA. >>>>>>> Thanks. >>>>>>> >>>>>>> On Sunday, July 22, 2018 at 8:09:16 PM UTC-4, Andrew Godwin wrote: >>>>>>>> >>>>>>>> I don't think redis-cluster supports BLPOP properly still so I >>>>>>>> don't think you can use it. >>>>>>>> >>>>>>>> Andrew >>>>>>>> >>>>>>>> On Sun, Jul 22, 2018 at 12:46 PM Filbert <tim...@gmail.com> wrote: >>>>>>>> >>>>>>>>> And just to be clear as to my options, using Channels 2.0 I can >>>>>>>>> use Redis cluster if I have to to get failover and fault tolerance, >>>>>>>>> correct? >>>>>>>>> >>>>>>>>> On Sunday, July 22, 2018 at 12:57:22 PM UTC-4, Andrew Godwin wrote: >>>>>>>>>> >>>>>>>>>> I'm afraid I've never used Twemproxy, so I can't really comment >>>>>>>>>> on it, but if you don't care about data loss, it's probably fine? >>>>>>>>>> >>>>>>>>>> Andrew >>>>>>>>>> >>>>>>>>>> On Fri, Jul 20, 2018 at 3:30 PM Filbert <tim...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Going to have 10 or more EC2 instances running Channels 2.0. I >>>>>>>>>>> really don't want to add the complexity of Redis Cluster or Redis >>>>>>>>>>> sentinel. Is a decent fault tolerant solution to implement >>>>>>>>>>> Twemproxy, >>>>>>>>>>> instead. I don't care about data loss if an instance crashes, but >>>>>>>>>>> I do >>>>>>>>>>> care that the application continues to operate on the remaining >>>>>>>>>>> instances. >>>>>>>>>>> >>>>>>>>>>> Is this a workable solution? >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "Django users" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to django-users...@googlegroups.com. >>>>>>>>>>> To post to this group, send email to django...@googlegroups.com. >>>>>>>>>>> Visit this group at https://groups.google.com/group/django-users >>>>>>>>>>> . >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/django-users/d1d50730-c1ba-4674-8ef8-54294158d0f6%40googlegroups.com >>>>>>>>>>> <https://groups.google.com/d/msgid/django-users/d1d50730-c1ba-4674-8ef8-54294158d0f6%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Django users" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to django-users...@googlegroups.com. >>>>>>>>> To post to this group, send email to django...@googlegroups.com. >>>>>>>>> Visit this group at https://groups.google.com/group/django-users. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/django-users/42fe264e-7f1f-426a-82c6-36aed944a923%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/django-users/42fe264e-7f1f-426a-82c6-36aed944a923%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Django users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to django-users...@googlegroups.com. >>>>>> To post to this group, send email to django...@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/django-users. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/django-users/ae01076f-732c-4574-b878-ccfd7d9ff20d%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/django-users/ae01076f-732c-4574-b878-ccfd7d9ff20d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Django users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to django-users...@googlegroups.com. >>>> To post to this group, send email to django...@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/django-users. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/django-users/fac4a285-610e-4814-9621-0246ed82ccb4%40googlegroups.com >>>> <https://groups.google.com/d/msgid/django-users/fac4a285-610e-4814-9621-0246ed82ccb4%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-users+unsubscr...@googlegroups.com. >> To post to this group, send email to django-users@googlegroups.com. >> Visit this group at https://groups.google.com/group/django-users. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-users/ca3c7d0a-435d-4d69-968f-acd1d4364d2f%40googlegroups.com >> <https://groups.google.com/d/msgid/django-users/ca3c7d0a-435d-4d69-968f-acd1d4364d2f%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "Django users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/django-users/3_IKqTn-SVE/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > django-users+unsubscr...@googlegroups.com. > To post to this group, send email to django-users@googlegroups.com. > Visit this group at https://groups.google.com/group/django-users. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-users/CAFwN1ur%2B%3DSg6Z5Nj0oTQ5W3AXXF8NLhAooCt74D4%2By%3DbzaAwBg%40mail.gmail.com > <https://groups.google.com/d/msgid/django-users/CAFwN1ur%2B%3DSg6Z5Nj0oTQ5W3AXXF8NLhAooCt74D4%2By%3DbzaAwBg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAK09zopPmSZAp5VsuqDaO3cYdJsRPi8gKWfk9SLm2Yg%3D8s3P8Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
