On Thu, May 5, 2016 at 9:28 PM, Anssi Kääriäinen <[email protected]> wrote:
> On Thursday, May 5, 2016, Andrew Godwin <[email protected]> wrote: >> >> Do you have a link to the presentation about them removing it? >> > > https://youtu.be/839rskyJaro around 34 minutes and onwards, another > version is https://youtu.be/2dG5HeM7MvA at 24 minutes. > Summarising their issues here for future people to save on video watching: - Large response bodies/streamed/chunked bodies are hard to serialise into Redis - Working out the order to restart things in is tricker than standard HTTP loadbalancing - Healthchecking the Rails process was difficult as they didn't listen on HTTP (just to Redis) - Hard to do rate-limiting and monitoring as it lacked context - HAProxy got a lot more featureful and grew the features they needed (hot-swapping of backends), and was a lot more battle-tested In general, it seems they were mainly using it to achieve HA, and it didn't quite pull through for them on that front. Some of these issues also apply to Channels, some less so as it's not designed as a HA solution and just being used for that. In particular, they were doing a lot more to keep requests around, whereas Channels will drop them if it's unsure what's going on, which gives it a lot more leeway (but makes it less "perfect" at HA). > > They were tackling a bit different problem, so their lessons might not > apply. Most importantly they weren't aiming for websockets at all, just for > high availability and throughput for normal HTTP traffic. On the other > hand, the architecture of broxy itself is pretty much the same as Daphne, > and they felt they had problems with it. > I need to start writing up my preferred large-scale deployment architecture for Channels more, which is "small clusters of interfaces and workers with an external loadbalancer across all clusters"; all of the stories I've heard, this included, the common theme is trying to push all of the site traffic through just the one bus and having great issues when it keels over under unexpected load or machine failure. I'm also starting to think we should have a local shared-memory channel layer that only works between processes on the one machine, so one could in theory just run clusters like this on decently sized multicore boxes; that would make a lot of people happier who don't want to have to run Redis in production but still want the multi-process behaviour that Channels gives. Running Channels would then just mean two processes, Daphne and workers. > > The way we have approached recent feature additions is that we let them > prove themselves outside of core. I think the crucial question is why > Channels can't do this, that is why can't we wait some time and let actual > production deployments prove the design? > > I know South is an example of why some features have a hard time living > outside core. But DRF is an example that a feature crucial to modern web > development can work very well outside of core. > > Channels sits somewhere between these two; it's not quite as deeply involved in hacking around core parts of Django as South was, but it's a bit deeper than DRF gets since it technically inserts itself under the entire view/URL routing system and sits alongside the WSGI handling code. My reason for pushing to get it in is that a lot of the utility of Channels is not in the code itself but what it enables; most of the things I would like to build can sit on top as third-party packages and get pulled in later if they make sense (e.g. a nice set of generic websocket consumers, or a high-level model-changes streaming solution). If it lives as some third-party package for another 8 months, it's a lot harder to sound that rally cry to build up the necessary community of layers on top of it that complete the ecosystem; plus, there's the obvious human factor that it will likely place the same workload on a much smaller set of main contributors. The situation is helped by the fact that the code that will likely need most polish and iteration - Daphne and asgi_redis - will still live outside Django's core codebase, giving us some more flexibility in how we handle changes (though I would like to still provide similar levels of stability and security assurances on them) - and that we'll be labelling it as "provisional" in 1.10 (similar to how PEP 411 describes it for Python), meaning we get one last chance to tweak things in 1.11 before we lock the API in for good. Andrew -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upU%2B_jm4VHP9C%2B1dtfsrfNqyWHd1Q9yAJ-jvcm5%3DULC%3DQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
