Hi Samuel, You'll notice if you look carefully that I avoid saying "async" almost anywhere in the Channels announcements or documentation, and when I do, it's qualified very carefully. Channels makes Django event-driven, but it does not provide full "in-process" async like Twisted, asyncio, or something like Pulsar.
Channels is actually kind of related to the actor model in some ways, but it's simpler; imagine it as a pool of identical actors that are all stateless, apart from the interface servers, which have the incoming socket and thus the state. The key thing is that Channels does nothing to make Django asynchronous. All the Django code will remain synchronous (the ORM, templating, etc.), and run in synchronous processes that just process one message after the next. The only part of the system that will require an async framework is the interface servers, as to be anywhere near efficient they need to serve more than one socket at once; right now we're using Twisted and asyncio for those, purely as they're incredibly mature and well-supported by the community, but you could swap out anything there that achieved the same goal (greenlets, for example, though I personally dislike their implicitness). Channels essentially takes the worker-prefork model that's been used in webservers for years and applies it to a lower-level message concept rather than HTTP requests (http requests are just one kind of message). This not only has the advantage of a lot of previous work and research into architecting and scaling these kinds of systems, it also means the actual amount of work in core Django isn't crazy; it's more about providing nice places to wrap up the worker loop and nice utilities around authentication and session management. If you read through the docs there's a bit more about this, but basically, the fundamental design goal here is that Channels be no harder to use than normal Django; indeed, it's almost exactly the same view concept, but instead of a request you get a message, and the newer developers I've interviewed about it seem to be comfortable with the extension of the current abstraction straight away. I don't ever want people to have to write asynchronous code if they don't want to; it's currently so easy to shoot yourself in the foot in almost any Python async solution that I don't think that makes for a good developer experience. The "complexity" of the message backends you describe is because they're network-transparent; there is indeed a simple, in-process one that uses threads too, but the whole point of this design is that you run Django over multiple processes on many machines, and in order to get something like that you need a place to funnel messages through (another potential backend design could use direct connection and process discovery, perhaps, or an Erlang-like process spawn-and-track design with an overarching controller process with access to many pre-selected machines). The idea of making an asynchronous Django is tangential to making Channels; this is more about changing the design abstraction of Django so we can fit the concept in, and then providing a neat solution that drops into the gap. There's nothing stopping someone making the worker processes for Channels async-aware as well, so they can run multiple message consumers at once, but that's such a complex and risk-fraught task that I don't think it's worth it when we can get 90% of the benefits with a much simpler and easier-to-design-and-understand system. If you want to see an example of actual async Django, see Amber's talk from Django Under the Hood this year, where she managed to get it working decently well under Twisted. I expect Channels to smooth out Django's performance mostly as a side-effect of the worker model providing demand-based load balancing by design, but I doubt it will be a huge increase until people start using the primitives it provides to offload tasks to other workers. I hope that somewhat long-winded explanation helps - if not, let me know what's still unclear and I'll try to tackle it. Clearing up what we're trying to do and communicating it is important to me; I want to make sure everyone knows so we can get good feedback on this stuff. For what it's worth, I did consider the actor model along with several other kinds of concurrency approaches (I've been prototyping bits of this for years); this approach ended up being the nicest design and the one I have the most confidence in that we can scale and shard to large work volumes. Andrew On Thu, Dec 17, 2015 at 10:30 AM, Samuel Bishop <[email protected]> wrote: > I'm uncertain how popular the suggestion will be but ... are "channels" > the right solution to our async/concurrency problems? ( I'm all for > django-channels the project and the work to solve these issues) > All the talk of channels, workers, queues and back ends in the > explanations I'm reading as I follow along with the progress of 'Django > Channels", just feels "less good" than solving this problem via Actor Model > Concurrency (championed by Erlang, Elixir, Stackless Python, etc). > > There is an excellent actor model concurrency library for python that > already supports 'async django'. "Pulsar" ( > http://pythonhosted.org/pulsar/index.html) and the documentation here > http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways > to use actor model concurrency to attack the issue of using Django with > asynchronous web stuff like Websockets, etc. > Some of what's being described in the explanations of Django channels > sounds very similar to what they provide to work around the limitations of > django's blocking request processing cycle other parts sound much more > complicated, for instance the need for various back ends > in-memory/redis/etc. > > I know Pulsar is currently Python 3.4 and newer only, but in the past it > supported 2.7, so anything we can learn from their concurrency approach can > work on Python 2.7 to maintain our backwards compatibility going forward. > You can see an example of Pulsar using python 2 if you look on the > benchmark page for their python powered 'redis clone' - > https://gist.github.com/lsbardel/8068579 - One of the benchmarks is using > pypy so the code definitely worked in a python 2.7 environment. > > Would the work to integrate channels as part of Django improve the ability > to run Django in a completely async mode and enhance its performance when > run under Pulsar? or would it just be something that needed to be turned > off if we wanted to use something like Pulsar to power our "async django". > Is there anything to be gained from using a more actor driven concurrency > model or adopting any of the other methods used in their 'pulse' django app? > I suppose I'm just trying to work out how to compare using Django-Channels > with using Pulse from the Pulsar project. > > On Wednesday, 16 December 2015 19:23:56 UTC+8, Andrew Godwin wrote: > >> >> On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann < >> [email protected]> wrote: >> >>> >>> >If I get it right -- Curtis' description is spot-on; some tasks will >>> >still >>> >need Celery, Channels will take care of many others. >>> >>> For me it's more a question of "is a client directly involved". >>> >> >> I don't agree; I think it's more "do you have enough workers to handle >> this". Channels makes all workers serve all requests, so if you're doing >> e.g. video encoding you can't segregate it from request handling. However, >> you could have multiple channel backends and thus multiple worker pools to >> achieve this. >> >> Andrew >> > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upsqkL9_qN6NfO7PB9xjHMaPrjL9whwpnmPo3Ru4ZbO0A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
