Thank you! Using put! callback to control backpressure is a very elegant solution. BTW there always has to be blocking somewhere. Netty uses selector to block at [1] and .setAutoRead causes respective channel to deregister itself from a selector [2], until put! completes.
Regarding the other way around, it seems to me from the quick look (sorry if I misinterpreted) that using futures to represent write completion will cause the eventual backpressure to block the thread, causing analogous drawback as a blocking put! in the "reading from multiple connection" case. [1] https://github.com/netty/netty/blob/220660e351b2a22112b19c4af45e403eab1f73ab/transport/src/main/java/io/netty/channel/nio/NioEventLoop.java#L625 [2] https://github.com/netty/netty/blob/220660e351b2a22112b19c4af45e403eab1f73ab/transport/src/main/java/io/netty/channel/nio/AbstractNioChannel.java#L173 Jozef On Wednesday, October 8, 2014 7:16:44 PM UTC+2, Zach Tellman wrote: > > The documentation for Manifold can explain the API better than I can > here. The point where that interacts with Netty w.r.t. backpressure is > here: > https://github.com/ztellman/aleph/blob/0.4.0/src/aleph/netty.clj#L109. > Here the stream represents data coming off the wire, and if the put onto > the stream is not immediately successful, backpressure is enabled until the > put completes. No blocking required anywhere. > > On Wed, Oct 8, 2014 at 10:10 AM, Jozef Wagner <jozef....@gmail.com > <javascript:>> wrote: > >> If you want to handle multiple TCP connections and async channels in one >> thread, you need a way how to block on both connections (wait for new input >> to arrive) and channels (wait for a free space in a buffer). Blocking only >> on connections will get you a busy loop if channels are full. If you could >> point me to the part of Aleph sources that handles this issue, I would be >> very grateful. I'm not familiar with netty API nor manifold's concepts, so >> I'm having trouble navigating in the Aleph sources. >> >> Thanks, >> Jozef >> >> On Wednesday, October 8, 2014 6:15:47 PM UTC+2, Zach Tellman wrote: >>> >>> I wasn't aware of hermod, that's interesting. I would still >>> characterize its approach to backpressure as "broken", though, since when >>> the queues get full it silently drops messages on the ground. In fairness, >>> this is very clearly documented, so it's less pernicious than some of the >>> other cases out there. >>> >>> Both core.async buffers and SelectableChannels have very particular >>> semantics, and I would be very surprised if they could be combined in that >>> way. It's perfectly possible to feed one into the other and handle >>> backpressure properly (again, I'm doing just that with Aleph 0.4.0, using >>> Netty), but it's a nuanced integration and easy to get wrong. >>> >>> On Wed, Oct 8, 2014 at 7:12 AM, <adrian...@mail.yu.edu> wrote: >>> >>>> Check out https://github.com/halgari/com.tbaldridge.hermod for an >>>> interesting take on this. >>>> >>>> On Wednesday, October 8, 2014 1:17:11 AM UTC-4, Sun Ning wrote: >>>>> >>>>> BTW, is there any network based core.async channel available now? >>>>> >>>>> On 10/08/2014 04:36 AM, adrian...@mail.yu.edu wrote: >>>>> >>>>> It's not about 'safety' (depending on what that means in this >>>>> context), but as Zach pointed out, if you aren't careful about >>>>> backpressure >>>>> you can run into performance bottlenecks with unrestrained async IO >>>>> operations because although they let you code as if you could handle an >>>>> unlimited amount of connections, obviously that isn't true. There is only >>>>> a >>>>> finite amount of data that can be buffered in and out of any network >>>>> according to its hardware. When you don't regulate that, your system will >>>>> end up spending an inordinate amount of time compensating for this. You >>>>> don't need to worry about this with "regular io" because the "thread per >>>>> connection" abstraction effectively bounds your activity within the >>>>> acceptable physical constraints of the server. >>>>> >>>>> On Tuesday, October 7, 2014 2:49:30 PM UTC-4, Brian Guthrie wrote: >>>>>> >>>>>> >>>>>> On Mon, Oct 6, 2014 at 12:10 AM, <adrian...@mail.yu.edu> wrote: >>>>>> >>>>>>> Zach makes an excellent point; I've used AsyncSocketChannels and its >>>>>>> irk (http://docs.oracle.com/javase/8/docs/api/java/nio/channels/ >>>>>>> AsynchronousServerSocketChannel.html), with core.async in the past. >>>>>>> Perhaps replacing your direct java.net.Sockets with nio classes that >>>>>>> can be >>>>>>> given CompletionHandlers (http://docs.oracle.com/javase >>>>>>> /7/docs/api/java/nio/channels/CompletionHandler.html) would be a >>>>>>> better fit. >>>>>>> >>>>>> >>>>>> Once I do some performance instrumentation I'll give that a shot. I >>>>>> admit that I'm not familiar with all the implications of using the nio >>>>>> classes; were I to switch, is it safe to continue using go blocks, or is >>>>>> it >>>>>> worth explicitly allocating a single thread per socket? >>>>>> >>>>>> Brian >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To post to this group, send email to clo...@googlegroups.com >>>>> Note that posts from new members are moderated - please be patient >>>>> with your first post. >>>>> To unsubscribe from this group, send email to >>>>> clojure+u...@googlegroups.com >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/clojure?hl=en >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to clojure+u...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To post to this group, send email to clo...@googlegroups.com >>>> Note that posts from new members are moderated - please be patient with >>>> your first post. >>>> To unsubscribe from this group, send email to >>>> clojure+u...@googlegroups.com >>>> For more options, visit this group at >>>> http://groups.google.com/group/clojure?hl=en >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Clojure" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>> topic/clojure/TVMQJwaij1U/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> clojure+u...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> <javascript:> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "Clojure" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/clojure/TVMQJwaij1U/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> clojure+u...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
