could you explain an example deadlock scenario concretely? by blocking we mean a function like *fn sendBlockingBoundedChan(&self, t:T)-> ()*
that doesn't return until the message in enqueued? Why should that even be the default sending procedure? On Thu, Dec 19, 2013 at 1:40 PM, Patrick Walton <[email protected]> wrote: > Again, isn't blocking just likely to lead to deadlocks? > > Patrick > > Matthieu Monrocq <[email protected]> wrote: >> >> Also working in a distributed system, I cannot emphasize enough how >> back pressure is essential. >> >> With back pressure, you offer the producer a chance to react: it can >> decide to drop the message, send it over another channel, keep it around >> for later, etc... >> >> Furthermore, it is relatively easy to build an unbounded channel over a >> bounded one: just have the producer queue things. Depending on whether >> sequencing from multiple producers is important or not, this queue can be >> either shared or producer-local, with relative ease. >> >> >> Regarding the various behaviors that may be implemented, most behaviors >> can actually be implemented outside of the channel implementation: >> >> + dropping the message can be implemented on producer side: if it cannot >> queue, it just goes on >> + crashing is similar: if it cannot queue, crash >> + blocking is generally a good idea, but if a timed-wait primitive exists >> then I imagine an infinite (or close enough) duration would be sufficient >> >> >> So it might be more interesting to reason in terms of primitives, and >> those might be more methods than types (hopefully): >> >> (1) immediate queueing (returning an error), a special case of time-bound >> queueing which may be slightly more efficient >> (2) time-bound queueing (returning an error after the timeout) >> (3) immediate + exchange with head (in which case the producer also >> locally acts as a consumer, this might be tricky to pull off efficiently on >> Single Consumer queues) >> (4) immediate + atomic subscription to "place has been freed" event in >> case of full-queue >> >> (Note: (4) somehow implies a dual channel, if you have a MPSC a >> back-channel SPMC is created to dispatch the "space available" >> notifications... which can be a simple counter, obviously; this >> back-channel must be "select"-able so that producers that usually block on >> other stuff can use a "space available" event to unblock) >> >> I cannot see another interesting primitive, at the moment. >> >> -- Matthieu >> >> >> On Thu, Dec 19, 2013 at 7:25 PM, Matthieu Monrocq < >> [email protected]> wrote: >> >>> >>> >>> >>> On Thu, Dec 19, 2013 at 7:23 PM, Kevin Ballard <[email protected]> wrote: >>> >>>> Here’s an example from where I use an infinite queue. >>>> >>>> I have an IRC bot, written in Go. The incoming network traffic of this >>>> bot is handled in one goroutine, which parses each line into its >>>> components, and enqueues the result on a channel. The channel is very >>>> deliberately made infinite (via a separate goroutine that stores the >>>> infinite buffer in a local slice). The reason it’s infinite is because the >>>> bot needs to be resilient against the case where either the consumer >>>> unexpectedly blocks, or the network traffic spikes. The general assumption >>>> is that, under normal conditions, the consumer will always be able to keep >>>> up with the producer (as the producer is based on network traffic and not >>>> e.g. a tight CPU loop generating messages as fast as possible). >>>> Backpressure makes no sense here, as you cannot put backpressure on the >>>> network short of letting the socket buffer fill up, and letting the socket >>>> buffer fill up with cause the IRC network to disconnect you. So the >>>> overriding goal here is to prevent network disconnects, while assu ming >>>> that the consumer will be able to catch up if it ever gets behind. >>>> >>>> This particular use case very explicitly wants a dynamically-sized >>>> infinite channel. I suppose an absurdly large channel would be acceptable, >>>> because if the consumer ever gets e.g. 100,000 lines behind then it’s in >>>> trouble already, but I’d rather not have the memory overhead of a >>>> statically-allocated gigantic channel buffer. >>>> >>> >>> I feel the need to point out that the producer could locally queue the >>> messages before sending over the channel if it were bounded. >>> >>> >>>> >>>> -Kevin >>>> >>>> On Dec 19, 2013, at 10:04 AM, Jason Fager <[email protected]> wrote: >>>> >>>> Okay, parallelism, of course, and I'm sure others. Bad use of the word >>>> 'only'. The point is that if your consumers aren't keeping up with your >>>> producers, you're screwed anyways, and growing the queue indefinitely isn't >>>> a way to get around that. Growing queues should only serve specific >>>> purposes and make it easy to apply back pressure when the assumptions >>>> behind those purposes go awry. >>>> >>>> >>>> On Thursday, December 19, 2013, Patrick Walton wrote: >>>> >>>>> On 12/19/13 6:31 AM, Jason Fager wrote: >>>>> >>>>>> I work on a system that handles 10s of billions of events per day, and >>>>>> we do a lot of queueing. Big +1 on having bounded queues. Unbounded >>>>>> in-memory queues aren't, they just have a bound you have no direct >>>>>> control over and that blows up the world when its hit. >>>>>> >>>>>> The only reason to have a queue size greater than 1 is to handle >>>>>> spikes >>>>>> in the producer, short outages in the consumer, or a bit of >>>>>> out-of-phaseness between producers and consumers. >>>>>> >>>>> >>>>> Well, also parallelism. >>>>> >>>>> Patrick >>>>> >>>>> _______________________________________________ >>>>> Rust-dev mailing list >>>>> [email protected] >>>>> https://mail.mozilla.org/listinfo/rust-dev >>>>> >>>> _______________________________________________ >>>> Rust-dev mailing list >>>> [email protected] >>>> https://mail.mozilla.org/listinfo/rust-dev >>>> >>>> >>>> >>>> _______________________________________________ >>>> Rust-dev mailing list >>>> [email protected] >>>> https://mail.mozilla.org/listinfo/rust-dev >>>> >>>> >>> >> ------------------------------ >> >> Rust-dev mailing list >> [email protected] >> https://mail.mozilla.org/listinfo/rust-dev >> >> > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > > _______________________________________________ > Rust-dev mailing list > [email protected] > https://mail.mozilla.org/listinfo/rust-dev > >
_______________________________________________ Rust-dev mailing list [email protected] https://mail.mozilla.org/listinfo/rust-dev
