Hi Jay, Good points. A few remarks below.
On Wed, Jul 29, 2015 at 11:16 PM, Jay Kreps <j...@confluent.io> wrote: > > I suggest we focus on threading and the current event-loop style of api > design since I think that is really the crux. > Agreed. I think ultimately though what you need to think about is, does an event > loop style of API make sense? That is the source of all the issues you > describe. This style of API is incredibly prevalent from unix select to > GUIs to node.js. It's a great way to model multiple channels of messages > coming in. It is a fantastic style for event processing. Programmers > understand this style of api though I would agree it is unusual compared to > blocking apis. But it is is a single threaded processing model. Even though this style of API is prevalent, my experience is that people often struggle to use it correctly in non-trivial apps. Typically due to blocking, misuse of threading or callback hell. Perhaps this is because people try to mix models as you suggest. If we want to move away from an event loop I'm not sure *any* aspect of the > current event loop style of api makes sense any more. I am not totally > married to event loops, but i do think what we have gives an elegant way of > implementing any higher level abstractions that would fully implement the > user's parallelism model. This is a strong reason in favour of the existing approach, I think. An interesting experiment would be to write an adapter from the new consumer to a higher-level API like scalaz-stream ( https://github.com/scalaz/scalaz-stream). It would be a positive indicator if that turns out to be straightforward. Best, Ismael
