Hello Werner, 1) Regarding the long transaction, we did discuss about this in the design doc ( https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#bookmark=id.z9b0a6k2a3zm), and as stated the main reason we chose to go with the current design is to preserve offset ordering; and if it is becoming a critical feature request, we may consider allow users to read either in offset ordering or transaction ordering, in latter case a single long transaction will not stall others.
2) Regarding the transactional consumption behavior, my understanding is you want to have `consumer.poll()` return all records committed in a single transaction right? My take is that this is again depending on the transaction ordering mentioned above, since a single topic partition may have multiple producers sending to it, resulting in multiple transactions interleaving with each other on that log. In order to return a single completed transaction's all messages you are likely to get "non-consecutive" messages. But again if we decide transaction ordering based consumption is common and necessary we may likely just to this while implementing the feature. Guozhang On Tue, Jul 4, 2017 at 4:29 AM, Daehn, Werner <werner.da...@sap.com> wrote: > While the new transactional consistency feature is definitely a move into > the right direction, I find the current design limitations quite severe. > Understandable but severe. > > First is the offset being the add-time, not the commit time. The > consequence of that is, if one transactional producer adds rows every once > a while and does commit minutes or hours later, all consumers with > read=committed will not see any other producer's data until this one did > the commit. Will decrease the stability of the overall solution, won't it? > Would have been better to use the commit time as the offset value. Harder > to implement but more stable. > > The other issue is at the consumer side. Current implementation does not > provide transaction guarantees on the consumers. Again, for logical reasons > and to simplify the implementation, but as a user you want to have both - > when data is committed together you want to read the entire transaction as > one unit and complete. > > Are there any plans to go beyond the current implementation? > > Thanks in advance > -- -- Guozhang
