[
https://issues.apache.org/jira/browse/KAFKA-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269597#comment-16269597
]
Jason Gustafson commented on KAFKA-6278:
----------------------------------------
Thanks for the report. This was an unfortunate tradeoff in the design of the
transactional producer, but it simplified the API and the implementation
considerably. One of the main benefits to sharing the producer instances is
potentially better batching. However, the message batch format does not support
messages from separate transactions, so we apparently lose some of that benefit
even if the API supported it. There is still some potential benefit in sharing
memory and network sockets. We have toyed with the idea of having a
{{ProducerPool}} which could be used to get {{KafkaProducer}} instances which
share the same underlying memory pool and network resources. That way, we
wouldn't need to make any changes to the existing API.
> Allow multiple concurrent transactions on a single producer
> -----------------------------------------------------------
>
> Key: KAFKA-6278
> URL: https://issues.apache.org/jira/browse/KAFKA-6278
> Project: Kafka
> Issue Type: Improvement
> Reporter: Tim Cuthbertson
>
> It's recommended to share a producer between threads, because it's likely
> faster / cheaper.
> However with the transactional API there's a big caveat. If you're using
> transactions, every message sent to a given producer instance will be
> considered part of the "active transaction" regardless of what thread it came
> from. Furthermore, if two threads want to use transactions on a shared
> producer instance, it (probably) won't work.
> Possible fix: add an API which exposes the transaction ID to the user,
> instead of making it internal state of the producer. e.g.:
> {noformat}
> Transaction tx = producer.beginTransaction()
> producer.send(tx, message)
> producer.commitTransaction(tx)
> {noformat}
> That way, it's explicit which transaction a message will be part of, rather
> than the current state which is "the open transaction, which may have been
> opened by an unrelated thread".
> See also initial discussion on slack:
> https://confluentcommunity.slack.com/archives/C488525JT/p1511739734000012
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
