> have a Kafka cluster across availability zones, data will be replicated to ... > single timeout on the broker side. In theory, different producers may want > to pick different replication time depending on the topics being sent.
I think Becket raises a good point here in that the above configurations are best known by the Kafka cluster operators and not necessarily by the users (producers). So right now users end up having to either know such details about the deployment when in fact it should be set by the people who (may manually) assign partitions to brokers; or they have to "guess" the timeouts or be content with defaults. Actually this would end up being a LogConfig which would be per-topic - i.e., it won't necessarily be a single timeout on the broker. Thanks, Joel On Thu, May 28, 2015 at 04:17:08PM -0700, Jun Rao wrote: > Hi, Jiangjie, > > The replication time may vary a bit for different partitions. For example, > a partition with more replicas may take a bit more time to propagate the > messages. Also, the replication time depends on network latency. If you > have a Kafka cluster across availability zones, data will be replicated to > nodes within the same zone a bit faster than those outside of the zone. So, > I am not sure if it's better to just reason about the replication time as a > single timeout on the broker side. In theory, different producers may want > to pick different replication time depending on the topics being sent. > > Thanks, > > Jun > > On Tue, May 26, 2015 at 4:46 PM, Jiangjie Qin <j...@linkedin.com.invalid> > wrote: > > > Hi, > > > > I am updating the wiki for KIP-19 and wondering why we have a replication > > timeout on producer side and in producer request? > > > > From what I understand this is a server side setting and the reasons we > > need this replication timeout is because we want to control the purgatory > > size. If that is the case should we just have the replication timeout as a > > broker configuration? > > The downside of having it on server side might be that producer could have > > a request timeout/socket timeout smaller than replication timeout. In this > > case we can put request timeout in producer request and if the request > > timeout is smaller than replication timeout on server side, we return a > > mis-cofiguration exception. > > > > So we can have a producer request V1 which removes ack timeout but adds > > request timeout. This will give user a cleaner timeout configurations on > > producer side as well. > > > > What do people think about this? > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > >
