[
https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627474#comment-14627474
]
Joel Koshy commented on KAFKA-1835:
-----------------------------------
My earlier comment was more to clarify the reasoning. i.e., the statement that
users that want non-blocking behavior are okay with losing messages which isn't
necessarily true. I'm actually not opposed to the approach of going with
max.block.ms == 0 for the non-blocking case and throwing an exception on
metadata not being ready. I thought we even covered that option at the hangout
(but Becket wasn't terribly excited about exposing any metadata-related
exception in the first send to the user). From my p.o.v I don't think that's an
issue since users cannot really escape from knowing about partitions and thus
metadata.
This (future API) was just an alternate approach that came out of a
conversation with [~onurkaraman] in KAFKA-2275. We have a need for list-topics
(for some use cases such as ETL). We could just go with a blocking API there or
a config-driven (max.block.ms type) approach as we are considering currently in
the producer. Now that we are working through the change in the consumer API to
make the commit return a future we felt this could benefit from a similar
pattern. And if we are considering it for the consumer it would be beneficial
to have an identical API in the producer that also helps address the blocking
concerns that people have raised. Personally I like the future-oriented API
since (i) it makes it clear to the caller that some asynchronous handling is
involved and the result can be collected from the future (ii) has clear
semantics on blocking - the user can decide (at runtime) whether (or how much)
blocking is acceptable. The other approach to go with max.block.ms==0 and throw
an exception would also work but to be equally clear in the API, the API should
throw a checked exception (which is again an API change).
Yes it involves an API change - but I don't think we want to be completely
closed to changes in the API of the new (okay one year old) producer if
everyone agrees that a better and clearer API exists. I agree this is a pain
and should be weighed carefully. The barrier for change is much less on the new
consumer but I would be surprised if majority of users would be opposed to any
API changes in the new producer even at this stage as long as it is in the
right direction and older APIs are deprecated over time.
> Kafka new producer needs options to make blocking behavior explicit
> -------------------------------------------------------------------
>
> Key: KAFKA-1835
> URL: https://issues.apache.org/jira/browse/KAFKA-1835
> Project: Kafka
> Issue Type: Improvement
> Components: clients
> Affects Versions: 0.8.2.0, 0.8.3, 0.9.0
> Reporter: Paul Pearcy
> Fix For: 0.8.3
>
> Attachments: KAFKA-1835-New-producer--blocking_v0.patch,
> KAFKA-1835.patch
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The new (0.8.2 standalone) producer will block the first time it attempts to
> retrieve metadata for a topic. This is not the desired behavior in some use
> cases where async non-blocking guarantees are required and message loss is
> acceptable in known cases. Also, most developers will assume an API that
> returns a future is safe to call in a critical request path.
> Discussing on the mailing list, the most viable option is to have the
> following settings:
> pre.initialize.topics=x,y,z
> pre.initialize.timeout=x
>
> This moves potential blocking to the init of the producer and outside of some
> random request. The potential will still exist for blocking in a corner case
> where connectivity with Kafka is lost and a topic not included in pre-init
> has a message sent for the first time.
> There is the question of what to do when initialization fails. There are a
> couple of options that I'd like available:
> - Fail creation of the client
> - Fail all sends until the meta is available
> Open to input on how the above option should be expressed.
> It is also worth noting more nuanced solutions exist that could work without
> the extra settings, they just end up having extra complications and at the
> end of the day not adding much value. For instance, the producer could accept
> and queue messages(note: more complicated than I am making it sound due to
> storing all accepted messages in pre-partitioned compact binary form), but
> you're still going to be forced to choose to either start blocking or
> dropping messages at some point.
> I have some test cases I am going to port over to the Kafka producer
> integration ones and start from there. My current impl is in scala, but
> porting to Java shouldn't be a big deal (was using a promise to track init
> status, but will likely need to make that an atomic bool).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
