[
https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14628413#comment-14628413
]
Guozhang Wang commented on KAFKA-1835:
--------------------------------------
I agree that either way there will be an API change. Given that, I think the
question is how could we make it a bit easier and perhaps more intuitive for
users to code. Personally I prefer "exception" over "future" since I feel it is
actually less clear of a function definition that "partitionsFor could also be
possibly blocked for up to a configured timeout value, yet it returns you a
future", plus the blocking behavior is not controllable dynamically at runtime
since it is defined by configs, right?
For [~becket_qin]'s follow-up case 3) example, in the exception approach it
will become :
{code}
try {
send(..); // send to any partition of the topic
} catch (TopicUnknownException tue){
// handle exception
} catch (BufferExhaustedException bee) {
// handle exception..
} catch (//other exceptions) { .. }
{code}
And if the user really wants to send to a specific partition, I think they need
to expect blocking somehow since in that case they really want to make sure the
topic exists already, hence calling a blocking partitionsFor() in order to
figure out the partition id would be fine.
For the same reason above, I would also prefer to NOT let partitionsFor's
blocking behavior be bounded by max.block.ms as well, but be
blocking-until-topic-available if it does not return the Future but the list of
partitions directly.
Regarding the preinit.topics config, I think it is completely fine to add it if
users do want that, and per implementation it would be blocking partitionsFor()
calls in the constructor, since it is less likely that users want to complete
the initialization of the producer if the specified topics are not even
available to send yet.
> Kafka new producer needs options to make blocking behavior explicit
> -------------------------------------------------------------------
>
> Key: KAFKA-1835
> URL: https://issues.apache.org/jira/browse/KAFKA-1835
> Project: Kafka
> Issue Type: Improvement
> Components: clients
> Affects Versions: 0.8.2.0, 0.8.3, 0.9.0
> Reporter: Paul Pearcy
> Fix For: 0.8.3
>
> Attachments: KAFKA-1835-New-producer--blocking_v0.patch,
> KAFKA-1835.patch
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The new (0.8.2 standalone) producer will block the first time it attempts to
> retrieve metadata for a topic. This is not the desired behavior in some use
> cases where async non-blocking guarantees are required and message loss is
> acceptable in known cases. Also, most developers will assume an API that
> returns a future is safe to call in a critical request path.
> Discussing on the mailing list, the most viable option is to have the
> following settings:
> pre.initialize.topics=x,y,z
> pre.initialize.timeout=x
>
> This moves potential blocking to the init of the producer and outside of some
> random request. The potential will still exist for blocking in a corner case
> where connectivity with Kafka is lost and a topic not included in pre-init
> has a message sent for the first time.
> There is the question of what to do when initialization fails. There are a
> couple of options that I'd like available:
> - Fail creation of the client
> - Fail all sends until the meta is available
> Open to input on how the above option should be expressed.
> It is also worth noting more nuanced solutions exist that could work without
> the extra settings, they just end up having extra complications and at the
> end of the day not adding much value. For instance, the producer could accept
> and queue messages(note: more complicated than I am making it sound due to
> storing all accepted messages in pre-partitioned compact binary form), but
> you're still going to be forced to choose to either start blocking or
> dropping messages at some point.
> I have some test cases I am going to port over to the Kafka producer
> integration ones and start from there. My current impl is in scala, but
> porting to Java shouldn't be a big deal (was using a promise to track init
> status, but will likely need to make that an atomic bool).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
