Refer the mail list
http://qnalist.com/questions/6002514/new-producer-metadata-update-problem-on-2-node-cluster
https://issues.apache.org/jira/browse/KAFKA-1843
On Wed, Sep 9, 2015 at 7:37 AM, Shushant Arora
wrote:
> Hi
>
> I have a kafka cluster with 3 brokers. I have a topic with ~50 partitio
We pass in all the producer properties when instantiating the partitioner.
Thanks,
Jun
On Thu, Oct 30, 2014 at 5:57 PM, Rajiv Kurian wrote:
> Hmm I am not sure how to do that. Do I have access to the
> VerifiableProperties object that is passed to the partitioner object?
>
> On Thu, Oct 30, 20
Yeah the intention of requiring that properties is to pipe through all the
configuration that goes to the producer to the partitioner. That way if
your partitioner needs to query some external system it can get the
configuration for that.
-Jay
On Thu, Oct 30, 2014 at 5:57 PM, Rajiv Kurian wrote:
Hmm I am not sure how to do that. Do I have access to the
VerifiableProperties object that is passed to the partitioner object?
On Thu, Oct 30, 2014 at 5:54 PM, Jun Rao wrote:
> Potentially, you can pass in state related info through the properties and
> use those to instantiate MyDeciderThingy.
Potentially, you can pass in state related info through the properties and
use those to instantiate MyDeciderThingy.
Thanks,
Jun
On Thu, Oct 30, 2014 at 11:46 AM, Rajiv Kurian wrote:
> Yes I am using the old producer. When I use the producer I do something
> like this:
>
> Properties config =
Will try to use the new producer ASAP but sadly dealing with an older code
base which cannot be migrated that easily. The idea to check for null and
instantiate seems like a good work around too. Thanks!
On Thu, Oct 30, 2014 at 11:58 AM, Joel Koshy wrote:
> I see - yes that is a limitation. You
I see - yes that is a limitation. You could get around that by doing
something similar - i.e., use reflection to create the MyDeciderThingy
object. Or you could check in the partition call if the
MyDeciderThingy object is null and if so create it.
Better yet, you can try using the new producer if
Yes I am using the old producer. When I use the producer I do something
like this:
Properties config = new Properties();
// put other stuff in the config.
// Put the partitioner class that kafka will instantiate.
config.put("partitioner.class", partitionerClass.getName());
return new Producer(
Not sure I follow - you just need to extend the Partitioner trait. You
don't _have_ to use that specific constructor.
It is slightly different with the new producer, but looks like you are
on the old producer.
On Thu, Oct 30, 2014 at 11:16:28AM -0700, Rajiv Kurian wrote:
> Actually I figured out
Actually I figured out what the problem was. My producer was using a
partitioner which was causing a null pointer exception. This actually
raises another question for me. I want some state in my partitioner and the
only constructor that Kafka seems to use is this one:
public MyPartitioner(Verifiab
Yes I see a ton of WARN messages on the broker logs of this form:
2014-10-30T17:21:54.281Z WARN [kafka-request-handler-6]
[state.change.logger ]: Broker 0 received invalid
LeaderAndIsr request with correlation id 158 from controller 0 epoch 29083
with an older leader e
Do you see any errors on the broker logs? Can you check the broker's
public access logs and see if there are topic metadata requests coming
in from the producer?
On Wed, Oct 29, 2014 at 07:15:15PM -0700, Rajiv Kurian wrote:
> I don't see anything else that is relevant. I traced the first of these
I don't see anything else that is relevant. I traced the first of these
error messages to figure out the ordering. It actually goes something like
this:
2014-10-30T01:51:32.400Z ERROR [ProducerSendThread-]
[k.producer.async.DefaultEventHandler] {}: Failed to collate messages by
top
This pattern seems to repeat:
2014-10-30T01:54:46.004Z ERROR [ProducerSendThread-]
[k.producer.async.DefaultEventHandler] {}: Failed to send requests for
topics myTopic with correlation ids in [1729,1736]
2014-10-30T01:54:46.008Z ERROR [ProducerSendThread-]
[k.prod
The log before that will show you the cause of the error. Could you dig
that out?
Thanks,
Jun
On Wed, Oct 29, 2014 at 6:43 PM, Rajiv Kurian wrote:
> I keep seeing these errors in my code that is just trying to send some data
> using an AsyncProducer:
>
> kafka.common.FailedToSendMessageExcepti
15 matches
Mail list logo
