Hi Rajasekar,
You said that ConsumerOffsetChecker shows that new topics are successfully
consumed and the lag is 0. If that's the case, can you verify that there
is data on the source cluster for these new topics? If there is no data at
the source, MirrorMaker will only assign consumer streams to
Magnus,
this sounds like an interesting idea. But at the actual state of Kafka
this would mean, that I would have to extend the Kafka Producer and
Consumer
classes on my own to support that kind of message/file transfer, wouldn't
it?
This is a bit too much effort for me at the moment.
But of cou
Hello,
I wrote a simple junit test to test a Kafka producer.
public class KafkaProducerTest {
private int brokerId = 0;
private String topic = "test";
@Test
public void producerTest() throws InterruptedException {
// setup Zookeeper
String zkConnect = TestZKUtil
The topic maybe is not created at the broker yet ... take a look at
ProducerTest.scala as example
You could try TestUtils.waitUntilMetadataIsPropagated(servers, topic, 0,
500) to assert that the topic is in fact at the broker before sending after
creating it
/*
I tried that, but I still got
junit.framework.AssertionFailedError: Partition [test,0] metadata not
propagated after timeout
at
kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:506)
at kafka.utils.TestUtils.waitUntilMetadataIsPropagated(TestUtils.scala)
...
Ho
The idea here is that many serializers actually need some configuration
properties and without this there is no way to pass these in. We could try
to be a little more user friendly by looking for a constructor that takes
properties, and, if that doesn't exist looking for a no-arg constructor. If
th
> public class MyKafkaEncoder implements Encoder {
> // This constructor is expected by the kafka producer, used by reflection
> * public MyKafkaEncoder(VerifiableProperties props) {*
> *// what can I do with this?*
> * }*
>
You don't need to do anything with the argument - the producer co
Sadly it's not easily reproducible, it seems rather random while we're starting
large numbers of consumer processes. Out of the 300 or so we restarted during
our last release, I saw this happen to 3 of them. And we stagger our restarts
in groups of 100.
--
Ian Friedman
On Thursday, Septe
I'll have to try it out, but I think we can just have
VerifiableProperties extend from java.util.Properties (so your encoder
can just have Properties in its constructor). The producer can
internally instantiate VerifiableProperties, pass that in to
instantiate the encoder and do verification afterw
So, it seems that if I want to set a custom serializer class on the
producer (in 0.8), I have to use a class that includes a special
constructor like:
public class MyKafkaEncoder implements Encoder {
// This constructor is expected by the kafka producer, used by reflection
* public MyKafkaEncod
Hi Monika,
For Kafka-0.8.0 you will have to register beans with JmxTransServer like
this:
Server kafkaJMXServer = new Server("locahost","");
Query byteInQuery = new Query();
*
byteInQuery.setObj("\"kafka.server\":type=\"BrokerTopicMetrics\",name=\"AllTopicsBytesInPerSec\"");
Filed: https://issues.apache.org/jira/browse/KAFKA-1049
On Fri, Sep 6, 2013 at 2:04 PM, Joel Koshy wrote:
> I'll have to try it out, but I think we can just have
> VerifiableProperties extend from java.util.Properties (so your encoder
> can just have Properties in its constructor). The producer
12 matches
Mail list logo
