We have a 3 node cluster ( 3 kafka + 3 ZK nodes ) . Recently we came across
a strange issue where we wanted to bring one of the node down from cluster
( 1 kafka + 1 zookeeper) for doing a maintenance . But the movement we
brought it to down on some of the topics ( only some partitions) consumers
of
I want to use some vital broker stats into a different logging system.
Can i read the broker specific data from zookeeper?
--
*Regards,*
*Ravi*
I am not sure I understand your question. Could you elaborate more on your
needs?
Guozhang
On Sat, Jun 21, 2014 at 9:24 AM, ravi singh wrote:
> How do I intercept Kakfa broker operation so that features such as
> security,logging,etc can be implemented as a pluggable filter. For example
> we h
Did you start the consumer after the producer? The default behavior of the
consumer is to "consume from the tail of the log", and hence if there is no
new messages coming in after the consumer started, it will get nothing. You
may set
auto.offset.reset="smallest"
and try again.
Guozhang
On Sun
What are your consumer config values?
On Mon, Jun 23, 2014 at 12:28 AM, Hemath Kumar
wrote:
> We have a 3 node cluster ( 3 kafka + 3 ZK nodes ) . Recently we came across
> a strange issue where we wanted to bring one of the node down from cluster
> ( 1 kafka + 1 zookeeper) for doing a maintenan
You may want to read this wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
Guozhang
On Mon, Jun 23, 2014 at 1:46 AM, ravi singh wrote:
> I want to use some vital broker stats into a different logging system.
> Can i read the broker specific data from
Ravi,
Our goal is to provide the best implementation of a set of useful
abstractions and features in Kafka. The motivation behind this philosophy
is performance and simplicity at the cost of flexibility. In most cases, we
can argue that the loss in flexibility is minimal since you can always get
t
Bringing one node down in a cluster will go smoothly only if your
replication factor is 1 and you enabled controlled shutdown on the brokers.
Also, bringing down 1 node our of a 3 node zookeeper cluster is risky,
since any subsequent leader election might not reach a quorum. Having said
that, a par
for (final KafkaStream stream : streams) {
ConsumerIterator consumerIte = stream.iterator();
while (consumerIte.hasNext())
System.out.println("Message from Single Topic :: "
+ new String(consumerIte.next().
message()));
}
Besides what Guo
What about ths pattern:
message = iter.peek(); //gets the message
process(message);
iter.next(); //consumes the message now
consumer.commit();
On Fri, Jun 20, 2014 at 11:53 AM, Kyle Banker wrote:
> Thanks for the advice, Guozhang.
>
> Jagbir: I'll report back on my progress. I intend to have
If there is a crash between
process(message);
and
iter.next(); //consumes the message now
you will still have duplicates upon restart.
Guozhang
On Mon, Jun 23, 2014 at 12:37 PM, Achanta Vamsi Subhash <
achanta.va...@flipkart.com> wrote:
> What about ths pattern:
>
> message = iter.peek(); /
Hello,
I am testing kafka 0.8.1.1 in preparation for an upgrade from
kafka-0.8.1-beta. I have a 4 node cluster with one broker per node, and a
topic with 8 partitions and 3 replicas. Each partition has about 6
million records.
I generated a partition reassignment json that basically causes all
Hello Neha, can you explain your statements:
>>Bringing one node down in a cluster will go smoothly only if your
replication factor is 1 and you enabled controlled shutdown on the brokers.
Can you elaborate your notion of "smooth"? I thought if you have
replication factor=3 in this case, you shoul
Slight nitpick, but a typo in using GetOffsetShell command can cause a
topic to be created which cannot be deleted (because deletion is still in
progress)
./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
kafka10:9092,kafka11:9092,kafka12:9092,kafka13:9092 --topic typo --time 1
./kafk
I attempted bin/kafka-reassign-partitions.sh for the topics in my cluster. It
was automatic reassignment of topics to new brokers.
Kafka duly initiated the partition reassignment. I believe kafka controller
will do the new leader election once the re-assigned partitions are in sync.
However, in
As of today, the latest Kafka docs show kafka.javaapi.producer.Producer in
their example of the producer API (
https://kafka.apache.org/documentation.html#producerapi).
Is there a reason why the latest producer client
(org.apache.kafka.clients.producer.Producer)
isn't mentioned? Is this client not
I think duplicate message is the right behavior for both patterns
iter.next(); process(message) ; CRASH; consumer.commit();
iter.peek();process(message) ; CRASH; iter.next(); CRASH; consumer.commit();
The only diff is fewer lines of code for the first pattern.
Jagbir
> Date: Mon, 23 Jun 2014 13:4
Hi Luke,
What are the exceptions/warnings you saw in the broker and controller logs?
Guozhang
On Mon, Jun 23, 2014 at 2:03 PM, Luke Forehand <
luke.foreh...@networkedinsights.com> wrote:
> Hello,
>
> I am testing kafka 0.8.1.1 in preparation for an upgrade from
> kafka-0.8.1-beta. I have a 4
However, in my case I realized that the new re-assigned partitions are not
getting in sync at all. The in-rate in the existing partition is far more
then the sync rate between earlier and newly reassigned partitions.
This is probably due to a bug. Partition reassignment is stable in 0.8.1.1.
I rec
Which version of Kafka are you using?
On Mon, Jun 23, 2014 at 2:42 PM, Luke Forehand <
luke.foreh...@networkedinsights.com> wrote:
> Slight nitpick, but a typo in using GetOffsetShell command can cause a
> topic to be created which cannot be deleted (because deletion is still in
> progress)
>
>
Hi Kyle,
We have not fully completed the test in production yet for the new
producer, currently some improvement jiras like KAFKA-1498 are still open.
Once we have it stabilized in production at LinkedIn we plan to update the
wiki in favor of the new producer.
Guozhang
On Mon, Jun 23, 2014 at 3
Luke,
Thanks for the findings, could you file a JIRA to keep track of this bug?
Guozhang
On Mon, Jun 23, 2014 at 2:42 PM, Luke Forehand <
luke.foreh...@networkedinsights.com> wrote:
> Slight nitpick, but a typo in using GetOffsetShell command can cause a
> topic to be created which cannot be d
I ended up coding against the new one,
org.apache.kafka.clients.producer.Producer, though it is not yet in
production here. It might be slightly more painful to select a partition
since there isn't a place to plug in a partitioner class, but overall it
was quite easy and had the key feature of an A
Hi, I’m new. We are considering using Kafka for our next generation product. We
have 300K devices that are connected to our cloud backend. We are currently
using a set of web services for our headless client devices to send state
information. We are moving to a message-based interface.
Does K
Hi All,
First, I¹m using 0.8.1. Is there anyway to cancel an ongoing
kafka-reassign-partitions job? I started a reassignment, and for some
reason it just takes forever. However, it won¹t let me start another
reassignment job while this one is running. So is there a way to just
cancel it?
Thank yo
Hi All,
Is there anyway to move a topic by force, even if it means loss of
messages? The problem I¹m looking at is the loss of multiple Kafka brokers
simultaneously in a Kafka cluster, which means that all replicas and
leader is lost. Under these circumstances, is there any way to forcibly
move th
no luck by adding props.put("auto.offset.reset", "smallest");
but running consumer after producer works.
But in my use case, it's not alwasy true for this.
Another question, The consumer should remember the offset. it's not
very easy to use.
On Mon, Jun 23, 2014 at 11:05 PM, Guozhang Wang wrote:
Thanks Guozhang/Neha for replies.
Here's my use case:
We use proprietary application logging in our apps. We are planning to use
Kafka brokers in production , but apart from the logs which are already
logged using log4j in kafka we want to log the broker stats using our
centralized application lo
Paul,
Reassignment is stable in 0.8.1.1, so you may be hitting a bug.
Nevertheless, could you please file a bug for canceling a partition
reassignment?
Thanks,
Neha
On Jun 23, 2014 5:16 PM, "Lung, Paul" wrote:
> Hi All,
>
> First, I¹m using 0.8.1. Is there anyway to cancel an ongoing
> kafka-re
If all replicas of a partition are offline, your partition is unavailable.
As such, there isn't a way for Kafka to copy data consistently from brokers
that are offline. If you are worried that an entire rack or data center
might go offline, you will need to ensure that your replicas are not on the
What kind of broker metrics are you trying to push to this centralized
logging framework?
Thanks,
Neha
On Jun 23, 2014 8:51 PM, "ravi singh" wrote:
> Thanks Guozhang/Neha for replies.
> Here's my use case:
>
> We use proprietary application logging in our apps. We are planning to use
> Kafka br
Hi there
One architecture pattern could be to have a number of app servers to receive
the messages from the devices and send them to the Kafka cluster. This would
let you handle authentication and authorisation from these devices, I don’t
know your exact scenario but I don’t think you’d want 30
Hi Ravi
You’ve probably seen this already but I thought I’d point it out just in case:
https://kafka.apache.org/documentation.html#monitoring. In our case we are
using https://github.com/pingles/kafka-riemann-reporter to send metrics to
Riemann but you could get the metrics through JMX to send
Hi Neha,
Bug created: https://issues.apache.org/jira/browse/KAFKA-1506
Thank you,
Paul Lung
On 6/23/14, 9:58 PM, "Neha Narkhede" wrote:
>Paul,
>
>Reassignment is stable in 0.8.1.1, so you may be hitting a bug.
>Nevertheless, could you please file a bug for canceling a partition
>reassignment?
Primarily we want to log below date(although this is not the exhaustive
list):
+ any error/exception during kafka start/stop
+ any error/exception while broker is running
+ broker state changes like leader re-election, broker goes down,
+ Current live brokers
+ new topic creation
+ when messages a
35 matches
Mail list logo
