Hmm, I think Svante is correct, writes on disk-full would probably cause
the underlying file system to get in a bad state, and in that sense the
broker needs to be brought down for maintenance.
Guozhang
On Thu, Mar 26, 2015 at 3:49 PM, svante karlsson wrote:
> >4. As for recovering broker from
Thanks Guozhang,
I am currently working on a project at my current company where I process
data from Kafka. The data is all tiny Kafka messages (25 -35 bytes) and so
far we were bottle necked on our processing speed. Recently we have made
significant improvements and our processing speed has gone
> 1) Does Offset Commit/Fetch API works with Simple Consumer ?
Yes - in 0.8.2.1. There is an example given at
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
> 2) With MM, can you selectively MM offset topic.
>
> Use Case:
> Let's suppose you
>4. As for recovering broker from disk full, if replication is enabled one
>can just bring it down (the leader of the partition will then migrate to
>other brokers), clear the disk space, and bring it up again; if replication
>is not enabled then you can first move the partitions away from this bro
Hi Bhavesh,
1. Server disk-full is treated the same as other error, that an error code
will be returned (in this case I think it is "Unknown" error though, as
disk IO exception is not specifically captured).
2. Upon receiving the error from the brokers, producer will retry based on
its configs. H
Rajiv,
Those are good points. As for implementation we have developed a class in
producer that can be probably re-used for the consumer as well.
org.apache.kafka.clients.producer.internals.BufferPool
Please feel free to add more comments on KAFKA-2045.
Guozhang
On Tue, Mar 24, 2015 at 12:21 P
Hello Josh,
We do not have a ticket open for idempotent producer as it is still in the
discussion process, but here is the wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer
As for transactional messaging, we have a prototype implementation at
LinkedIn which is not yet i
No, we don't normally see conflicts. We'll see endless attempts to
rebalance.
-Mike
On Thu, Mar 26, 2015 at 5:15 PM, Mayuresh Gharat wrote:
> Did you see something like this in any of the consumer logs :
>
> "Conflict in ….. data : ……. stored data :……” ?
>
> Thanks,
>
> Mayuresh
>
> On Thu, Ma
Did you see something like this in any of the consumer logs :
"Conflict in ….. data : ……. stored data :……” ?
Thanks,
Mayuresh
On Thu, Mar 26, 2015 at 1:50 PM, Mike Axiak wrote:
> Hi guys,
>
> At HubSpot we think the issue is related to slow consumers. During a
> rebalance, one of the first t
Hi guys,
At HubSpot we think the issue is related to slow consumers. During a
rebalance, one of the first things the consumer does is signal a shutdown
to the fetcher [1] [2], in order to relinquish ownership of the partitions.
This waits for the shutdown of all shutdown fetcher threads, which ca
Hi,
We are seeing really weird GC behaviour on our Kafka cluster. After a clean
stop/start of the whole cluster, the servers go into a loop of increasingly
slow GC cycles for an hour and then seem to recover to a normal state
without any external changes to the servers or traffic. The load to the
Both consumer-1 and consumer-2 are properties of source clusters mirror
maker transfers data from. Mirror maker is designed to be able to consume
data from N sources (N >= 1) and transfer data to one destination cluster.
You are free to supply as many consumer properties as you want to instruct
mir
Victor,
Its under kaka.tools.DumpLogSegments you can use kafka-run-class to
execute it.
--
Harsha
On March 26, 2015 at 5:29:32 AM, Victor L (vlyamt...@gmail.com) wrote:
Where's this tool (DumpLogSegments) in Kafka distro? Is it Java class in
kafka jar, or is it third party binary?
thanks Mayuresh and Jiangjie for your response.
I have actually not understood Mirror maker clearly and hence bit skeptical
if i will be able to execute it effectively.
Online i hv seen the following command to be execute, but not understood
what is consumer-1 & -2.properties here? do i need to
Where's this tool (DumpLogSegments) in Kafka distro? Is it Java class in
kafka jar, or is it third party binary?
Thank you,
On Wed, Mar 25, 2015 at 1:11 PM, Mayuresh Gharat wrote:
> DumpLogSegments will give you output something like this :
>
> offset: 780613873770 isvalid: true payloadsize: 805
15 matches
Mail list logo
