Kafka

gards, Muqtafi Akhmad

Re: Kafka

atibility. They could completely change between > releases. (The command line tools themselves, however, are considered part > of the public interface, so we try to keep those compatible.) > > -Ewen > > On Wed, Sep 2, 2015 at 8:46 PM, Muqtafi Akhmad > wrote: > > > hello a

Message Id in Kafka Keyed Message

t randomize value of key? is there any rules/ restrictions about key's value? Thank you, Regards, -- Muqtafi Akhmad

Re: Message Id in Kafka Keyed Message

I just figured out that it is ok to assign random value as message key after reading kafka protocol guide ( https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol ) Thank you, On Mon, Sep 7, 2015 at 10:26 AM, Muqtafi Akhmad wrote: > hello all, > >

Problem in Get Kafka Consumer

k.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:111) Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Re: Problem in Get Kafka Consumer

39 AM, Muqtafi Akhmad wrote: > Hello all, > > I am trying to get consumer metadata from my Java program, using blocking > channel as described in this link > <https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka>. >

Kafka Out Of Memory Error

3 hosts, @ with memory 16 G partitions : ~200 replication factor : 3 rate in : 300 K/s rate out : 800 K/s Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Re: Kafka Out Of Memory Error

Thu, Nov 5, 2015 at 4:07 PM, Prabhjot Bharaj wrote: > Hi, > > Not sure. But, I had hit OOM when using too many partitions and many topics > with a lesser heap size assigned to the jvm > > Regards, > Prabhjot > > On Thu, Nov 5, 2015 at 10:40 AM, Muqtafi Akhmad > wrote:

Kafka Broker Max Connection

appreciated, Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Re: Kafka Broker Max Connection

is Int.MaxValue. > > Here's the doc: https://kafka.apache.org/documentation.html#brokerconfigs > > 2015-11-24 18:55 GMT+08:00 Muqtafi Akhmad : > > > Hello guys, currently I am trying to figure out some things about Kafka > > broker connections, there are two things I

Flush Messages in KafkaProducer Buffer

flush these messages to kafka to prevent event loss? will be calling the close() method do the trick? Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Does Simple Consumer Thread Safe?

Hello guys, is it safe to build a simple consumer pool that accessed by many threads? does SimpleConsumer thread safe? Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Re: 0.9.0.0[error]

ollowerLogReadResults(ReplicaManager.scala:849) > > at > > kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:467) > > at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:434) > > at kafka.server.KafkaApis.handle(KafkaApis.scala:69) > > at > > kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60) > > at java.lang.Thread.run(Thread.java:722) > > > -- Muqtafi Akhmad Software Engineer Traveloka

Re: message order problem

u, Nov 19, 2015 at 10:59 PM, Yonghui Zhao < > > zhaoyong...@gmail.com> > > > > > wrote: > > > > > > > > > > > Broker setting is: 8 partitions, 1 replica, kafka version 0.8.1 > > > > > > > > > > > > We send 2 message at almost same time. > > > > > > > > > > > > Msg1 first, Msg2 second. > > > > > > > > > > > > We have more than 1 producers in sync mode. > > > > > > > > > > > > We may send msg1 in one broker, after *producer.send return > > response* > > > > > > And send msg2 in the other broker. > > > > > > > > > > > > Both 2 msg has same partition key. > > > > > > > > > > > > In consumer side, we find the 2 msg is in same partition as > > expected, > > > > > but > > > > > > the order is inverted as msg2, msg1. > > > > > > > > > > > > Is it possible for kafka? > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > -- Guozhang > -- Muqtafi Akhmad Software Engineer Traveloka

Re: kafka producer and client use same groupid

urender" > wrote: > > > Hi all > > Is there anyway we can ensure in 0.8 that kafka remote producer and > > remote consumer work on the same groupId as my java consumer cannot > > consume messages from remote producer. Thanks > > > > > > > > > -- Muqtafi Akhmad Software Engineer Traveloka

Re: Flush Messages in KafkaProducer Buffer

en Shapira wrote: > In 0.9.0, close() has a timeout parameter that allows specifying how long > to wait for the in-flight messages to complete (definition of complete > depends on value of "acks" parameter). > > On Wed, Nov 25, 2015 at 3:58 AM, Muqtafi Akhmad > wrote

Re: flush() vs close()

being sent (either successfully or > > not).” and javadocs for close() say that “Close this producer. This > > method blocks until al in-flight requests complete.” Does ‘in-flight > > requests complete’ mean same as enqueued records are sent to Kafka? > > > > > > > > Kashif > > > > > > -- > Thanks, > Ewen > -- Muqtafi Akhmad Software Engineer Traveloka

Kafka Broker Network Error

nation about this log? The running Kafka version is 0.8.2 and the consumer used is SimpleConsumer. Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Possible Error Code for Kafka API Calls

s with their description . My question is, is there documentation of possible error code mapping for each API request? Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Re: Possible Error Code for Kafka API Calls

gt; > > > Not exactly - there is some documentation in the source code, but I > agree > > > that a wiki on this would be extremely useful. > > > > > > Can anyone create a wiki page? If so, I'm happy to get something > started. > > > It is r

Kafka Topic Leadership

partitions by altering topic will affect the topic leadership? Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Kafka broker decommission steps

with minimal effect to Kafka clients? Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Re: Kafka broker decommission steps

Any broker that is live in > >> the > >> > > cluster can, and will, get partitions assigned to it by the > >> controller. > >> > > > >> > > For what you want to do, you need you use the partition reassignment > >> &g

Re: steps path to kafka mastery

l there isn't really a course per say that teaches > you > > all things kafka. > > > > There are books but the one I read was more of a tutorial on certain > > aspects of kafka but it doesn't give you the insights of building a real > > production system. > > > > Any suggestions or tips? > > > -- Muqtafi Akhmad Software Engineer Traveloka

Tune Kafka offsets.load.buffer.size

dear Kafka users, Is there any tips about how to configure *offsets.load.buffer.size* configuration to speed up the offsets loading after leader change? The default value for this configuration is 5242880 Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Re: Tune Kafka offsets.load.buffer.size

Thank you Ben! On Thu, Apr 21, 2016 at 12:59 AM, Ben Stopford wrote: > If you have a relatively small number of consumers you might further > reduce offsets.topic.segment.bytes. The active segment is not compacted. > B > > On 18 Apr 2016, at 23:45, Muqtafi Akhmad wrote: >

Automatic Broker Id Generation

broker id should not be changed once broker has been successfully started, will a broker retain the same broker id after restarted? Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

Re: Automatic Broker Id Generation

.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaServer.scala#L632 > > > On Thu, May 19, 2016 at 3:28 PM, Muqtafi Akhmad > wrote: > > > dar Kafka users, > > I have two questions about automatic broker id generation when > > broker.id.generation.enab

Re: soft failure for kakfa 0.8.2.2

hello Fredo, Can you elaborate the 'soft' failure? On Sat, May 28, 2016 at 1:53 PM, Fredo Lee wrote: > we have a kafka cluster with 19 nodes. every week we suffer a soft failure > from this cluster. how to resolve this problem. > -- Muqtafi Akhmad Software Engineer Traveloka

Re: soft failure for kakfa 0.8.2.2

eduledThreadPoolExecutor.java:165) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11

Kafka Producer Buffer Exhausted

stion regarding how to solve this problem? Increasing buffer size looks appealing but I don't think it will be sustainable. Is there any way to reduce client-broker latency so that producer will use fewer buffer? (2) Is there any way to monitor producer buffer usage? Thanks, -- Muqtafi Akhma