Can someone explain how message queueing in partition(s) work?
Specifically:
Is there any buffering before message is committed to log (written to disk) ?
Does the term "partition" refer to sequence of messages written to disk, or
are the messages stored in memory queue and written to disk only aft
Can someone clarify it:
- "At most once": messages may be lost but are never redelivered
Does it mean: "commit, when message is received by leader partition,
retries disabled for producer no matter of ISR status"?
- "At least once": commit only when message is ackd by all partitions in
replication
Can someone clarify difference between partition replication and partition
replication? Or, are they referring to the same thing?
I am looking for best method to keep consumption of messages in the same
order as client produced them, one thing i am looking at is using of
message key to select partition:
If a valid partition number is specified that partition will be used when
sending the record. If no partition is specified b
I have to perform frequent re- deployments, and i run into the offset
problem:
Unable to Receive Message:kafka server: The requested offset is outside the
range of offsets maintained by the server for the given topic/partition
I tried to re-create topics and reset topic metadata in zookeeper from
s
I am using kafka 0.8.1 /zookeeper combination...Can someone advise what the
proper procedure would be to fix the issue with producer / consumers
offsets going out of range when consumer has to be re- started? Do I need
to delete meta- data pertaining to topics? What would be the proper way of
doing
t; Do you use it?
> If so, when you delete/re-create topics you should clean data in Zookeeper
> too. For example:
>
> run
> user@host:~$ KAFKA_PATH/bin/zookeeper-shell.sh 127.0.0.1:2181
> and type theese commands:
> rmr /kafka
> create /kafka ''
>
> Hope
i have 0.8.1 producer/consumer... I start producer first but when i start
consumer i see the following :
Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|main.go:34|2015/04/15 21:26:29|Starting
ConsumerService
Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
INFO|10cf2
like they don't line up.
>
> On Wed, Apr 15, 2015 at 2:10 PM, Victor L wrote:
>
> > I see the following error in consumer;
> > Unable to Receive Message:kafka server: The requested offset is outside
> the
> > range of offsets maintained by the server for the given
essages before triggering a flush to the
> broker. The default was 1 millisecond (this value does not even exist in
> the most recent release). Setting it to higher values trades off latency
> for efficiency, whatever your use case dictates.
>
> Evan
>
> On Wed, Apr 15, 2015 a
I see the following error in consumer;
Unable to Receive Message:kafka server: The requested offset is outside the
range of offsets maintained by the server for the given topic/partition
On Wed, Apr 15, 2015 at 2:09 PM, Victor L wrote:
> I set it to 10ms and it did fix cpu consumption prob
I inherited this code, it just set to 1000, what would be reasonable time:
1000 * time.Millisecond?
On Wed, Apr 15, 2015 at 1:41 PM, Evan Huus wrote:
> On Wed, Apr 15, 2015 at 1:33 PM, Victor L wrote:
>
> > Hi Evan,
> > It's hardcoded:
> > MaxBufferTime = 1
very old version of Sarama. You
> might have better luck using a more recent stable version.
> - Are you setting `MaxBufferTime` in the configuration to 0 or a very small
> value? If so the loop will spin on that timer. Try making this value
> larger.
>
> Evan
>
> On Wed
I am using sarama "golang" kafka 1.8.1 client (
https://github.com/Shopify/sarama) to send messages to message queue once
in 3secs and this task drives cpu consumption to 130% on the quad-cpu
blade; The number stays this high regardless of number of
partitions/consumers According to results of
> On Wed, Mar 25, 2015 at 9:33 AM, tao xiao wrote:
>
> > You can use kafka-console-consumer consuming the topic from the beginning
> >
> > *kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
> > --from-beginning*
> >
> >
> > On Thu, Mar 26, 2
Can someone let me know how to dump contents of topics?
I have producers sending messages to 3 brokers but about half of them don't
seem to be consumed. I suppose they are getting stuck in queues but how can
i figure out where?
Thks,
I have the following error at kafka broker, even if I run single instance:
Unable to get Messaging Service:kafka server: In the middle of a leadership
election, there is currently no leader for this partition and hence it is
unavailable for writes
Any advise on how i can change my configuration to
ime to time, but the zkClient will
> retry and usually it will succeed. Can you verify if you were finally be
> able to connect or not?
>
> Jiangjie (Becket) Qin
>
> On 2/27/15, 12:53 AM, "Victor L" wrote:
>
> >I seem to unable to connect single instance of
I seem to unable to connect single instance of Kafka server to single
instance of local zookeeper server on local host (Ubuntu 14.04LTS/ open jdk
1.7.0_75):
[2015-02-27 08:10:11,467] INFO Initiating client connection,
connectString=localhost:2181 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkC
19 matches
Mail list logo
