Hi All,
Greetings..!!! This is my first email to Kafka Community.
I have just started exploring Kafka on CDH5.5 cluster which ships with
Kafka 0.8.2.1.
I am able to run sample programs for producer as well as consumer (both
high level and low level).
Now I am trying to load messages from Kafka
OK I understand the explanation. Thanks you for sharing your knowledge !
Regards,
Robin
2016-02-18 18:56 GMT+01:00 Jason Gustafson :
> Woops. Looks like Alex got there first. Glad you were able to figure it
> out.
>
> -Jason
>
> On Thu, Feb 18, 2016 at 9:55 AM, Jason Gustafson
> wrote:
>
> > H
Hi All,
the same happens for Kafka 0.9.0.1. The controller is unable to setup a
connection on the local machine. I tried to set the following configurations in
config/server.properties
host.name=localhost
advertised.host.name=localhost
controller.socket.timeout.ms=245000
without any luck. The
Using kafka 0.9.0.0, if there are multiple consumers in a group and one
consumer pauses the topic+partition it's consuming, does that allow/cause
other consumers in the group to pick it up?
kafka.apache.org:
The kafka version I use is kafka_2.10-0.8.2.0,I create topic with 3
replication in kafka cluster.
When the kafka cluster runs a long time,some topic partitions become dead,so
can't work.Restart kafka
can't solve recover topic partition.The solution I use is delete all kafka d
Hi Marcus,
One thing that changed in 0.9.0.0 is that we now use `NetworkClient` for
inter-broker communication. `NetworkClient` is also used by the new Java
producer and consumer. Does the Java producer in 0.8.2 work fine for you?
Ismael
On Mon, Feb 22, 2016 at 3:06 PM, Marcus Gründler wrote:
Our use case for kafka-streams only needs to use the in memory state store but
the current code on trunk still requires a valid ’state.dir’ that the client
can write too. A lock file per partition and an offset file is created but the
offset file always contains zero values for offsets since al
Hi,
Does recently released Kafka 0.9.0.1 have final release of security features,
initiated in 0.9.0.0 or it is still should be considered beta quality?
I’m asking because Cloudera recently send an announcement of their parcel
release that promotes Kafka security features and does not mention th
Hi John,
I'm glad the info was helpful.
It's hard to diagnose this issue without monitoring. I suggest setting up
graphite to graph JMX metrics. There's a good (not designed for production)
script here (as part of a Vagrant VM):
https://github.com/gwenshap/ops_training_vm/blob/master/bootstrap.sh
Nope. Pausing a partition just stops the consumer from sending any more
fetches for it. It will not trigger a partition reassignment. One thing to
be wary of, however, is that the partition will automatically be unpaused
after the next normal rebalance.
-Jason
On Mon, Feb 22, 2016 at 7:19 AM, cra
so if I had a bunch of consumers coming online while some consumers had
partitions paused, those consumers that were paused would become unpaused?
On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson wrote:
> Nope. Pausing a partition just stops the consumer from sending any more
> fetches for it. I
That is correct. The typical way to handle this (if you need to) is to
repause the partitions in the ConsumerRebalanceListener
onPartitionsAssigned() hook.
-Jason
On Mon, Feb 22, 2016 at 11:03 AM, craig w wrote:
> so if I had a bunch of consumers coming online while some consumers had
> partiti
I think I'll need to do that.
My current setup is where I have 200 partitions for a topic with 200
consumers, all part of the same group. Each consumer does a "poll". Upon
receiving messages it will pause the topic/paritions it received, while the
messages are processed, as each message is process
Hi Ismael,
Yes, producers (and consumers) work just fine on 0.8.2.
We see the exceptions even without any producer/consumers. The problem can be
reproduced easily - I just start the following scripts an wait for 30 seconds:
bin/kafka-zookeeper-start.sh path/to/zkconf
bin/kafka-server-start.sh
Hi Tom,
That is right that currently for in-memory state store the directory is not
actually used, i.e. the directory will be created but no data will be
written. But the "state.dir" is not a required config in the sense that if
user do not specify a value, it will default to "/tmp/kafka-streams".
On 2/21/16 10:23 AM, Jay Kreps wrote:
Hey Andrew,
Yeah I think the current state is that we did several design and prototypes
(both the transaction work and the idempotence design and the conditional
write KIP), but none of these offshoots is really fully rationalized with
the other ones. Slow p
Jason:
Thanks for your reply.
I did turn on logging for kafka as well when I ran the test. I do not see any
partition assignment for the topic happening.
If you don’t see anything obviously wrong in what I am doing, I can file a
ticket with Confluent and attach the logs to it.
Regards
Venkat
Hey Venkat,
The logs will help, but I'll probably need to see some code to understand
what is going on (consumer usage can be a little subtle sometimes). You're
welcome to open a ticket if you like or post here.
Thanks,
Jason
On Mon, Feb 22, 2016 at 5:09 PM, Venkatesan Subramanian <
vsubraman...
Let us say there are two Producers P1 and P2.
P1 makes attempt to write log L1 at time t1 to a partition PT however it
gets written at t2 due to network latency.
P2 makes attempt to write log L2 at time t3 to a partition PT however it
gets written at t4 due to network latency.
Let us say t1t4 so w
Hi All,
I saw an issue today wherein the producers (new producers) started to fail
with org.apache.kafka.common.errors.TimeoutException: Failed to update
metadata after 6 ms.
This issue happened when we took down one of the 6 brokers (running version
0.8.2.1) for planned maintenance (graceful
Hi,
How do we run Performance Tests on kafka 2.11-0.9.0.1?
--
Cheers!
Harihara Subramaniam
what kind of performance tests are you looking for?
I am currently using kafka-producer-perf-test.sh to benchmark my brokers.
There is also a consumer version too
On Mon, Feb 22, 2016 at 8:05 PM, Harihara Subramaniam <
subramaniam.harih...@gmail.com> wrote:
> Hi,
>
> How do we run Performance Tes
Something i do not understand about this perf-test tool.
1. The legend shows 5 columns but the data shows 6 columns.
I am assuming the 0 column is the one that is throwing everything off?
2. does nMsg.sec = number of message consumed per second?
[bin]$ sudo ./kafka-consumer-perf-test.sh --group
Hi,
There is a producer which uses librdkafka to send messages over a Kafka
infrastructure. There is an error which occurred from sync API call. It
gives error message "Broker: Request timed out".
To give some context of the scenario, there are two brokers - broker 2 on
local host being the pr
24 matches
Mail list logo