Hello there,
We ran into a situation on our dev KAFKA cluster (3 nodes, v0.8.2) where
we ran out of disk space on one of the nodes. To free up disk space, we
reduced log.retention.hours to something more manageable (from 72hrs to
52hrs) as well as we moved the log directory to disk of 200GB. We
Hi Clay,
not really sure what you mean by socket, but if you want something
listening on a network port and forwards/produces all data to Kafka then
you might want to look at n2kafka: https://github.com/redBorder/n2kafka
Another alternative would be to use inetd, socat, or similar to pipe a
netwo
Thanks for the input. I've tried flume but the performance is not nearly as
good as kafkacat.
On Tue, May 19, 2015 at 2:40 AM, Magnus Edenhill wrote:
> Hi Clay,
>
> not really sure what you mean by socket, but if you want something
> listening on a network port and forwards/produces all data to
Thanks Magnus. I'll take a look at n2kafka. I have many data sources
sending data to kafka and I don't want to spawn lots of kafkacat processes.
On Tue, May 19, 2015 at 2:40 AM, Magnus Edenhill wrote:
> Hi Clay,
>
> not really sure what you mean by socket, but if you want something
> listening o
Try out bruce
https://github.com/ifwe/bruce it's a daemon listening socket producer, does
exactly what you are looking for I think.
~ Joestein
On May 19, 2015 7:05 AM, "clay teahouse" wrote:
> Thanks Magnus. I'll take a look at n2kafka. I have many data sources
> sending data to kafka and I don'
Hi,
I am testing Kafka-0.8.2.1 new producer API. For synchronous sending, I am
calling future.get() just after producer send.
I killed my broker and started Produce, noticed that it is throwing
ExecutionException but after that It is still trying to re-connect to
broker and this is keep on going
Hello there,
We ran into a situation on our dev KAFKA cluster (3 nodes, v0.8.2) where we ran
out of disk space on one of the nodes. To free up disk space, we reduced
log.retention.hours to something more manageable (from 72hrs to 52hrs) as well
as we moved the log directory to disk of 200GB. We
Good day Kafka-users.
To support our transition to Kafka as the central hub for data in our Big Data
Platform, we created a new producer named Klogger
(https://github.com/blackberry/klogger). It's a stripped down, high
performance producer that can take a TCP port or file as an input, and prod
Hi All
Has anyone tried this? We have two data centers A and B. We would like data
replicated between A and B. So I would like to have a kafka cluster set up
in A and B. When we need to replicate from A-->B I would like the app in A
publish a topic to the kafla cluster in data center A. The corres
Hi Bill,
I don't know if this is exactly the same case (last part "when they get the
topic tehy apply locally" is bit unclear), but we have setup with Kafka in
DC A and consumers both in DC A and DC B. Actually we also have producers
in A and B writing to Kafka in A, but we are trying to change th
I am using kafka 0.8.2.1 old producer. When one of the kafka node in the
remote cluster is down the producer is waiting about 15 minutes before it
disconnects and tries to connect to another node. (kafka takes < 1 min to
change leaders).
Producer config used:
request.required.acks=1
partitioner.cl
Hi Mayank,
The client should expose a configuration property to enable TCP keepalives
(SO_KEEPALIVE) on its broker sockets,
SO_KEEPALIVE provides speedier detection of connection loss on idle
connections.
(as a positive side effect it also helps keeping connections alive through
NAT/firewalls/LBs)
Thanks Magnus.
In this case the connections are not idle. There is active traffic between
the producer/client and the kafka node when the node goes down.
There are socket timeouts arguments for SimpleConsumer. But there are none
when creating the producer. If there a configuration/poroperty item to
I came across this google group conversation that suggests KafkaConsumer will
not be complete until the next release.
(https://groups.google.com/forum/#!msg/kafka-clients/4VLb-_wI22c/imYRlxogo-kJ)
```
org.apache.kafka.clients.consumer.KafkaConsumer consumer = new
org.apache.kafka.clients.cons
The links below shows the code is definitely in trunk.
Does anyone know when the source in trunk might be released?
Thanks!
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache
/kafka/clients/consumer/KafkaConsumer.java#L634
https://github.com/apache/kafka/blob/0.8.2/cli
The new consumer in trunk is functional when used similarly to the old
SimpleConsumer, but none of the functionality corresponding to the high
level consumer is there yet (broker-based coordination for consumer
groups). There's not a specific timeline for the next release (i.e. "when
it's ready").
Thanks!
On 5/19/15, 3:12 PM, "Ewen Cheslack-Postava" wrote:
>The new consumer in trunk is functional when used similarly to the old
>SimpleConsumer, but none of the functionality corresponding to the high
>level consumer is there yet (broker-based coordination for consumer
>groups). There's not
You can use MirrorMaker to mirror the topic from cluster A to cluster B.
This has the benefit of avoiding consumers in B connecting to A which could
have some latency
regards
On Tue, May 19, 2015 at 11:46 AM, Bill Hastings
wrote:
> Hi All
>
> Has anyone tried this? We have two data centers A an
Hi,I'mtrying to use low-level Consumer JavaAPI to manage offsets manually,
with the latest kafka_2.10-0.8.2.1To verify that theoffsets I commit/read form
Kafka are correct, I use thekafka.tools.ConsumerOffsetChecker tool.Here is an
exampleof the output for a topic/consumer group
./bin/kafka-ru
Sorry, the formatting seems to be all screwed up... I'll try to make it all
plain text:
Hi,
I'm trying to use low-level Consumer Java API to manage offsets manually, with
the latest kafka_2.10-0.8.2.1
To verify that the offsets I commit/read form Kafka are correct, I use the
kafka.tools.Consum
20 matches
Mail list logo
