The high level consumer stores its state in ZooKeeper. Theoretically, you
should be able to go into ZooKeeper, find the consumer-group, topic and
partition, and increment the offset past the "corrupt" point.
On Tue, Aug 4, 2015 at 10:23 PM, Henry Cai
wrote:
> Hi,
>
> We are using the Kafka high-
Hi,
We are using the Kafka high-level consumer 8.1.1, somehow we got a
corrupted message in the topic. We are not sure the root cause of this,
but the problem we are having now is the HL consumer is stuck in that
position:
kafka.message.InvalidMessageException: Message is corrupt (stored crc =
5
you are right.
after I run kafka in my localhost directly, it works just fine
after further google, i found that need to set two parameters below if the
kafka is running on some other machines
#advertised.host.name=
#advertised.port=
more precisely, if the kafka is running within a docker conta
Hi Everyone,
We're trying the deploy the Kafka behind the network balancer and we have
created the port map for each Kafka brokers under that network balancer--we
only have one public IP and the Kafka clients are in other system and thus
cannot access the brokers via internal IP directly.
So for e
Hi guys,
I was reading a paper today in which the latency of kafka and rabbitmq is
compared:
http://downloads.hindawi.com/journals/js/2015/468047.pdf
To my surprise, kafka has shown some large variations of latency as the
number of records per second increases.
So I am curious about why is that.
Thats correct. Thanks for catching that.
On Tue, Aug 4, 2015 at 3:27 PM, Andrew Otto wrote:
> Thanks!
>
> > In fact if you use a "Controlled Shutdown" migrating the replicas and
> > leaders should happen for you as well.
>
> Just to clarify, controlled shutdown will only move the leaders to othe
The way Kafka is currently implemented is that Kafka is not aware of the
content of messages, so there is no Selector logic available.
The way to go is to implement the Selector in your client - i.e. your
consume() loop will get all messages but will throw away those that don't
fit your pattern.
Hi Robert
Here is the kafka benchmark for your reference.
if you want to use Flink, Storm, Samza or Spark, the performance will be
going down.
821,557 records/sec(78.3 MB/sec)
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
Best
I am on Kafka 0.8.2.1 (Java 8) and have happened to run into this same
issue where the KafkaServer (broker) goes into a indefinite while loop
writing out this message:
[2015-08-04 15:45:12,350] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1438661432074","host":"foo-bar",
If this
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+I
nterface) is what you need then watch for
https://reviews.apache.org/r/34492/ to get committed to trunk.
Thanks
Parth
On 8/4/15, 1:57 PM, "Alvaro Gareppe" wrote:
>Can someone point me to documentation about ac
The is way to implement a "selector" logic in kafka (similar to JMS
selectors)
So, allow to consume a message if only the message contains certain header
or content ?
I'm evaluating to migrate from ActiveMQ to kafka and I'm using the selector
logic widely in the application
--
Ing. Alvaro Garep
Can someone point me to documentation about access control in kafka. There
is something implemented in the current or plan for future versions ?
I need something that allows me to define what users are allowed to connect
to certain topic, and of course user management.
Thank you guys in advance!
Thanks!
> In fact if you use a "Controlled Shutdown" migrating the replicas and
> leaders should happen for you as well.
Just to clarify, controlled shutdown will only move the leaders to other
replicas, right? It won’t actually migrate any replicas elsewhere.
-Ao
> On Aug 4, 2015, at 13:00,
To add some internals, the high level consumer actually does read entire
batches from Kafka. It just exposes them to the user in an event loop,
because its a very natural API. Users can then batch events the way they
prefer.
So if you are worried about batches being more efficient than single
even
Thanks a lot Shaminder for clarification and thanks Raja for pointing me to
the example.
best,
/shahab
On Tue, Aug 4, 2015 at 6:06 PM, Rajasekar Elango
wrote:
> Here is an example on what sharninder suggested
>
> http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/
Sorry for the very late reply ...
The performance issue was not caused by network latency. I had a job like
this:
FlinkKafkaConsumer --> someSimpleOperation --> FlinkKafkaProducer.
I thought that our FlinkKafkaConsumer is slow, but actually our
FlinkKafkaProducer was using the old producer API of
I couldn't find a jira for this, so I added KAFKA-2403.
-Jason
On Tue, Aug 4, 2015 at 9:36 AM, Jay Kreps wrote:
> Hey James,
>
> You are right the intended use of that was to have a way to capture some
> very small metadata about your state at the time of offset commit in an
> atomic way.
>
> T
The broker will actually unregister itself from zookeeper. The brokers id
path uses ephemeral nodes so they are automatically destroyed on shutdown.
In fact if you use a "Controlled Shutdown" migrating the replicas and
leaders should happen for you as well. Though, manual reassignment may be
prefer
The /brokers/ids nodes are ephemeral nodes that only exists while the
brokers maintain a session to zookeeper. There is more information on
Kafka's Zookeeper usage here:
- http://kafka.apache.org/documentation.html
- look for "Broker Node Registry"
-
https://cwiki.apache.org/confluence/
Hey James,
You are right the intended use of that was to have a way to capture some
very small metadata about your state at the time of offset commit in an
atomic way.
That field isn't exposed but we do need to add it to the new consumer api
(I think just no one has done it yet.
-Jay
On Mon, Au
Hi Sreeni,
Using the SimpleConsumer you can send a TopicMetadataRequest for a topic
and the TopicMetadataResponse will contain TopicMetadata for each topic
requested (or all) which contains PartitionMetadata for all all partitions.
The PartitionMetadata contains the leader, replicas, and isr.
Is
Hey Simon,
The new consumer has the ability to forego group management and assign
partitions directly. Once assigned, you can seek to any offset you want.
-Jason
On Tue, Aug 4, 2015 at 5:08 AM, Simon Cooper <
simon.coo...@featurespace.co.uk> wrote:
> Reading on the consumer docs, there's no men
Here is an example on what sharninder suggested
http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/
Thanks,
Raja.
On Tue, Aug 4, 2015 at 12:01 PM, Sharninder wrote:
> You can't. Kafka is essentially a queue, so you always read messages one
> by one. What you can d
You can't. Kafka is essentially a queue, so you always read messages one by
one. What you can do is disable auto offset commit, read 100 messages, process
them and then manually commit offset.
--
Sharninder
> On 04-Aug-2015, at 9:07 pm, shahab wrote:
>
> Hi,
>
> While we the producer can pu
Hi,
While we the producer can put data as batch in kafka server, I couldn't
find any API (or any document) saying how we can fetch data as batch from
Kafka ?
Even when data is placed as batch in kafka server, still using High Level
consumer I can only read one by one, and I can not specify. for e
I'd like to save a snapshot of a processing node's state in a compacted kafka
topic. A large number of nodes would save their snapshots in the same
partition.
What is an efficient way for a (restarted) node to find the offset of its
latest snapshot? Using just Kafka (no database, local file, e
Reading on the consumer docs, there's no mention of a relatively simple
consumer that doesn't need groups, coordinators, commits, anything like that -
just read and poll from specified offsets of specific topic partitions - but
automatically deals with leadership changes and connection losses (s
Thanks Jun for heads up!
On Mon, Aug 3, 2015 at 7:17 PM, Jun Rao wrote:
> Hi, Stevo,
>
> Yes, we are still iterating on the new consumer a bit and are waiting for
> some of the security jiras to be committed. So now, we are shooting for
> releasing 0.8.3 in Oct (just updated
> https://cwiki.apac
Some suggestions:
Check the existence of the topic.
Check the firewall of the broker... Try telnet or something to
make sure it's available.
Try run the producer on the broker machine.
Since you get this error, this code is functioning. I think it's some
configuration an
Hi Jason and Kafka Dev Team,
First of all thanks for responding and I think you got expected behavior
correctly.
The use-case is offset range consumption. We store each minute highest
offset for each topic per partition. So if we need to reload or re-consume
data from yesterday per say 8AM
30 matches
Mail list logo
