We are using the High Level Consumer API to interact with Kafka.
However, on restart in the case of failures, we want to be able to manually
reset offsets in certain situations.
What is the recommended way to do this?
Should we use the Simple Consumer API just for this restart case?
Ideally, it
This is what I've found so far.
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
The high-level consumer just worked for me by setting offsets.storage =
kafka.
Scroll down to the offsets.* config params.
http://kafka.apache.org/documentation.ht
d the client offset api but I
wanted to see if anyone had an implementation.
-Suren
On Fri, Dec 12, 2014 at 12:41 PM, Steve Morin wrote:
>
> Suren,
> Like out of the box storage or roll your own?
> -Steve
>
> On Fri, Dec 12, 2014 at 6:33 AM, Surendranauth Hiraman <
>
My team is using Kafka 0.8.1 and we may not be able to upgrade to 0.8.2 to
take advantage of the broker-side commit of client offsets.
Is anyone aware of a Java/Scala library for client offset storage outside
of ZK?
-- Suren
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440
1, 2014 at 3:39 AM, Surendranauth Hiraman <
> suren.hira...@velos.io> wrote:
>
> > Hi,
> >
> > Is there a way to add a partition to an existing topic in 0.7.1? I could
> > only find such scripts in 0.8.1.
> >
> > Thank you.
> >
> >
&
Hi,
Is there a way to add a partition to an existing topic in 0.7.1? I could
only find such scripts in 0.8.1.
Thank you.
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@
Here are use cases for contrast, if you haven't seen them.
https://kafka.apache.org/documentation.html#uses
Also, the most significant operational difference versus traditional
message queue brokers is that responsibility for management of where you
are in the queue lies on the client side. This
I think TTL/expiration to another queue might work, depending on your time
window tolerance for "recency".
JMS queues have this (to a dead letter queue but you could use it
differently) but I'm not sure if Kafka does.
If you need a fast (though not as great as Kafka) tradition MQ, there are a
few
013 at 11:11 AM, Surendranauth Hiraman <
suren.hira...@sociocast.com> wrote:
> Depending on your exact requirements, you may consider priority queues as
> well. I'm not sure if Kafka has this (without rolling your own) but it is
> in JMS implementations.
>
> I guess you ar
Depending on your exact requirements, you may consider priority queues as
well. I'm not sure if Kafka has this (without rolling your own) but it is
in JMS implementations.
I guess you aren't looking for strict LIFO order, since messages are
constantly being put on the queue, so "LIFO" is only with
S Ahmed,
This combination of Kafka and Storm to process streaming data is becoming
pretty common. Definitely worth looking at.
The throughput will vary depending on your workload (cpu usage, etc.) and
if you're talking to a backend, of course. But it scales very well.
-Suren
On Mon, Dec 2, 20
I'm a fan of kafka as well. We've been using 0.7.2 for about a year.
I recommend it strongly.
But I will point one thing. Not an issue with Kafka itself but when the
client side has failed, choosing what offset to reset to is not an exact
science. You will have to decide how much data loss or dat
the ports and still no luck. Does it work with JConsole
> and/or do I need anything in my class path?
> >
> >
> > On Aug 29, 2013, at 3:44 PM, Surendranauth Hiraman <
> suren.hira...@sociocast.com> wrote:
> >
> >> Maybe kafka is on a different port?
>
Maybe kafka is on a different port?
I do 'ps -ef | grep kafka' and look at the command line to see the jmx port
usually.
-Suren
On Thu, Aug 29, 2013 at 6:41 PM, Mark wrote:
> Can you view Kafka metrics via JConsole? I've tried connecting to port
> with no such luck?
--
14 matches
Mail list logo
