on
> wrote:
>
> > Sounds like a goer then :) Those strings in the protobuf always get ya,
> > can't use clever encodings for them like you can with numbers.
> >
> > On Wed, 16 Mar 2022 at 11:29, Dan Hill wrote:
> >
> >> We're using protos but
wrote:
> Hi Dan,
>
> Okay, so if you're looking for low latency, I'm guessing that you're using
> a very low linger.ms in the producers? Also, what format are the records?
> If they're already in a binary format like Protobuf or Avro, unless they're
> composed
ant than storage
size.
On Mon, Mar 14, 2022 at 3:27 PM Liam Clarke-Hutchinson
wrote:
> Hi Dan,
>
> Decompression generally only happens in the broker if the topic has a
> particular compression algorithm set, and the producer is using a different
> one - then the broker will deco
Hi. I looked around for advice about Kafka compression. I've seen mixed
and conflicting advice.
Is there any sorta "if X, do Y" type of documentation around Kafka
compression?
Any advice? Any good posts to read that talk about this trade off?
*Detailed comments*
I tried looking for producer v
The canonical form for Avro schemas is to use a single "name" key whose
value is the concatenation of the namespace, if any, with the record
name:
https://avro.apache.org/docs/current/spec.html#Transforming+into+Parsing+Canonical+Form
There is a common, non-canonical alternative out in the wil
I'm trying to understand the interactions between stateful processing in
Kafka Streams and topic retention size restrictions. I'm fairly naive
about both Kafka and Kafka Streams so I'm sure I have something wrong
here, but my understanding is that, under stateful processing, local
state in the
e
to get to a working configuration.
On Thu, Mar 19, 2020 at 4:55 PM Dan Hill wrote:
> I'll prefix this with I'm new to AWS. I'm used to Google's stack. I also
> noticed the helm instructions
> <https://github.com/helm/charts/tree/master/incubator/kafka#connecting-t
ou've configured your listeners & advertised
> listeners correctly. This should help:
> https://rmoff.net/2018/08/02/kafka-listeners-explained/
>
>
> --
>
> Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff
>
>
> On Thu, 19 Mar 2020 at 01
Problem: I'm hitting an error: "no such host" for "
kafka-0.cluster.local:19092".
Has anyone done this before? Any help would be appreciated. Thanks! - Dan
My long-term goal is to get an AWS Lambda to send events to a Kafka running
in AWS EKS.
I used the foll
Following up on this. It turned out to be 100% user error on my part. I was
still sending the v0 OffsetFetch request after committing v1+.
On Tue, Sep 17, 2019 at 9:14 PM Dan Swain wrote:
> Hi!
>
> I'm a maintainer of an open source Kafka client and working on adding
> support
t's hard to debug this without code. I can actually share the
code since it's an open source client, but my hope is that someone
recognizes this problem and has a quick answer.
My test cluster is Kafka 0.11.
Thanks!
- Dan
Usually for such a use case you'd have a physical load balancer box (F5,
etc.) in front of Kafka that would handle the SSL termination, but it
should be possible with NGINX as well:
https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-tcp/
On Fri, 24 Aug 2018 at 18:35, Jack
Ensure ids/nodes are correct in zoo.cfg and zookeeper is running on each. Also
any changes to ports being open? If it’s aws, beck security groups. Node 1
cannot talk to the other two nodes.
> On Aug 16, 2018, at 6:02 PM, Raghav wrote:
>
> Hi
>
> Our 3 node Zookeeper ensemble got powered d
Are you looking more for an Actor or orchestration layer and visibility? I
don’t know of one per se, it would also be interested.
Dan
From: Jonathan Roy
Sent: Thursday, July 19, 2018 3:22 AM
To: users@kafka.apache.org
Subject: Monitor the state of an event
In Azure we recommend using managed disks for Kafka. HD Insight Kafka uses
them. I generally see SSD for Kafka, but I guess part of that could depend on
if you write larger writes from fewer publishers or small writes from many
publishers. What does your workload look like?
Kind Regards,
-Dan
h, which I think is one.
Kind Regards,
-Dan
-Original Message-
From: chinchu chinchu
Sent: Tuesday, July 10, 2018 9:06 AM
To: users@kafka.apache.org
Subject: Create a Topic if Not Exists
Hi All,
I am looking for a way to script(bash) kafka topic creation . We have around
30 to 40 t
2 connection strings in the
*listeners* config?
Thank you!
Dan
Does that make sense?
Dan
On Mon, Apr 2, 2018 at 8:27 PM Matthias J. Sax
wrote:
> Atm, Kafka does not ship any ready-to-use transformers. All operators
> that are ready-to-use are provided via the DSL only and their
> implementation itself is not public API.
>
> The question arises
s the I want. If we find
this useful at my company I'll consider submitting it as a PR.
Is there any current precedent around transformers like this in
KafkaStreams? Transformers you insert for interrogation only?
Dan
On Mon, Apr 2, 2018 at 2:10 PM Matthias J. Sax
wrote:
> Dan,
>
&
/partition is lagging to debug my problem.
Does something like this exist? Is this a reasonable feature request?
Thanks!
Dan
Hi,
I recently switched my Kafka Streams 1.0.0 app to use exactly_once
semantics and since them my cluster has been stuck in rebalancing. Is
there an explanation as to what is going on, or how I can resolve it?
I saw a similar issue discussed on the mailing list, but I don't know if a
ticket was
You can try generating a random consumer group ID to ensure that no
existing offset is used.
On 20 January 2018 at 01:11, Matthias J. Sax wrote:
> That is not supported out-of-box.
>
> Configuration "auto.offset.reset" only triggers, if there are not
> committed offsets and there is KS config to
It is also used for rewinding consumer offsets.
On 19 January 2018 at 06:25, Matthias J. Sax wrote:
> The timestamp has many different purposes. As mentioned already, it used
> to expired data via retention time. It's also used for stream processing
> via Streams API. All processing is based on
You can add more instances of your application, to allow processing the
incoming data in parallel.
On 6 November 2017 at 20:11, Ranjit Kumar wrote:
> Hi,
>
> I am using kafka streaming and state store in my application using java but
> my application logic is taking more time (around 5 ms) to pi
ed Yu wrote:
> Can you update librdkafka and try again ?
> Thanks
> Original message ----From: Dan Markhasin
> Date: 10/25/17 11:26 PM (GMT-08:00) To: users@kafka.apache.org Subject:
> Re: Failure to reset consumer offsets for specific topics
> For beats (the topic
ed if it's pretty old and doesn't set timestamp on the messages it
produces.
On 26 October 2017 at 08:17, Elyahou Ittah wrote:
> Which driver is used yo produce these messages ?
>
> On Thu, Oct 26, 2017 at 8:14 AM, Dan Markhasin
> wrote:
>
> > Furthermore, when look
this
is where the broker takes the timestamps from, it explains why they are all
empty for data1_log.
On 26 October 2017 at 08:07, Dan Markhasin wrote:
> After a bit more checking it seems that Kafka isn't writing timestamps at
> all in the .timeindex file for the topics where offset re
ed yo produce these messages ?
>
> On Oct 26, 2017 07:11, "Dan Markhasin" wrote:
>
> > No, that flag doesn't affect which offsets are returned, only executes
> the
> > action (and resets the consumer to latest offset when used, regardless of
> > datetime val
-hans
>
> > On Oct 25, 2017, at 1:24 PM, Ted Yu wrote:
> >
> > I wonder if you have hit KAFKA-5600.
> >
> > Is it possible that you try out 0.11.0.1 ?
> >
> > Thanks
> >
> >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin
> wrote:
ably check the broker log where data1_log is hosted - see if there is
> some clue.
>
> Thanks
>
> On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin
> wrote:
>
> > I'm trying to use the kafka-consumer-groups.sh tool in order to rewind a
> > consumer group's off
ct
offsets fixed in the time for the requested datetime.
What could be the issue here?
Thanks,
Dan
If you run kafka-topics.sh --describe --topic __consumer_offsets, does it
show that all replicas are in sync?
On 23 August 2017 at 23:11, Murad Mamedov wrote:
> Hi David,
>
> Thanks for reply. However, I don't have problem with number of replicas. I
> have 3 brokers. And topics configured accord
Is your storm consumer set to auto.offset.reset="earliest"?
On 22 August 2017 at 10:05, Elyahou Ittah wrote:
> I checked the __consumer_offsets topic and here is an extraction from this
> log for the same consumer group, a specific topic (users) and specific
> partition (15):
>
> [storm_kafka_to
.10.1.0 if I'm not mistaken, and has
changed again in 0.11.0.0) and each time we upgrade we have to adjust our
script to account for the format difference.
I agree it would be great if Kafka either didn't change the format so much
or better yet exposed it via JMX.
Dan
On 12 August 2017 a
res.create("store")
.withKeys(stringSerde)
.withValues(countersSerde)
.persistent()
.build();
Can you please advice how to proceed from here? any suggestion? any other
logs you might need?
Thanks
Dan Ofir
Hi,
I have an application that needs to be low latency writing to Kafka.
With the 0.81 producer I set queue.buffering.max.messages to the number of
messages I would like to producer to store in memory and
queue.enqueue.timeout.ms to 0 to have the producer throw an exception if the
server was
Hi Chris,
The new producer returns a future so it works in an async manner hence no need
for the producer.type property.
0.8 -> 0.10
batch.num.messages -> batch.size
queue.buffering.max.ms -> linger.ms
Your assumptions for the serializers look correct.
Take a look at the documentation for bot
.
- Dan
On 20 July 2015 at 17:39, Elliot West wrote:
> I'm trying to understand the difference between Encoders/Decoders and
> Serializers/Deserializers and when I should be using each. Specifically the
> types I'm referring to are:
>
>-
500 which fixed that issue
until the kernel support for that improves.
We're using CoreOS and run datastores (including kafka) with m3/r3/c3
instance types.
We found this towards the start of this year, I've not checked recently if
this has been fixed yet.
- Dan
On 3 June 2015 at 05
a property of the server, not the unit
so that wouldn't happen any more for us.
I've not tried the case where the ip -> id mapping doesn't change, given
the above setup that hard for us to test.
Thanks,
Dan
On 8 May 2015 at 18:13, Mayuresh Gharat wrote:
> Also it would
a subtle bug we've
hit?
Thanks,
Dan
Why not use the snappy/gzip compression?
On Saturday, March 22, 2014, Vishal Kapoor
wrote:
> Hello,
>
> We are in the process of building a general purpose messaging system, our
> messages are in xml format to start with, and usually are longer than 1 mb,
> what are the best practices around bre
Has anyone implemented anything? We'd like to restrict access to
individual topics, etc
Hi- wondering if anyone knows of a working .net API for kafka. Last time I
checked only .7 was available. At this point only publishing/no consumption
required.
eparate generic part from our proprietary resources/Apis
> and make it public,
>
> KR
>
> Tianning
>
> --------
> Dan Hoffman > schrieb am Mi,
> 19.3.2014:
>
> Betreff: Re: Kafka C++ Windows Client for 0.8 Releases
> An: &q
Curious- does your impl require Cygwin? If not, I'd love to use it
On Wednesday, March 19, 2014, Tianning Zhang
wrote:
> Dear all,
>
> I am writing to this list to check if there is any new progress concerning
> 0.8 Windows C++ clients and if there are experiences we can share.
>
> My company (
e guarantees.
>
> You can download the release from: http://kafka.apache.org/downloads.html
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> http://kafka.apache.org/
>
>
--
Dan Di Spaltro
Hi - am I right in that for this tool to be effective, consumers must be
using the high level consumer or otherwise keeping their offsets in
zookeeper? Is there any way to track performance without that?
On Fri, Mar 7, 2014 at 3:08 PM, Steve Morin wrote:
> Very nice
>
> > On Mar 7, 2014, at 11
Claude,
FYI I've submitted a few issues for you to look at/consider.
Dan
On Saturday, March 1, 2014, Claude Mamo wrote:
> Announcing the first release of Kafka Web Console:
> https://github.com/claudemamo/kafka-web-console/tree/v1.0. Tested on
> Chrome.
>
> Enjoy,
>
> Claude
>
ance of an existing Kafka cluster?
The jmx beans will expose the metrics per broker. You would need some
utility to aggregate across all brokers in a cluster.
Thanks,
Neha
On Thu, Feb 27, 2014 at 8:31 PM, Dan Hoffman wrote:
> Are the numbers for the entire cluster or just the broker connecte
gt;
> On Thu, Feb 27, 2014 at 7:50 PM, Dan Hoffman wrote:
>
> > Let's say I have a running cluster and users/apps are pounding away at
> it.
> > Is there a quick and easy way to measure its current throughput? I
> know
> > there are utilities for generatin
Let's say I have a running cluster and users/apps are pounding away at it.
Is there a quick and easy way to measure its current throughput? I know
there are utilities for generating volume to get stats, but I'd like to
simply get some stats about its current operation. Is there a good way to
do
ast parameter in the following call larger (say to
> 1,000,000).
>
> .addFetch(a_topic, a_partition, readOffset, 10)
>
> Thanks,
>
> Jun
>
>
> On Wed, Feb 26, 2014 at 9:32 PM, Dan Hoffman wrote:
>
> > I'm not sure what you mean - could you be more speci
gt;
> On Wed, Feb 26, 2014 at 5:40 PM, Dan Hoffman wrote:
>
> > Publisher (using librdkafka C api) has sent both gzip and snappy
> compressed
> > messages. I find that the java Simple Consumer (
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+
The kafka-console-consumer is fine.
On Wednesday, February 26, 2014, Neha Narkhede
wrote:
> Actually I meant the simple consumer shell that ships with kafka in the bin
> directory.
>
>
> On Wed, Feb 26, 2014 at 6:17 PM, Dan Hoffman wrote:
>
> > I haven't tried
On Wed, Feb 26, 2014 at 5:58 PM, Dan Hoffman
> >
> wrote:
>
> > Publisher (using librdkafka C api) has sent both gzip and snappy
> compressed
> > messages. I find that the java Simple Consumer (
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA
Publisher (using librdkafka C api) has sent both gzip and snappy compressed
messages. I find that the java Simple Consumer (
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example#)
is unable to read the snappy ones, while the High Level one is. Is this
expected? Is ther
Publisher (using librdkafka C api) has sent both gzip and snappy compressed
messages. I find that the java Simple Consumer (
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example#)
is unable to read the snappy ones, while the High Level one is. Is this
expected? Is ther
*'Not that partitioning means Kafka only provides a total order over
messages within a partition. This combined with the ability to partition
data by key is sufficient for the vast majority of applications. However,
if you require a total order over messages this can be achieved with a
topic that h
We use Kafka at work. Occasionally Zookeeper gets sad, but I'd like the
Kafka brokers and producers to be able to at least limp along instead of
totally go out of commission.
Is Kafka's dependence on Zookeeper deep or shallow?
ed-up metrics (5m, 20m, 60m, etc). It has the capability to re-emit
data when some arrives late.
-Dan
On Wed, Aug 28, 2013 at 7:55 AM, Travis Brady wrote:
> This is a very common problem in my experience. Late-arriving and
> semi-ordered data make a lot of stream processing problems mor
Someone pointed out a particularly easy fix: don't reuse files after a
restart. Done. I really like that. Simple. Any chance of this happening any
time soon?
On Sun, Apr 28, 2013 at 2:04 AM, Swapnil Ghike wrote:
> @Dan: Upon restart of the broker, if a segment already has data, the
ttp://kafka.apache.org/configuration.html ?
> >
> > It's possible to find this setting (and several other undocumented
> > settings) by looking at the source code. I'm just not sure why the
> > complete set of options is not documented on the site (is
the file creation time. However, it doesn't seem you can
> get the creation time in java.
>
> Thanks,
>
> Jun
>
>
> On Thu, Apr 25, 2013 at 11:12 PM, Dan Frankowski
> wrote:
>
> > We have high-volume topics and low-volume topics. The problem occurs more
> >
> Jun
>
>
> On Thu, Apr 25, 2013 at 12:44 PM, Dan Frankowski
> wrote:
>
> > We are running Kafka 0.7.2. We set log.roll.hours=1. I hoped that meant
> > logs would be rolled every hour, or more. Only, sometimes logs that are
> > many hours (sometimes days) old hav
We are running Kafka 0.7.2. We set log.roll.hours=1. I hoped that meant
logs would be rolled every hour, or more. Only, sometimes logs that are
many hours (sometimes days) old have more data added to them. This perturbs
our systems for reasons I won't get in to.
Have others observed this? Is it a
66 matches
Mail list logo
