Off the top of my head, it looks like it lost network connectivity to some
extent.
Question - what settings were used for topics like efGamePlay? What is min
insync replicas, replication factor, and what acks settings is the producer
using?
Cheers,
Liam
On Fri, 3 Jun 2022 at 22:55, dhiraj praja
e stream API can be done in Java only. Our
> development environment is strictly Python and while at it, we also looked
> into Faust but we're preferring something that is pure Kafka without Java
> development.
>
>
>
> On Monday, May 2, 2022, 07:11:55 PM PDT, Liam
Hi Dave,
I think you forgot to ask your question :D However, if you're looking to
group those events from the input topic and then putting the aggregated
event onto another topic, have you looked into Kafka Streams?
https://kafka.apache.org/31/documentation/streams/
Cheers,
Liam
On Tue, 3 May
Hi Emily,
Nope, Kafka doesn't have that capability built in, it's just a distributed
log that's great for streaming events. However, you can easily write a
program that consumes those events from Kafka and then does what you want
:)
Cheers,
Liam
On Tue, 3 May 2022 at 06:30, Emily Schepisi
wrot
Hi David,
There's a couple of operators you can use that I'm aware of:
Confluent for K8s:
https://docs.confluent.io/operator/current/co-deploy-cfk.html
Strimzi: https://strimzi.io/
I haven't used CFK, and big disclaimer, I work on Strimzi, so any
particular opinion I'd have on it is totally bias
Hi Peter,
Firstly, I'd check disk health, then I'd check owners and permissions on
files in your log dir, eliminate those as issues.
Secondly, are you tracking metrics on offline log dirs?
Cheers,
Liam
On Thu, 28 Apr 2022 at 15:55, Dong, Peter A. (NSB - CN/Qingdao) <
peter.a.d...@nokia-sbell.co
Hi Lavanya,
Apache Kafka is a free open source software project and as such, doesn't
provide paid support. However, numerous companies do offer a supported
Kafka product, you will be able to find them via googling "kafka support"
or similar.
Cheers,
Liam
On Wed, 27 Apr 2022 at 02:20, Lavanya Vo
Hi Matt,
Apache Kafka is a free open source software project, not too much swag on
hand ;)
Cheers,
Liam
On Tue, 19 Apr 2022 at 02:39, Matt Gilbert
wrote:
> I just wanted to reach out and thank you for all you do. I am a Senior
> Test Engineer and wanted to reach out to see if you had any swa
Hi Peter,
Are you running the Prometheus exporter on a broker? Or on a Kafka client?
Also, what version?
Cheers,
Liam Clarke-Hutchinson
On Wed, 6 Apr 2022 at 19:14, Peter Schrott (Extern)
wrote:
> Hi users,
>
> I am getting this following exceptions occasionally:
> JMX s
Hi Francis,
The rebalancing is your consumer group, it looks like your consumer is
repeatedly entering and leaving when hitting that error. Do you have logs
from the consumer side?
Cheers,
Liam
On Tue, 5 Apr 2022 at 11:30, Francis Chuang
wrote:
> Not sure if this is due to the broker rebalanc
Hi Pushkar,
Could be a lot of things. What's your linger.ms configured for?
Cheers,
Liam
On Tue, 5 Apr 2022 at 05:39, Pushkar Deole wrote:
> Hi All,
>
> We are intermittently seeing KafkaProducerException. The nested exception
> is as below:
>
> org.springframework.kafka.core.KafkaProducerExc
we will need to work out the math
> to calculate exactly how many records were dropped.
>
> Regards,
> Neeraj
> On Tuesday, 5 April, 2022, 07:47:21 am GMT+10, Liam Clarke-Hutchinson
> wrote:
>
> Hi Neeraj,
>
> However, I am unclear what the record-error-rate
people think.
Cheers,
Liam Clarke-Hutchinson
On Mon, 4 Apr 2022 at 19:29, Neeraj Vaidya
wrote:
> Thank you David and Liam for your excellent responses.
> Checking in the consumer will be extremely difficult.
> However, I am unclear what the record-error-rate|total metric for a
> pro
iling list members if that's the case.
Cheers,
Liam Clarke
On Mon, 4 Apr 2022 at 16:29, Neeraj Vaidya
wrote:
> Hi Liam,
> Brokers are on Apache Kafka v2.7.0
> However, the Producer client is using the v2.6 libraries.
>
> Regards,
> Neeraj On Monday, 4 April, 2022,
er errors indicate that these messages never made it to the
> Kafka topic at all ?
>
> Regards,
> Neeraj
> On Monday, 4 April, 2022, 12:23:30 pm GMT+10, Liam Clarke-Hutchinson <
> lclar...@redhat.com> wrote:
>
> Hi Neeraj,
>
> First off, what are your p
Hi Neeraj,
First off, what are your producer settings?
Secondly, do you have brokers logs for the leaders of some of your affected
topics on hand at all?
Cheers,
Liam Clarke-Hutchinson
On Mon, 4 Apr 2022 at 14:04, Neeraj Vaidya
wrote:
> Hi All,
> For one of the Kafka producers that I h
ide or FAQ?
>
> Kind regards
>
> Andy
>
> On 3/29/22, 5:27 PM, "Liam Clarke-Hutchinson"
> wrote:
>
> Hi Andrew,
>
> So if you've downloaded Apache Kafka, you can run a standalone connect
> instance using the bin/connect-standalone.sh
ns that use
> purely FOSS software (like the Wikimedia Foundation) makes Kafka Connect
> effectively unusable.
>
> Okay carry on! :)
>
> - Andrew Otto
>
>
>
>
>
> On Tue, Mar 29, 2022 at 8:27 PM Liam Clarke-Hutchinson <
> lclar...@redhat.com>
> wrote:
>
/www.confluent.io/hub/C0urante/kafka-connect-reddit <- I haven't
used this, but could be interesting?
I hope this helps you get started, and please let me know if I can help
with anything else :)
Cheers,
Liam Clarke
On Wed, 30 Mar 2022 at 11:54, andrew davidson
wrote:
> I found the
Hi Richard,
Yeah, the old 2.8.1 version of Kafka clients used by trunk fs2-kafka is
what I think might be the issue, not the wrapper itself, sorry if I was
unclear on that.
Please let us know how your testing with the latest fs2-kafka that's using
3.1.0 goes. :)
Kind regards,
Liam C
st version that has correct support for the
> > messaging from the ConsumerCoordinator. I am planning to do a test with
> the
> > 3.0.0-M5 version which is built on the updated 3.1.0 kafka-clients
> library
> > and will let the list know.
> >
> > -Richard Ney
>
ssue.
Ngā mihi,
Liam Clarke-Hutchinson
On Sat, 19 Mar 2022 at 14:24, Richard Ney wrote:
> Thanks for the additional information Bruno. Does this look like a possible
> bug in the CooperativeStickyAssignor? I have 5 consumers reading from a 50
> partition topic. Based on the log messages t
Oh, and meant to say, zstd is a good compromise between CPU and compression
ratio, IIRC it was far less costly on CPU than gzip.
So yeah, I generally recommend setting your topic's compression to
"producer", and then going from there.
On Wed, 16 Mar 2022 at 11:49, Liam Clarke-Hu
edundant strings.
>
> My local test is showing 75% reduction in size if I use zstd or gzip. I
> care the most about Kafka storage costs right now.
>
> On Tue, Mar 15, 2022 at 2:25 PM Liam Clarke-Hutchinson <
> lclar...@redhat.com>
> wrote:
>
> > Hi Dan,
> >
&g
ure of Kafka record size. 10% are large (>100kbs) and 90% of
> the records are smaller than 1kb. I'm working on a streaming analytics
> solution that streams impressions, user actions and serving info and
> combines them together. End-to-end latency is more important than stora
oblem domain, there might be more
advice that's applicable :)
Cheers,
Liam Clarke-Hutchinson
On Tue, 15 Mar 2022 at 10:05, Dan Hill wrote:
> Hi. I looked around for advice about Kafka compression. I've seen mixed
> and conflicting advice.
>
> Is there any sorta
Thanks,
> Dhirendra.
>
> On Mon, Mar 7, 2022 at 5:45 AM Liam Clarke-Hutchinson >
> wrote:
>
> > Ah, I may have seen this error before. Dhirendra Singh, If you grep your
> > logs, you may find an IllegalStateException or two.
> >
> > https://issues.apache.o
policies:
> - type: Percent
> value: 100
> periodSeconds: 60
> metrics:
> - resource:
> name: cpu
> target:
> averageUtilization: 30
> type: Utilization
> type: Resource
>
>
>
> Than
Ah, I may have seen this error before. Dhirendra Singh, If you grep your
logs, you may find an IllegalStateException or two.
https://issues.apache.org/jira/browse/KAFKA-12948
You need to upgrade to 2.7.2 if this is the issue you're hitting.
Kind regards,
Liam Clarke-Hutchinson
On Sun,
If you have any more details on what problem you're trying to solve, I
might be able to give more specific advice.
TL;DR - I've found using HPAs to scale applications in the same consumer
group is very useful, but it needs to be tuned to minimise scaling that can
cause pauses in consum
I've used HPAs scaling on lag before by feeding lag metrics from Prometheus
into the K8s metrics server as custom metrics.
That said, you need to carefully control scaling frequency to avoid
excessive consumer group rebalances. The cooperative sticky assignor can
minimise pauses, but not remove th
u.
>>>> Luke
>>>>
>>>>
>>>> On Mon, Dec 13, 2021 at 4:59 PM Jigar Shah
>>>> wrote:
>>>>
>>>> > Hello ,
>>>> > Answering your first mail, indeed I am using consumer groups using
>>>> >
rrect)
So yeah, this is why you're seeing that initial delay before consuming
records.
Kind regards,
Liam Clarke-Hutchinson
On Mon, 13 Dec 2021, 7:19 pm Liam Clarke-Hutchinson,
wrote:
> Hi,
>
> I'm assuming you're using consumer groups? E.g., group.id=X
>
> Cheers,
&g
Hi,
I'm assuming you're using consumer groups? E.g., group.id=X
Cheers,
Liam
On Mon, 13 Dec 2021, 6:30 pm Jigar Shah, wrote:
> Hello,
> I am trying to test the latency between message production and message
> consumption using Java Kafka-Client*(2.7.2)* library.
> The configuration of cluster
Hi Giuseppe,
That class was replaced in Kafka 2.4, I think, with
kafka.security.authorizer.AclAuthorizer.
Cheers,
Liam Clarke-Hutchinson
On Sat, Nov 13, 2021 at 1:15 AM Giuseppe Ricci Sysman
wrote:
> Hi,
>
>
>
> I'm new on security in Apache Kafka. I have Apache
elp!
>
> On Wed, Nov 10, 2021 at 3:22 PM Liam Clarke-Hutchinson <
> lclar...@redhat.com>
> wrote:
>
> > Hi David, those log messages are INFO level logged to controller.log when
> > the cluster starts up and selects a broker to act as a controller, or a
> new
&g
or.
Admittedly I could be barking up an entirely wrong tree, and if anyone who
understands the replica assignment algorithm better than I is reading,
please do correct me!
Cheers,
Liam Clarke-Hutchinson
On Thu, 11 Nov 2021, 5:16 am David Ballano Fernandez, <
dfernan...@demonware.net> wro
Hi Davide, your inline image didn't arrive intact. Are you able to upload
it to an image host and provide a link?
On Wed, Nov 10, 2021 at 10:26 PM Davide Restivo
wrote:
> Hi,
>
> I'm currently facing an integration issue between two different kafka
> versions and I was wondering if someone has a
Thanks Matthias :)
On Wed, 10 Nov 2021, 5:31 pm Matthias J. Sax, wrote:
> Done.
>
> On 11/9/21 5:06 PM, Liam Clarke-Hutchinson wrote:
> > Hi,
> >
> > My JIRA username is lclarkenz.
> >
> > Many thanks,
> >
> > Liam Clarke-Hutchinson
> >
>
>
Hi,
My JIRA username is lclarkenz.
Many thanks,
Liam Clarke-Hutchinson
Sorry forgot to mention they'll usually be under $KAFKA_DIR/logs.
On Wed, 10 Nov 2021, 1:53 pm Liam Clarke-Hutchinson,
wrote:
> Thanks :)
>
> If you grep for "broker epochs cache" in the controller.log.* files, are
> you seeing all of your brokers listed?
>
at all of your brokers are live
at the time of topic creation. If their id is in that hashmap, they're
alive.
Cheers,
Liam
On Wed, Nov 10, 2021 at 1:21 PM David Ballano Fernandez <
dfernan...@demonware.net> wrote:
> We are using Kafka with zookeeper
>
> On Tue, Nov 9, 202
ers. Is that broker
> config?
>
>
> On Tue, Nov 9, 2021 at 3:50 PM Liam Clarke-Hutchinson >
> wrote:
>
> > Thanks David,
> >
> > Hmm, is the property create.topic.policy.class.name set in
> > server.properties at all?
> >
> > Cheers,
&
admin API from
> confluent kafka python.
> The same happened for both.
>
> thanks!
>
> On Tue, Nov 9, 2021 at 2:58 PM Liam Clarke-Hutchinson >
> wrote:
>
> > Hi David,
> >
> > What tool(s) are you using to create new topics? Is it the
> kafka-topics.sh
Hi David,
What tool(s) are you using to create new topics? Is it the kafka-topics.sh
that ships with Apache Kafka?
Cheers,
Liam Clarke-Hutchinson
On Wed, Nov 10, 2021 at 11:41 AM David Ballano Fernandez <
dfernan...@demonware.net> wrote:
> Hi All,
> Trying to figure out why my
Spark Structured Streaming has some significant limitations compared to
Kafka Streams.
This one has always proved hard to overcome:
"Multiple streaming aggregations (i.e. a chain of aggregations on a
streaming DF) are not yet supported on streaming Datasets."
On Thu, 29 Apr. 2021, 8:13 am Pa
everything is happy in your cluster, network etc., you won't see
immediate negative effects on idempotence when increasing that property.
But when the cluster/network/etc. is heavily loaded, then you'll start to
see the negative effects.
Does that make sense?
Kind regards,
Liam Clarke-Hutc
shed it on one bus ride, but gives a great description of the
log-based architecture LinkedIn built around Kafka, and was instrumental in
convincing me, and my bosses(!) that Kafka would be an ideal backbone for
our data streams in my previous company.
Cheers,
Liam Clarke-Hutchinson
On Tue, Apr 1
Ah, thanks Todd :)
Was it causing issues back in the day of consumer rebalances always being
stop the world? I was wondering if the statement had perhaps predated the
cooperative / stick assignors we're able to run now.
Cheers,
Liam Clarke-Hutchinson
On Tue, Apr 13, 2021 at 2:34 AM
uster / auto-balancing features were
preferable, but in my very brief Google didn't see any advice to set
auto.leader.rebalance.enabled to false to use those tools.
So yeah, just curious if this rings any bells.
Cheers,
Liam Clarke-Hutchinson
the same issue.
>
> is it related to the source connecter(file pulse in my case). It is
> scanning around 20K files and stuck. With smaller scan directory, it is
> properly running.
>
> On Mon, Apr 5, 2021 at 2:52 PM Liam Clarke-Hutchinson <
> liam.cla...@adscale.co.n
KC sink or source connector, response times from
the underlying data source/store can impact performance, so you may also
need to look into that aspect also.
Cheers,
Liam Clarke-Hutchinson
On Thu, 1 Apr. 2021, 5:26 pm Himanshu Shukla,
wrote:
> Hi,
> I am using kafka-connect-file-pulse connec
Hi Mazzen,
Have you looked into incremental cooperative rebalancing? It may help with
your issues, at least it can avoid stop the world rebalances.
https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka/
Cheers,
Liam Clarke
On Mon, 29 Mar. 2021, 8:04 pm Mazen Ezzeddine
isk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, da
Hi Mich,
Your bigquery-sink.properties file is in a JSON format - which won't work.
It needs to follow the usual format of a Java properties file.
Kind regards,
Liam Clarke-Hutchinson
On Fri, Mar 12, 2021 at 12:13 AM Mich Talebzadeh
wrote:
> Hi,
>
>
> Trying to stream fro
As Ryanne said,
MM2 "syncs" offsets - in that it maintains a mapping of "cluster A offsets"
to "cluster B offsets" in cluster B, so if you have to move a consumer
group from A to B, you can relatively easily point the consumer group at
the offsets on B that map to its offsets on A.
On Mon, Mar 8,
orage after processing.
> Also when the user is inactive for a 10 min period then send a special
> event that marks the user as inactive. I'm trying to implement the special
> event here.
>
> Thanks
>
>
> On Tue, Feb 16, 2021 at 1:18 AM Liam Clarke-Hutchinson <
&
On Mon, Feb 15, 2021 at 1:42 AM Liam Clarke-Hutchinson <
> liam.cla...@adscale.co.nz> wrote:
>
> > Hi Navneeth,
> >
> > What is the purpose of holding these user records? Is it to join against
> > other streams, or emit to data storage?
> >
> > Cheers
Hi Navneeth,
What is the purpose of holding these user records? Is it to join against
other streams, or emit to data storage?
Cheers,
Liam Clarke-Hutchinson
On Mon, 15 Feb. 2021, 9:08 pm Navneeth Krishnan,
wrote:
> Hi All,
>
> I have a question about how I can use window stores t
Hi Gilles,
Looks like 2.5.0+ will do, as far as I can tell from the source,
print.partition came in with changes made for KAFKA-9417 which was released
as part of 2.5.0.
Cheers,
Liam
On Thu, 11 Feb. 2021, 11:36 am Gilles Philippart,
wrote:
> Hi Lorenzo
>
> It seems to have been added only r
ma version compatibility management.
Kind regards,
Liam Clarke-Hutchinson
On Mon, 1 Feb. 2021, 3:49 am Peter Penzov, wrote:
> Hello All,
> I'm working on proof of concept for sending several Java Objects
> under one Kafka Topic. More about the requirements:
>
> https://stacko
Hi,
What version of Gradle?
Cheers,
Liam
On Wed, 27 Jan. 2021, 10:35 pm felixzh, wrote:
> ~/Downloads/kafka-2.7.0# gradle clean
>
>
>
>
> > Configure project :
>
> Building project 'core' with Scala version 2.13.3
>
>
>
>
> FAILURE: Build failed with an exception.
>
>
>
>
> * Where:
>
> Build
Hi John,
The default for the C# client is actually 5ms, not that it really makes a
difference. Are you able to share any of your producer code, especially
around the configuration?
Kind regards,
Liam Clarke
On Thu, Dec 3, 2020 at 5:01 PM John Coleman wrote:
> Hi,
>
> We
Topic limits can't override broker limits.
On Tue, Dec 1, 2020 at 6:42 PM Sakke Ferraris wrote:
> Hi all!
>
> I have questions regarding configuration for large messages. I understand
> that kafka has settings for broker and topic for message max sizes;
> message.max.bytes (broker config) and ma
blog post has some good ideas:
https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/
Cheers,
Liam Clarke
On Tue, Nov 24, 2020 at 10:51 PM Gowtham S wrote:
> Hi,
> Can we have 1000 partitions in a Single topic? If not how many partitions
> will a single topic h
Can you upgrade Kafka to 2.5.1? This problem was fixed in that release.
https://issues.apache.org/jira/browse/KAFKA-9839
On Wed, Nov 25, 2020 at 9:44 PM Dev Op wrote:
> Hello community! Hasn't anyone faced a similar problem? I see nobody can
> give me advice on what's happening with our Kafka cl
;s logs to figure out more.
Cheers,
Liam Clarke-Hutchinson
On Tue, 24 Nov. 2020, 4:15 am Mohmmad Ahmad, wrote:
> Hi There,
>
> we are facing following issue in kafka. We have gone through some of the
> references and found it as a bug from kafka side. Please let us know if
>
That said John, nothing wrong with being explicit in code. :)
On Fri, 20 Nov. 2020, 1:46 pm John Roesler, wrote:
> Oh, nice. Thanks, Daniel!
>
> That’s much nicer than my ham-handed approach.
>
> Thanks,
> John
>
> On Thu, Nov 19, 2020, at 17:44, Daniel Hinojosa wrote:
> > Hope this helps, I tri
Hi,
So yeah, you can set the max.block.ms to 0, at the cost of perhaps some
data loss and occasional noise on the console.
But usual logging solution to this is to wrap the Kafka appender in an
async appender with an appropriately sized buffer.
Cheers,
Liam Clarke
On Mon, 2 Nov. 2020, 11:27
t; There are many possible variations on this approach. For example each
> processor thread can have its own queue, and the consumer threads can hash
> into these queues using the TopicPartition to ensure in-order consumption
> and simplify commit.
Cheers,
Liam Clarke-Hutchinson
On Tue, Oct
Hi Jonathon,
Kafka Connect is designed to, well, connect things to Kafka.
Kafka Streams is for stream processing using Kafka.
They're rather descriptive and apt names, don't you think? If it helps, we
abbreviate Connect as KC and Streams as KStreams in our organisation.
Kind rega
n the first replica in the list, hence preferred leader for 80% of the
partitions...
...and I think this incident has very much sold my team on Cruise Control.
Thanks for the reply :)
Liam Clarke
On Tue, 20 Oct. 2020, 11:49 pm Tom Bentley, wrote:
> Hi Liam,
>
> I think you have a misun
e, or in the worst case, recommendations on how to
manually elect leaders for a topic-partition to get us through the weekend.
Kind regards,
Liam
On Fri, 16 Oct. 2020, 6:22 pm Liam Clarke-Hutchinson, <
liam.cla...@adscale.co.nz> wrote:
> Kia ora,
>
> I am rather bemused by this on
ed rack.ids,
but what are the origins of a "LeaderElectionNotNeededException"? And for
the love of God, where is the parameter in the admin client for "I
significantly disagree".
Kind regards,
Liam Clarke-Hutchinson
slower.
So it really depends on how you intend to use the two topics.
Kind regards,
Liam Clarke-Hutchinson
On Thu, 15 Oct. 2020, 12:32 am Victoria Zuberman, <
victoria.zuber...@imperva.com> wrote:
> Hi,
>
> Background: Java, Kafka 2.1.0
>
> I have application that sends to
gest looking in the logs for brokers 1 and 2 for any ReplicaFetcher
errors as a next step.
Cheers,
Liam Clarke-Hutchinson
On Sun, 27 Sep. 2020, 5:54 pm Franz van Betteraey,
wrote:
> Hi all,
> <https://stackoverflow.com/posts/64080819/timeline>
>
> I have a strange Kafka Serv
Hi Gowtham, that output indicates that there is no lag for the consumer
group involved. What makes you think there is?
On Fri, Sep 18, 2020 at 7:21 PM Gowtham S wrote:
> Thanks for the reply Liam Clarke-Hutchinson, Shaohan Yin
>
> > Hi, what is the output of kafka-consumer-groups.s
Hi, what is the output of kafka-consumer-groups.sh --describe for that
consumer's group please?
On Thu, 17 Sep. 2020, 7:37 pm Gowtham S, wrote:
> Hello All,
> We are consuming a topic with a single partition, the consumer.poll(1000)
> returns "0" records mostly event if we have more than 1 r
g for AWS to catch up - latest supported version is 2.4.1.
I'd suggest trialling MSK alongside an existing cluster to get some ideas
of how the costs will stack up for you, to decide if it's the right path.
Kind regards,
Liam Clarke-Hutchinson
On Mon, Sep 7, 2020 at 9:27 PM Himanshu Shu
Hi Pirow,
You can configure the auto offset reset for your stream source's consumer
to "earliest" if you want to consume all available data if no committed
offset exists. This will populate the state store on first run.
Cheers,
Liam Clarke-Hutchinson
On Thu, 20 Aug. 2020,
Brilliant, thanks for that Matthias :)
On Tue, 18 Aug. 2020, 3:07 am Matthias J. Sax, wrote:
> Your understanding is correct.
>
> On 8/16/20 10:25 PM, Liam Clarke-Hutchinson wrote:
> > Kia ora koutou katoa,
> >
> > Just double checking my understanding - the R
rect in my understanding?
Kind regards,
Liam Clarke-Hutchinson
One option could be to configure the topic to use log append time as the
timestamp, not producer timestamp.
On Sun, 16 Aug. 2020, 7:23 am Matthias J. Sax, wrote:
> What is you configs? There are many configs that interact with each other.
>
> I did a Kafka Summit talk about this topic that may h
Hi Kumar,
You can restore but you'll lose data since your last snapshot. I use KC ->
SO instead for obvious reasons.
All that said, I've never had multiple Kafka nodes fail to the extent that
I need my S3 backup. But it's good to have it, for peace of mind.
Thanks,
Liam Cla
some overhead around getting your Kafka Connect
workers deployed and configured correctly. But if you're looking to run a
hot/warm (main/backup) cluster scenario, MM 2 is ideal.
Kind regards,
Liam Clarke-Hutchinson
On Sun, Aug 9, 2020 at 11:56 PM Dor Ben Dov wrote:
> Hi Liam,
>
blems you're trying to
solve, our advice can be more directed :)
Kind regards,
Liam Clarke-Hutchinson
On Sun, Aug 9, 2020 at 11:43 PM Dor Ben Dov wrote:
> Hi All,
> What is the best recommended way, and tool to backup kafka in production?
> Regards,
> Dor
>
> This email a
t; available topics and export metrics for all of them?
>
> On Thu, Jul 16, 2020 at 10:04 AM Liam Clarke-Hutchinson <
> liam.cla...@adscale.co.nz> wrote:
>
> > Whoops, just spotted a typo - the second $1 in the above snippet should
> of
> > course be $2.
> >
>
Whoops, just spotted a typo - the second $1 in the above snippet should of
course be $2.
On Thu, Jul 16, 2020 at 4:33 PM Liam Clarke-Hutchinson <
liam.cla...@adscale.co.nz> wrote:
> Hi Pushkar,
>
> There are broker side metrics for messages in / bytes in / bytes out per
> topic
want to
then use a Prometheus aggregating rule to collate all the per-consumer apps
into a per-consumer-group metric, unless the per consumer granularity is of
interest to you: https://prometheus.io/docs/practices/rules/
Hope that helps,
Kind regards,
Liam Clarke-Hutchinsons
On Thu, 16 Jul. 2020,
Hi Ann,
It's common practice in many Spark Streaming apps to store offsets external
to Kafka. Especially when checkpointing is enabled.
Are you sure that the app is committing offsets to Kafka?
Kind regards,
Liam Clarke
On Thu, 9 Jul. 2020, 8:00 am Ann Pricks, wrote:
> Hi
Hi Nag,
ISR is the replicas that are in sync with the leader, and there's a
different ISR set for each partition of a given topic. If you use
`kafka/bin/kafka-topics --describe --topic ` it'll show you the replicas
and ISR for each partition.
min.insync.replicas and replication factor are all abo
...the classpath is required to make the formatter available to the console
producer consumer...
On Thu, Jun 25, 2020 at 9:44 AM Liam Clarke-Hutchinson <
liam.cla...@adscale.co.nz> wrote:
> To investigate, you can read the messages on the topic to figure out how
> consumer groups ar
3ec471c21>.
You can then parse and analyse the output data using your preferred
scripting language.
Hope that helps,
Kind regards,
Liam Clarke-Hutchinson
On Thu, Jun 25, 2020 at 3:25 AM Karolis Pocius
wrote:
> Check if any of your consumers have auto commit turned off and instead
> com
ite.
Hope that makes sense?
Kind regards,
Liam Clarke-Hutchinson
On Wed, Jun 24, 2020 at 3:22 PM Can Zhang wrote:
> Hello,
>
> I'm doing some tests about behaviors of Kafka under faulty circumstances.
>
> Here are my configs(one of total 3, comments removed):
>
> broke
auto created topics have your
desired partitions, you will need to change that property on the brokers
and rolling restart.
Cheers,
Liam Clarke-Hutchinson
On Mon, Jun 22, 2020 at 4:16 PM sunil chaudhari
wrote:
> Manoj,
> You mean I have execute this command manually for all 350 Top
Hi Sunil,
The broker setting num.partitions only applies to automatically created
topics (if that is enabled) at the time of creation. To change partitions
for a topic you need to use kafka-topics.sh to do so for each topic.
Kind regards,
Liam Clarke-Hutchinson
On Mon, Jun 22, 2020 at 3:16 AM
cated partition
prevents consuming.
Cheers,
Liam Clarke-Hutchinson
On Sun, 21 Jun. 2020, 2:08 pm Nag Y, wrote:
> Thanks D C. Thanks a lot . That is quite a detailed explanation.
> If I understand correctly, ( ignoring the case where producers
> create transactions) - since the replica
replicas are only B and C, but the reassignment request fails.
It's causing issues in that our alerting on under-replicated partitions
keeps picking up partition 10 as being underreplicated.
Any ideas on how to convince the cluster to discard the replica on broker A?
Kind regards,
Liam Clarke-Hutchinson
or more --config options.
Hope this helps :)
Cheers,
Liam Clarke-Hutchinson
On Sun, Jun 14, 2020 at 6:48 PM Nag Y wrote:
> Is it documented somewhere the property is shared or we need to look it in
> multiple places ?
>
> Thanks
>
> On Sun, Jun 14, 2020 at 12:15 PM L
Hi Nag,
If topic and brokers share the same property, then it is safe to assume
that the broker property is the default, which can be overridden by
explicit topic configs.
Kind regards,
Liam Clarke
On Sun, Jun 14, 2020 at 6:42 PM Nag Y wrote:
> I am going through the documentation and of
1 - 100 of 219 matches
Mail list logo
