n is: Since there are 2 volumes, is the second just a
> better update?, or should I read the first one too?.
>
> Thanks in advance,
> Miguel Suárez
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Capacity Engineering
linkedin.com/in/toddpalino
I'm assuming the writers were perhaps intending to emphasise the Cruise
> Control or Confluents self-balancing-cluster / auto-balancing features were
> preferable, but in my very brief Google didn't see any advice to set
> auto.leader.rebalance.enabled to false to use those tool
Going to pile on here, and also say take that blog post with a grain of
salt :)
Look at your data size, and your desired degree of parallelism for
consumers, and guide to that. Yes, 1000 partitions takes a while to create.
But unless your data size is something like 1 TB per day or more, you
proba
elp me to configure the topics and partitions.
>
> On Tue, Jan 8, 2019 at 9:19 PM Todd Palino wrote:
>
> > I think you’ll need to expand a little more here and explain what you
> mean
> > by processing them in parallel. Nearly by definition, parallelization and
> > strict o
000 sensor messages in parallel but each sensor
> message should be in order.If I create 1 partition it doesn't give high
> throughput .Order is guaranteed only inside the partition. How can
> parallelize messages without changing the order pls help me to find the
> solution.
> ———-
> <http://www.trimble.com/>
> Srinivasa Balaji L
> Principal Architect Cloud & DevOPS - TPaaS
> 10368, Westmoor Drive, Westminster, CO 80021
> *M*: +1(303) 324-9822 <+919790804422>
> *Email*: lsbal...@trimble.com
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
gt; Can I have 20k partition on a single kafka broker ?
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
the Github issues, or PRs for contributing!
-Todd
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
ble to change the replication factor in runtime? We're using
> 10.x version.
>
> Thanks,
> Devendar
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
n the source data center (near the source Kafka
> cluster), especially if I can't aggregate the metrics from the 2 data
> centers. Is there anything else that would influence me to deploy in
> MirrorMaker in either data center?
>
> Thanks
>
> Vu
>
--
*Todd
arch will index its.
>
> Is it a bad practice to have all these JVMs on the same virtual machine ?
> What do you recommend (number of machines, quantity of GB, CPU...) ? For
> the moment, each node has 4 vcpu.
>
> Gabriel.
>
> 2017-08-07 15:45 GMT+02:00 Todd Palino :
>
> &
To avoid swap you should set swappiness to 1, not 0. 1 is a request (don't
swap if avoidable) whereas 0 is a demand (processes will be killed as OOM
instead of swapping.
However, I'm wondering why you are running such large heaps. Most of the ZK
heap is used for storage of the data in memory, and
well. How does the OS/broker tuning
> affect those consumers that are close to the source datacenter? Will they
> continue to function well?
>
> -James
>
> > On Jul 23, 2017, at 7:16 AM, Todd Palino wrote:
> >
> > One of the best pieces of advice I can offer is th
c from EU to US. Only one of them
> > is high throughput. We also have a message handler to strip off some
> > sensitive information from EU to US but it only works on a low thru put
> > topic; the message handler still try to process the other topics but let
> it
> > pass thru.
> >
> > Thanks,
> > Sunil Parmar
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
> subsidiaries that is proprietary, privileged, confidential and/or
> subject
> > > to copyright. Any review, retransmission, dissemination or other use
> of,
> > or
> > > taking of any action in reliance upon, this information by persons or
> > > entities other than the intended recipient(s) is prohibited and may be
> > > unlawful. If you received this in error, please contact the sender
> > > immediately and delete and destroy the communication and all of the
> > > attachments you have received and all copies thereof.
> > >
> > >
> > >
> >
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
eassign-partition.sh)*
> *One of the blogs mentioned that - it is preferable to Re-balance Kafka
> topics manually, since setting *
>
> *auto.leader.rebalance.enable = true causes issues.*
>
> Pls let me know.
> Any other best practices wrt. Re-balancing kafka topics ?
>
> t
1
> > #zookeeper-path=/kafka-cluster/stormconsumers
> >
> > [tickers]
> > broker-offsets=20
> >
> > [lagcheck]
> > intervals=10
> > expire-group=604800
> >
> > [notify]
> > interval=10
> >
> > [httpserver]
> > server=on
> > port=8000
> > ; Alternatively, use listen (cannot be specified when port is)
> > ; listen=host:port
> > ; listen=host2:port2
> >
> > [smtp]
> > server=mailserver.example.com
> > port=25
> > from=burrow-nore...@example.com
> > template=config/default-email.tmpl
> >
> > [emailnotifier "b...@example.com"]
> > group=local,critical-consumer-group
> > group=local,other-consumer-group
> > interval=60
> >
> > [notify]
> > interval=10
> >
> > [httpnotifier]
> > url=http://notification.server.example.com:9000/v1/alert
> > interval=60
> > extra=app=burrow
> > extra=tier=STG
> > template-post=config/default-http-post.tmpl
> > template-delete=config/default-http-delete.tmpl
> > timeout=5
> > keepalive=30
> >
> > So Can you please let me know what I am missing and how to fix these
> > issues.Any help would be appreciated.
> >
> >
> >
> > Regards,
> > Abhimanyu
> >
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
pics now? Do we need to come up
> with a manual assignment plan for each of the partitions? Is there any
> quicker way to achieve this?
>
> Thanks!
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
he broker to
> failover it to other brokers. Is there a way to failover controller to a
> specific broker? Is there a way to failover it without restart the broker?
>
> Thanks,
> Jun
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
at the right offset...
> >
> > Also I m wondering if it s the case if we don t use keys (pure queuing
> > system with key=null).
> >
> > Cheers
> > Nico
> >
> >
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
ms to suggest no, but doesn't
> address the point directly:
> http://events.linuxfoundation.org/sites/events/files/slides/
> Kafka%20At%20Scale.pdf
>
> --
> Jack Foy
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
r.properties and acks=all on
> producer? min.insync.replicas only applies when acks=all.
>
> -James
>
> >
> > -Original Message-
> > From: Todd Palino [mailto:tpal...@gmail.com]
> > Sent: Monday, March 06, 2017 6:48 PM
> > To: users@kafka.apache.org
> > S
any unauthorized
> action based on information contained herein is strictly prohibited.
> Unauthorized use of information contained herein may subject you to civil
> and criminal prosecution and penalties. If you are not the intended
> recipient, please immediately notify the sender by t
ub.com/ktls/af_ktls
> http://www.phoronix.com/scan.php?page=news_item&px=FreeBSD-Faster-Sendfile
>
> Ismael
>
> On Mon, Mar 6, 2017 at 4:18 PM, Todd Palino wrote:
>
> > So that’s not quite true, Hans. First, as far as the performance hit
> being
> > not a big impact (25%
;
> > If it is true, I don t get why the message has to be decoded by Kafka. I
> > would assume that whether the message is encrypted or not, Kafka simply
> > receives it, appends it to the file, and when a consumer wants to read
> it,
> > it simply reads at the right offse
ndering if it s the case if we don t use keys (pure queuing
> system with key=null).
>
> Cheers
> Nico
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
ill –9 processID” to stop the Kafka Mirror
> Maker. I am wondering whether there is a better way (e.g. a command) to do
> so? I don’t expect to stop the mirror maker frequently but I would like to
> have a script to automate the start and stop.
>
> Thanks a lot!
> Qian Zhu
>
-
to? I have it at 65535 but I just read a
> doc that suggested > 100K is better
>
>
> On Tue, Feb 21, 2017 at 10:45 AM, Todd Palino wrote:
>
> > So I think the important thing to look at here is the IO wait on your
> > system. You’re hitting disk throughput issues, an
ng kafka 0.9.0.1 and we use org.apache.kafka.common.
> serialization.ByteArrayDeserializer and
> GroupMetadataManager.OffsetsMessageFormatter
> to parse the message.
>
> Thanks,
> Jun
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
for our message size we are at the max. This would
> argue that we need to shrink the message size - so perhaps switching to
> avro is the next step?
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
en I try to reassign with the config...
>
> {"version":1,"partitions":[{"topic":"foo","partition":2,"
> replicas":[1004,1001]}]}
>
> I see that it doesn't resolve.
>
> Status of partition reassignment:
> Reas
the servers. It’s worked for the last couple without a problem.
-Todd
On Tue, Dec 20, 2016 at 7:55 PM, Sanjeev T wrote:
> Hi,
>
> Can some of you share points on, the versions and handling leap second
> delay on Dec 31, 2016.
>
> Regards
> -Sanjeev
>
--
*Todd Palino
the same problem.
> nvalid receive (size = 1347375956 larger than 104857600).
>
> When trying to increase the size, Java Out of Memory Exception.
> Did you find a work around for the same ??
>
> Thanks.
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructur
t; > -- Surendra Manchikanti
> >
> > On Sat, Dec 10, 2016 at 10:59 AM, Todd Palino wrote:
> >
> > > Are you running something else besides the consumers that would
> maintain
> > a
> > > memory of the topics and potentially recreate them by iss
recreating a topic that has been deleted as it issues a metadata request to
try and find out what happened after an offset request for the topic fails.
-Todd
On Fri, Dec 9, 2016 at 8:37 AM, Tim Visher wrote:
> On Fri, Dec 9, 2016 at 11:34 AM, Todd Palino wrote:
>
> > Given that
> > > >
> > > > After doing all that, the topic comes back, every time.
> > > >
> > > > What can we do to delete that topic?
> > > >
> > > > --
> > > >
> > > > In Christ,
> > > >
> > > > Timmy V.
> > > >
> > > > http://blog.twonegatives.com/
> > > > http://five.sentenc.es/ -- Spend less time on mail
> > > >
> > >
> >
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
r hand-wavy ideas that came to mind might be:
> * handling topic creation in a MirrorMakerMessageHandler
> * handling topic creation in an interceptor
>
> Anyway, was hoping to get some thoughts from people who are already doing
> this.
>
> Thanks!
> -James
>
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
the brokers (B-16) got completely messed up
> and is sent for repair.
>
> But I can still see some partitions including the B-16 in its replicas,
> thereby becoming under-replicated.
>
> Is there a proper way to take broker out of rotation?
>
> Praveen
>
--
*Todd Palino*
S
past trying to automate
> legitimate mail to GMail. Getting washed out with the spammers is a real
> hazard.
>
> Thanks!
>
>
>
> --
> information informs. analysis enlightens.
>
> Brian M. Dennis, PhD
> 43403 Coton Commons Dr, Leesburg, VA 20176
> e: b...@crossjam.net
pics? Is it safe to reassign the new
> layout for this internal topic, using kafka-reassign-partitions.sh?
>
>
> Thanks, Anderson
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
y-1_/10.100.0.113
> voidbridge-oneworks-dummy integration-oneworks-dummy 15
> 0 0 0
> integration-oneworks-dummy-voidbridge-oneworks-dummy-1_/10.100.0.113
>
> > On 8 Jul 2016, at 17:20, Tom Dearman wrote:
> >
> > When you say ‘for
thing in them. I know this is not
> very like production, but on my local this I was only testing with one user
> so get just one partition filled.
>
> Tom
> > On 6 Jul 2016, at 18:08, Todd Palino >
> wrote:
> >
> > Yeah, I've written dissertations at thi
consuming data over the internet. Is there a way to encrypt this data
> without turning SSL on. There would be some performance degradation and it
> is not required in other scenarios. Has anyone tried doing cross-regional
> replication in production?
>
> Thanks,
> Reenal
>
--
*
in real time per partition? I
> cannot find any official documentation on it? How safe is it to use?
>
> Also we're planning on upgrading to 0.10.0.0 - is it still available?
>
> Cheers,
> -Kristoffer
>
> [1] kafka.log:type=Log,name=LogEndOffset,topic=xxx,partition=xxx
&g
my colleague Jon Bringhurst
profusely for helping to get the structure around the project and the
documentation cleaned up.
-Todd
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
--
*Todd Palino*
Staff Site Reliability Engineer
Data
otstrap-server
> 10.10.1.61:9092 --describe —group consumer_group_name and parsing the
> response. Is it safe or advisable to do this? I like the fact that it
> tells me each partition lag, although it is also not available if no
> consumer from the group is currently consuming.
> 3. Is
ficient, way to do it?
>
> Cheers,
> -Kristoffer
>
> [1] https://gist.github.com/krisskross/a49e462bedb89505e372672cd81129ab
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
mar wrote:
> >
> >> Hey,
> >>
> >> How can I delete particular messages from particular topic?Is that
> >> possible?
> >>
> >> Thanks,
> >> Mudit
> >>
> >>
>
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
g a few more.
> The
> > new broker machines would have a lot more storage available to them than
> > the existing brokers. Am I setting myself up for operational headaches by
> > deploying a heterogeneous (in terms of storage capacity) cluster?
> >
> > (Asked on I
iatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC
> > > > > > > -Djava.awt.headless=true"
> > > > > > > fi
> > > > > > >
> > > > > > >
> > > > > > > Is this the confluent doc you were r
#design_loadbalancing is a
> > good
> > > start.
> > >
> > > Thanks
> > >
> > > Tom Crayford
> > > Heroku Kafka
> > >
> > > On Fri, Jun 3, 2016 at 1:15 PM, cs user wrote:
> > >
> > > > Hi All,
> > >
. I don't know if there
> is anything in log file format info or Zoo info attached to original topic
> which would prevent it to work.
>
> Petr
>
> -----Original Message-
> From: Todd Palino [mailto:tpal...@gmail.com]
> Sent: 1. června 2016 10:48
> To: users@kafka.ap
gt; it´s possible to change an existing topic name?
>
> Thanks and best wishes
> Johannes
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
consumer,
but if you’re not doing that you have to maintain some mapping of consumers
to topics. And if you are using a wildcard consumer, you’re going to run
into issues with the number of topics any given group is consuming at some
point. Your system may work fine for 5 topics, but what a
timise for this level of traffic
>
> To help your answer, we're looking at potentially 16GBit/sec inbound which
> concerns our network team.
>
> If you can please share pointers to existing materials or specific details
> of your deployment, that will be great.
>
>
> Reg
”. If you are performing a rolling bounce, this can
conflict seriously with our shutdown check which assures that the cluster
under replicated count is zero before performing a shutdown.
-Todd
On Tue, Mar 29, 2016 at 1:29 PM, James Cheng wrote:
>
> > On Mar 29, 2016, at 10:33 AM, Todd Pali
alancing option?
>
> Regards,
> Srikanth
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
decrease
> partitions? What can be done to increase the amount of logs being read
> from the cluster and ingested into Elastisearch?
>
> Like I said, very new to kafka.
>
> Thanks for the help
> Tim
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
ode to hold no
> topic/group leadership (acting as passive copy) so that it can be
> decommissioned with minimal effect to Kafka clients?
>
> Thank you,
>
> --
> Muqtafi Akhmad
> Software Engineer
> Traveloka
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
jens.ran...@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> <
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> >
> Twitter <https://twitter.com/tink>
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
be interested
> > in any opinions on this anyway.
> >
> > Thanks!
> > \EF
> >
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
manager where I can delete the group?
>
> Thanks
>
> On Sat, Dec 19, 2015 at 11:47 PM, Todd Palino wrote:
>
> > If what you want to do is reset to smallest, all you need to do is stop
> the
> > consumer, delete the group from Zookeeper, and restart the consumer. It
.
-Todd
On Saturday, December 19, 2015, Akhilesh Pathodia <
pathodia.akhil...@gmail.com> wrote:
> What is the process for deleting the consumer group from zookeeper? Should
> I export offset, delete and then import?
>
> Thanks,
> Akhilesh
>
> On Fri, Dec 18, 2015 at 11:
lt;http://sematext.com/about/contact.html>
>
> On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino wrote:
>
> > That works if you want to set to an arbitrary offset, Marko. However in
> the
> > case the OP described, wanting to reset to smallest, it is better to just
> >
t;
> >
> >
> > --
> > Jens Rantil
> > Backend engineer
> > Tink AB
> >
> > Email: jens.ran...@tink.se
> > Phone: +46 708 84 18 32
> > Web: www.tink.se
> >
> > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > <
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > >
> > Twitter <https://twitter.com/tink>
> >
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
offset.reset to "smallest",
> but it does not reset the offset in zookeeper and that's why flume will not
> read messages from first offset.
>
> Is there any way to reset kafka offset in zookeeper?
>
> Thanks,
> Akhilesh
>
--
*—-*
*Todd Palino*
Staff S
al to have a topic with 1000 partitions? I was thinking about about
> two/four partitions per node. is it wrong my thought?
>
> As I'm going to process data with Spark, I could have numberPartitions
> equals numberExecutors in Spark as max, always thinking in the future and
>
mmon.serialization.StringSerializer"
>
>
>
>
> kafkaProducer.send(new ProducerRecord[String,String](topic, key, data),new
> Callback {
> def onCompletion(recordMetadata: RecordMetadata, e: Exception):Unit = {
>if(e != null) {
> logger.error(s"Could
Replicas and Isr are both a comma separated list of broker IDs. So in this
output, I am seeing that you have two Kafka brokers with IDs 1 and 2. You
have a topic, capture, with 16 partitions at replication factor 1 (1
replica per partition). The broker with ID 2 is not online, which is why it
shows
I’m not quite sure why you would need to do this - the broker IDs are not
significant outside of the internal metadata. But this is what you would
have to do for each move (assuming you are running with at least
replication factor 2):
1) Shut down the broker
2) Clear its partition data
3) Reconfig
We use loadbalancers for our producer configurations, but what you need to
keep in mind is that that connection is only used for metadata requests.
The producer queries the loadbalancer IP for metadata for the topic, then
disconnects and reconnects directly to the Kafka brokers for producing
messag
(you can use partition reassignment to
change it). But if they are not all the same, some of the tooling will
break (such as altering the partition count for the topic).
-Todd
On Fri, Oct 16, 2015 at 5:39 PM, Todd Palino wrote:
> Actually, be very careful with this. There are two differ
Actually, be very careful with this. There are two different things stored
in Zookeeper, and depending on what you're interested in you want to make
sure you're looking at the right one.
If you want to know the replica assignment - that is, what brokers a given
partition is assigned to - you need
We've had no problems with G1 in all of our clusters with varying load
levels. I think we've seen an occasional long GC here and there, but
nothing recurring at this point.
What's the full command line that you're using with all the options?
-Todd
On Wed, Oct 14, 2015 at 2:18 PM, Scott Clasen
Yes. As long as you have not reassigned the partitions to other brokers,
the wiped broker will rebuild from replication. Keep in mind, however, that
if you are using the retention by time configuration, you will have 2x
retention on that broker for the length of retention. This means that if
your r
To answer the question, yes, it is incorrect. There are a few things you
can do to minimize problems. One is to disable unclean leader election, use
acks=-1 on the producers, have an RF of 3 or greater, and set the min ISR
to 2. This means that the topic will only be available if there are at
least
Multiple topics is the model I would recommend for what you have described.
LinkedIn has an environment where we have a wide mix, in a lot of different
clusters. We have some topics that have one producer and one consumer
(queuing). We have some topics that are multi-producer (tracking and
metrics,
What Python library are you using?
In addition, there's no real guarantee that any two libraries will
implement consumer balancing using the same algorithm (if they do it at
all).
-Todd
On Wednesday, September 30, 2015, Rahul R wrote:
> I have 2 kafka consumers. Both the consumers have the sa
So I disagree with the idea to use custom partitioning, depending on your
requirements. Having a consumer consume from a single partition is not
(currently) that easy. If you don't care which consumer gets which
partition (group), then it's not that bad. You have 20 partitions, you have
20 consumer
> On Fri, Sep 25, 2015 at 12:15 AM, Todd Palino wrote:
>
> > For now, that's the way it is. Historically, we've only monitored the lag
> > for our infrastructure applications. Other users are responsible for
> their
> > own checking, typically using the maxlag
We also still have developers creating topics semi-regularly, which it
> seems like can cause the high level consumer to disconnect?
>
>
> On Fri, Sep 25, 2015 at 6:16 PM Todd Palino > wrote:
>
>> That rebalance cycle doesn't look endless. I see that you started 23
>
> Could it be the way we are bring up multiple consumers at the same time is
> hitting some sort of endless rebalance cycle? And/or the resulting
> thrashing is causing them to time out, rebalance, etc.?
>
> I've tried attaching the logs again. Thanks!
>
> On Fri, Sep 25, 20
I don't see the logs attached, but what does the GC look like in your
applications? A lot of times this is caused (at least on the consumer side)
by the Zookeeper session expiring due to excessive GC activity, which
causes the consumers to go into a rebalance and change up their connections.
-Todd
or so consumers moved
over to Kafka committed offsets at this point.
Of course, just those apps do cover well over a hundred consumer groups :)
-Todd
On Thursday, September 24, 2015, James Cheng wrote:
>
> > On Sep 24, 2015, at 8:11 PM, Todd Palino > wrote:
> >
> > W
are considered infrastructure
applications for Kafka), but we're not encouraging other internal users to
switch over just yet.
-Todd
On Wed, Sep 23, 2015 at 3:21 PM, James Cheng wrote:
>
> On Sep 18, 2015, at 10:25 AM, Todd Palino wrote:
>
> > I think the last major
Retention is going to be based on a combination of both the retention and
segment size settings (as a side note, it's recommended to use
log.retention.ms and log.segment.ms, not the hours config. That's there for
legacy reasons, but the ms configs are more consistent). As messages are
received by K
move the cleaner-offset-checkpoint file from the data directory and
> restart the servers. The log indicates all is well.
>
> Do you know what version the fix to this will be in? I'm not looking
> forward to dealing with this on a reoccurring basis.
>
> -John
>
> On
Yes, this is a known concern, and it should be fixed with recent commits.
In the meantime, you'll have to do a little manual cleanup.
The problem you're running into is a corrupt message in the offsets topic.
We've seen this a lot. What you need to do is set the topic configuration
to remove the c
This message is regarding a normal connection close. You see it in the logs
for any connection - consumers, producers, replica fetchers. It can be
particularly noisy because metadata requests often happen on their own
connection.
The log message has been moved to debug level in recent commits (it
Consumer offsets in Zookeeper are not handled by the Kafka brokers at all -
the consumer writes those directly to Zookeeper. Most likely, what you are
seeing is the interval over which the consumer is committing offsets.
Assuming that you are using the auto.commit.enable setting (it defaults to
tru
I put an answer to this on Stack Overflow. Basically, that's not how RF
works for Kafka. It's not a guarantee, it's just how the partitions are
created, and how it is reported when something is down (under replicated
partitions). While there is an option to do auto leader rebalancing,
there's no eq
t; testing for weeks, compared configuration and setups, without finding the
> main cause.
> Can this be a Kernel (version/configuration) or Java(7) issue?
>
> Cheers
> Jörg
>
>
>
> On 04.09.2015 20:24, Todd Palino wrote:
>
>> Jun's post is a good start, but I fi
Jun's post is a good start, but I find it's easier to talk in terms of more
concrete reasons and guidance for having fewer or more partitions per topic.
Start with the number of brokers in the cluster. This is a good baseline
for the minimum number of partitions in a topic, as it will assure balan
What Gwen said :)
We developed a python web service internally called Menagerie that provides
this functionality for both Kafka and Zookeeper. We use it to drive a web
dashboard for stats, our (old style) lag checking, and some other CLI
tools. Unfortunately it ties into too much internal LinkedIn
You can use the emailer config in Burrow to send alerts directly (it will
monitor specific groups and send emails out when there is a problem). If
you need something more complex than that, I think the best practice is
always to send the output into an general alert/notification system.
-Todd
On
At LinkedIn, we are using a RAID-10 of 14 disks. This is using software
RAID. I recently did some performance testing with RAID 0, 5, and 6. I
found that 5 and 6 underperformed significantly, possibly due to the parity
calculations. RAID 0 had a sizable performance gain over 10, and I would
expect
It looks like you did an unclean shutdown of the cluster, in which case
each open log segment in each partition needs to be checked upon startup.
It doesn't really have anything to do with RF=3 specifically, but it does
mean that each of your brokers has 6000 partitions to check.
What is the setti
That's exactly right. We've been talking about this internally at LinkedIn, and
how to solve it. I think the best option would be to have the broker throw an
error on offset commits until there are enough brokers to fulfill the
configured RF.
We've seen this several times now when bootstrapping
To make sure you have a complete answer here, the order of the replica list
that you specify in the partition reassignment will affect the leader
selection, but if the current leader is in the new replica list, it will
not change the leadership to change.
That is, if your current replica list is [
1 - 100 of 179 matches
Mail list logo
