Seems reasonable. Removing brokers should be done with care, and it's tricky to reassign partitions off of brokers that aren't online (i don't even try, I just bring up a broker with that ID and reassign them off).
Depending on what you have running connected to kafka the reassigning may have been automatic. Here cruise-control and kafka-monitor can do that behind our backs. Or if new topics are created while the extra broker is online, it may just be included in the new assignment list. The reassignment tool allows throttles to be set when reassigning, but even when using that they will not be ISRs until the reassignments have finished. So we expect ISR alerts when any of those things are happening. Glad you worked it out! On Wed, Sep 19, 2018 at 6:31 AM Emmett Butler <emm...@parsely.com> wrote: > In the particular case I'm referring to, I believe some partitions were > replicated to new nodes. These new nodes had been added to the cluster as a > result of human error, and were removed shortly thereafter. It's my > hypothesis that this resulted in many partitions having their replicas on > nodes that were no longer in the cluster and were never expected to rejoin. > > Maybe this was a case in which the partition reassignment CLI tool would > have been useful? > > On Thu, Sep 6, 2018 at 11:09 PM Brett Rann <br...@zendesk.com.invalid> > wrote: > > > We have partitions that are in the 100s of GBs. > > > > It shouldn't have to shuffle around GB chunks of data unless you have > done > > partition moves, or had a broker offline for a while. Is that the case? > > > > If not your ISR problem is probably related to something else other than > > retention size. > > > > On Fri, Sep 7, 2018 at 7:25 AM Emmett Butler <emm...@parsely.com> wrote: > > > > > Hi users, > > > > > > What's the biggest topic you've seen in production use? The factors I'm > > > interested in are log.retention.* config parameters and number of > > > partitions. > > > > > > We run several 200-partition topics that have log retention set to > > roughly > > > 1GB per partition. Is this big by common usage standards, or not so > big? > > > > > > I ask because we've had some situations in which ISRs appeared to take > an > > > inordinately long time to synchronize, and I'm guessing this is > partially > > > related to the need to shuttle around 1GB hunks of data. > > > > > > -- > > > Emmett Butler | Senior Software Engineer > > > < > > > > > > http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature > <http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature> > > > < > > > http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature > <http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature> > > >> > > > > > > > > -- > Emmett Butler | Senior Software Engineer > < > http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature > <http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature>> >
