Hey guys,
I have > 5k topics with 5 partitions each in my cluster today.
My actual cluster configuration is:
6 brokers - 16 vCPUs, 14.4 GB
Nowadays, I’m having some lags when the consumer tries to do rebalance and
sometimes even kafka doesn’t get up.
What would be a best cluster configurati
The replication factor is 4.
> On Jul 28, 2016, at 5:55 PM, David Garcia wrote:
>
> What is your replication for these topics?
>
> On 7/28/16, 3:03 PM, "Kessiler Rodrigues" wrote:
>
>Hey guys,
>
>I have > 5k topics with 5 partitions each
Hi guys,
Should I use SSD for my brokers or not?
What are the pros and cons?
Thank you!
Hi guys,
I have been experiencing some issues on kafka, where its throwing too many open
files.
I have around of 6k topics and 5 partitions each.
My cluster was made with 6 brokers. All of them are running Ubuntu 16 and the
file limits settings are:
`cat /proc/sys/fs/file-max`
200
`uli
> We normally don't see more than maybe 4000 per broker and most
> clusters have a lot fewer, so consider adding brokers and spreading
> partitions around a bit.
>
> Gwen
>
> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
> wrote:
>> Hi guys,
>>
&
?
> On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues
> wrote:
>
> I have changed it a bit.
>
> I have 10 brokers and 20k topics with 1 partition each.
>
> I looked at the kaka’s logs dir and I only have 3318 files.
>
> I’m doing some tests to see how many topics
pervisord, and maybe its config is lowering the limit, or the limits for
>> root are as you say but the limits for the kafka user aren't being set
>> properly, that sort of thing).
>>
>> If you do have 1M lines in the output, at least this might give you a
>> place t
Hey Drew,
You can easily use a WhiteList passing as parameter your regex pattern.
E.g:
Whitelist filter = new Whitelist(“topic_\\d+”);
consumer.createMessageStreamsByFilter(filter, 1);
> On Aug 19, 2016, at 2:46 AM, Drew Kutcharian wrote:
>
> Hi,
>
> Can I create a Kafka Streams app that co
