Thanks, that fix would be nice :-) Paul
On Mon, 12 Sept 2022 at 10:41, Colin McCabe wrote:
> Thanks, Paul. I would be really curious to see the talk when you're done :)
>
> BTW, David Arthur posted a KIP recently that should avoid the upper limit
> on the number of elements in a batch for Create
Thanks, Paul. I would be really curious to see the talk when you're done :)
BTW, David Arthur posted a KIP recently that should avoid the upper limit on
the number of elements in a batch for CreateTopics or CreatePartitions when
it's done.
best,
Colin
On Fri, Sep 9, 2022, at 17:22, Paul Brebn
Colin, hi, current max partitions reached is about 600,000 - I had to
increase Linux file descriptors, mmap, and tweak the JVM heap settings a
bit - heap error again.
This is a bit of a hack to, as RF=1 and only a single EC2 instance - a
proper 3 node cluster would in theory give >1M partitions whi
Hi Paul,
As Keith wrote, it does sound like you are hitting a separate Linux limit like
the max mmap count.
I'm curious how many partitions you can create if you change that config!
best,
Colin
On Tue, Sep 6, 2022, at 14:02, Keith Paulson wrote:
> I've had similar errors cause by mmap counts;
I've had similar errors cause by mmap counts; try with
vm.max_map_count=262144
On 2022/09/01 23:57:54 Paul Brebner wrote:
> Hi all,
>
> I've been attempting to benchmark Kafka KRaft version for an ApacheCon
talk
> and have identified 2 problems:
>
> 1 - it's still impossible to create large numbe
Hi all,
I've been attempting to benchmark Kafka KRaft version for an ApacheCon talk
and have identified 2 problems:
1 - it's still impossible to create large number of partitions/topics - I
can create more than the comparable Zookeeper version but still not
"millions" - this is with RF=1 (as anyt
