Hi, Peter, following what we talked about before, I want to understand what
will happen when one broker goes down, I would say it will be very similar
to what happens under disk failure, except that the rules apply to all the
partitions on that broker instead of only one malfunctioned disk. Am I
ri
Yes, that’s correct. While a broker is down:
all topic partitions assigned to that broker will be under-replicated
topic partitions with an unmet minimum ISR count will be offline
leadership of partitions meeting the minimum ISR count will move to the next
in-sync replica in the replica list
if n
So log.dirs should contain only one entry per HDD disk, to avoid random seeks.
What about SSDs? Can throughput be increased by specifying multiple directories
on the same SSD?
> On Mar 11, 2020, at 4:28 PM, Eugen Dueck wrote:
>
> So log.dirs should contain only one entry per HDD disk, to avoid random seeks.
> What about SSDs? Can throughput be increased by specifying multiple
> directories on the same SSD?
Given a constant number of partitions, I don’t see any advan
So there is not e.g. a single thread responsible per directory in log.dirs that
could become a bottleneck relative to SSD throughput of GB/s?
This is in fact the case for Apache Pulsar, and the openmessaging benchmark
uses 4 directories on the same SSD to increase throughput.
__
Couldn’t the same be accomplished by increasing the num.io.threads broker
setting?
> On Mar 11, 2020, at 5:15 PM, Eugen Dueck wrote:
>
> So there is not e.g. a single thread responsible per directory in log.dirs
> that could become a bottleneck relative to SSD throughput of GB/s?
>
> This is
I'm asking the questions here! 🙂
So is that the way to tune the broker if it does not achieve disk throughput?
差出人: Peter Bukowinski
送信日時: 2020年3月12日 9:38
Couldn’t the same be accomplished by increasing the num.io.threads broker
setting?
> On Mar 11, 2020, at 5
Thanks, very helpful !
Peter Bukowinski 于2020年3月12日周四 上午5:48写道：
> Yes, that’s correct. While a broker is down:
>
> all topic partitions assigned to that broker will be under-replicated
> topic partitions with an unmet minimum ISR count will be offline
> leadership of partitions meeting the minim
The reason I'm asking is because the documentation is quite brief:
num.io.threads: The number of threads that the server uses for processing
requests, which may include disk I/O
And none of the google hits revealed any more details.
I'm asking the questions her
Hah :)
I think this deserves an experiment. I’d try setting up some tests with one,
two, four, and eight log directories per disk and running some performance
tests. I’d be interested to see your results.
> On Mar 11, 2020, at 5:45 PM, Eugen Dueck wrote:
>
> I'm asking the questions here! 🙂
>
Thanks! Will let this list know if and when I run a log.dirs vs. num.io.threads
test.
Hah :)
I think this deserves an experiment. I’d try setting up some tests with one,
two, four, and eight log directories per disk and running some performance
tests. I’d be i
A question about something that was always in the back of my mind.
According to Jay Kreps
> The first [reason that Kafka is so fast despite writing to disk] is that
> Kafka does only sequential file I/O.
I wonder how true this statement is, because Kafka uses 3 segments per
partition. so even
12 matches
Mail list logo
