One of the beautiful things about Kafka is that it uses the disk and OS disk caching really efficiently.
Because Kafka writes messages to a contiguous log, it needs very little seek time to move the write head to the next point. Similarly for reading, if the consumers are mostly up to date with the topic then the disk head will be close to the reading point, or the data will already be in disk cache and can be read from memory. Even cooler, because disk access is so predictable, the OS can easily prefetch data because it knows what you're going to ask for in 50ms. In summary, data locality rocks. What I haven't worked out is how the data locality benefits interact with having multiple partitions. I would assume that this would make things slower because you will be writing to multiple physical locations on disk, though this may be ameliorated by only fsyncing every n seconds. I'd be keen to hear how this impacts it, or even better to see some benchmarks. -- Daniel. On Thu, 21 May 2015 at 12:56 pm Manoj Khangaonkar <khangaon...@gmail.com> wrote: > With knowing the actual implementation details, I would get guess more > partitions implies more parallelism, more concurrency, more threads, more > files to write to - all of which will contribute to more CPU load. > > Partitions allow you to scale by partitioning the topic across multiple > brokers. Partition is also a unit of replication ( 1 leader + replicas ). > And for consumption of messages, the order is maintained within a > partitions. > > But if you put 100 partitions per topic on 1 single broker, I wonder if it > is going to be an overhead. > > > > On Wed, May 20, 2015 at 1:02 AM, Carles Sistare <car...@ogury.co> wrote: > > > Hi, > > We are implementing a Kafka cluster with 9 brokers into EC2 instances, > and > > we are trying to find out the optimal number of partitions for our > topics, > > finding out the maximal number in order not to update the partition > number > > anymore. > > What we understood is that the number of partitions shouldn’t affect the > > CPU load of the brokers, but when we add 512 partitions instead of 128, > for > > instance, the CPU load exploses. > > We have three topics with 100000 messages/sec each, a replication factor > > of 3 and two consumer groups for each partition. > > > > Could somebody explain, why the increase of the number of partitions has > a > > so dramatic impact to the CPU load? > > > > > > Here under i paste the config file of kafka: > > > > broker.id=3 > > > > default.replication.factor=3 > > > > > > # The port the socket server listens on > > port=9092 > > > > # The number of threads handling network requests > > num.network.threads=2 > > > > # The number of threads doing disk I/O > > num.io.threads=8 > > > > # The send buffer (SO_SNDBUF) used by the socket server > > socket.send.buffer.bytes=1048576 > > > > # The receive buffer (SO_RCVBUF) used by the socket server > > socket.receive.buffer.bytes=1048576 > > > > # The maximum size of a request that the socket server will accept > > (protection against OOM) > > socket.request.max.bytes=104857600 > > > > > > > > # A comma seperated list of directories under which to store log files > > log.dirs=/mnt/kafka-logs > > > > # The default number of log partitions per topic. More partitions allow > > greater > > # parallelism for consumption, but this will also result in more files > > across > > # the brokers. > > num.partitions=16 > > > > # The minimum age of a log file to be eligible for deletion > > log.retention.hours=1 > > > > # The maximum size of a log segment file. When this size is reached a new > > log segment will be created. > > log.segment.bytes=536870912 > > > > # The interval at which log segments are checked to see if they can be > > deleted according > > # to the retention policies > > log.retention.check.interval.ms=60000 > > > > # By default the log cleaner is disabled and the log retention policy > will > > default to just delete segments after their retention expires. > > # If log.cleaner.enable=true is set the cleaner will be enabled and > > individual logs can then be marked for log compaction. > > log.cleaner.enable=false > > > > # Timeout in ms for connecting to zookeeper > > zookeeper.connection.timeout.ms=1000000 > > > > auto.leader.rebalance.enable=true > > controlled.shutdown.enable=true > > > > > > Thanks in advance. > > > > > > > > Carles Sistare > > > > > > > > > -- > http://khangaonkar.blogspot.com/ >
