Uneven disk usage in Kafka 0.8.1.1
Hi, With Kafka 0.8.1.1, I'm continuously running into the issue with no disk space remaining. I'm observing that on the problematic brokers partitions are distributed across several identical disks very unevenly. For example, provided the fact that partitions have similar size, I see 6 partitions being written to one disk, while the other disks get only 1-3 partitions each. What could cause that? Is that possible to somehow affect the partition distribution? Thanks, Yury
Re: Uneven disk usage in Kafka 0.8.1.1
I also noticed that some disks do not contain any partitions which is even more strange. Why does broker not use them, while other disks are overloaded? 2014-12-25 12:07 GMT+03:00 Yury Ruchin : > Hi, > > With Kafka 0.8.1.1, I'm continuously running into the issue with no disk > space remaining. I'm observing that on the problematic brokers partitions > are distributed across several identical disks very unevenly. For example, > provided the fact that partitions have similar size, I see 6 partitions > being written to one disk, while the other disks get only 1-3 partitions > each. > > What could cause that? Is that possible to somehow affect the partition > distribution? > > Thanks, > Yury >
Re: Uneven disk usage in Kafka 0.8.1.1
Hi, LogManager.nextLogDir() has the logic for choosing which directory to use. The documentation of the method says: /** * Choose the next directory in which to create a log. Currently this is done * by calculating the number of partitions in each directory and then choosing the * data directory with the fewest partitions. */ A possible explanation for what you are seeing is that some of the directories were added recently and simply did not fill up yet. Kafka doesn't attempt to rebalance logs across disks or directories (although this could be a nice feature request, at least for a utility). Gwen On Thu, Dec 25, 2014 at 1:30 AM, Yury Ruchin wrote: > I also noticed that some disks do not contain any partitions which is even > more strange. Why does broker not use them, while other disks are > overloaded? > > 2014-12-25 12:07 GMT+03:00 Yury Ruchin : > > > Hi, > > > > With Kafka 0.8.1.1, I'm continuously running into the issue with no disk > > space remaining. I'm observing that on the problematic brokers partitions > > are distributed across several identical disks very unevenly. For > example, > > provided the fact that partitions have similar size, I see 6 partitions > > being written to one disk, while the other disks get only 1-3 partitions > > each. > > > > What could cause that? Is that possible to somehow affect the partition > > distribution? > > > > Thanks, > > Yury > > >
Re: Kafka 0.8.2 release - before Santa Claus?
IMO: KAFKA-1790 - can be pushed out (or even marked as "won't fix") KAFKA-1782 - can be pushed out (not really a blocker) The rest look like actual blockers to me. Gwen On Tue, Dec 23, 2014 at 1:32 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi, > > I see 16 open issues for 0.8.2 at > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.8.2%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20updated%20DESC%2C%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC > - some with patches, some blockers, some blockers without patches. > > Are all issues listed as blockers truly blockers for 0.8.2? > > Thanks, > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > > > On Mon, Dec 1, 2014 at 8:13 PM, Joe Stein wrote: > > > If we can have a build ready by Dec 26th I think that is feasible. I > could > > prepare and post that if we think we have the votes and a stable version. > > > > /*** > > Joe Stein > > Founder, Principal Consultant > > Big Data Open Source Security LLC > > http://www.stealth.ly > > Twitter: @allthingshadoop > > / > > On Dec 1, 2014 7:56 PM, "Neha Narkhede" wrote: > > > > > +1 for doing a 0.8.2 final before the December break. > > > > > > On Mon, Dec 1, 2014 at 8:40 AM, Jun Rao wrote: > > > > > > > We are currently discussing a last-minute API change to the new java > > > > producer. We have also accumulated a few more 0.8.2 blockers. > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1642?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%200.8.2 > > > > > > > > So, we likely will have another 0.8.2 release in Dec. However, I am > not > > > > sure if that's beta2 or final. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Wed, Nov 26, 2014 at 12:22 PM, Otis Gospodnetic < > > > > otis.gospodne...@gmail.com> wrote: > > > > > > > > > Hi, > > > > > > > > > > People using SPM to monitor Kafka have been anxiously asking us > about > > > the > > > > > 0.8.2 release and we've been telling them December. Is that still > > the > > > > > plan? > > > > > > > > > > Thanks, > > > > > Otis > > > > > -- > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log > > Management > > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > >
subscribe to kafka
Question about kafka throughput
Hi all, I have a question about kafka, Kafka producer speed is different from internet. And the capacity is different with different versions. I test this through kafka-producer-perf-test.sh. 1. Could you let me know the general send speed through thread: 1 batch:1, and the corresponding server info. 2. And the maximum throughput as most producers as we can. 3. Which is the mainly resource limits about broker? Thanks, Derek
