Hi, I'd start small and grow if needed.... in general. There is one bit of info missing here. Did you try with a smaller heap and had issues? If so, which issues? A few references from our internal Wiki:
- XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/gc/gc.log-$$-$(hostname)-$(date +'%Y%m%d%H%M').log - https://github.com/chewiebug/GCViewer - Good general JVM info: http://blog.asquareb.com/blog/2014/12/13/jvm-gc-settings-and-hbase/ - Good general JVM info: https://blog.codecentric.de/en/2012/08/useful-jvm-flags-part-5-young-generation-garbage-collection/ - Lower Xmx to <32GB heap because of pointer compression. https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/ Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Feb 19, 2015 at 1:53 PM, Zakee <kzak...@netzero.net> wrote: > Well are there any measurement techniques for Memory config in brokers. We > do have a large load, with a max throughput 200MB/s. What do you suggest as > the recommended memory config for 5 brokers to handle such loads? > > On Wed, Feb 18, 2015 at 7:13 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > 40G is really huge, generally you would want more like 4G. Are you sure > you > > need that? Not sure what you mean by lsof and index files being too > large, > > but the index files are memory mapped so they should be able to grow > > arbitrarily large and their memory usage is not counted in the java heap > > (in fact by having such a large heap you are taking away OS memory from > > them). > > > > -Jay > > > > On Wed, Feb 18, 2015 at 4:13 PM, Zakee <kzak...@netzero.net> wrote: > > > > > I am running a cluster of 5 brokers with 40G ms/mx for each. I found > one > > of > > > the brokers is constantly using above ~90% of memory for > jvm.heapUsage. I > > > checked from lsof output that the size of the index files for this > broker > > > is too large. > > > > > > Not sure what is going on with this one broker in the cluster? Why > would > > > the index file sizes be so hugely different on one broker? Any ideas? > > > > > > > > > Regards > > > Zakee > > > ____________________________________________________________ > > > Invest with the Trend > > > Exclusive Breakout Alert On Soaring Social Media Technology > > > > http://thirdpartyoffers.netzero.net/TGL3231/54e52a9fe121d2a9f4a27st01vuc > > ____________________________________________________________ > > Have you been injured? > > Get a free evaluation today to see what your injury case is worth. > > http://thirdpartyoffers.netzero.net/TGL3255/54e55ad9894265ad90bcbmp13duc >
