All, I sort of went against the grain and reached out to Mark Payne directly with some questions; whereas, I should have posted my issue here for a wider discussion... Let me recap...
We are using NiFi-0.3.0. In our flow, we've been using some ExtractText and ReplaceText processors, along with HandleHTTPRequest/Response processors. We had a fair number of the ExtractText processors, and the maximum buffer size we set on them was 10 MB each. As a result, we encountered out-of-memory issues. Mark made the following recommendations, which might be useful to other people. I'd also be interested to know whether other people have had problems with this... Ways to deal with OOM issues when using HTTP processors and ExtractText and Replace Text processors: -Reduce the max size configured for HttpContextMap controller service -Reduce the size of the buffers if you can for ReplaceText, ExtractText -Increase heap size to 1GB -If increasing the heap does not help, do a Java Heap Dump Use the Eclipse Memory Analyzer to find out where all the memory is being eaten up. This can show you really quickly exactly what is eating up your heap. To do this, look at the contents of $NIFI_HOME/bin/nifi.pid to get the pid of the nifi process. run: $JAVA_HOME/bin/jmap -dump:format=b,file=nifi-heap-dump.bin <http://dumpformat=b,file=nifi-heap-dump.bin> <pid> Eclipse Memory Analyzer will show exactly what class (which processor, which part of the framework, etc) is using the heap.The heap dump option will probably take a few hours to figure out how to analyze the heap dump, but it is well worth the time. Not only will it give you a definitive answer about what is going on, as a java developer it is invaluable to learn so that you can figure out exactly what your software is doing. Another question from me: Can you use AppDynamics to figure this out also? -Jenn
