Roshan, Hari, This explains a lot of the problems we have been experiencing.
Thank you very much! All the best, Chris Shannon. On Monday, August 4, 2014, Hari Shreedharan <hshreedha...@cloudera.com> wrote: > This was fixed in FLUME-2416. Codecs using direct memory were leaking > direct buffers if full GC did not happen (since the direct buffers are > actually cleaned up only on full gc). > > > On Mon, Aug 4, 2014 at 2:57 PM, Christopher Shannon <cshannon...@gmail.com > <javascript:_e(%7B%7D,'cvml','cshannon...@gmail.com');>> wrote: > >> The one we are using is the bzip2 codec. That is something we could test. >> >> >> On Monday, August 4, 2014, Roshan Naik <ros...@hortonworks.com >> <javascript:_e(%7B%7D,'cvml','ros...@hortonworks.com');>> wrote: >> >>> One of the hdfs compression codecs had a memory leak. dont recall which >>> right now. if you are using one, try changing to a diff codec and see if >>> the leak goes away. >>> -roshan >>> >>> >>> On Sat, Aug 2, 2014 at 6:19 AM, Christopher Shannon < >>> cshannon...@gmail.com> wrote: >>> >>>> I do want to add that the Windows agents in our configuration were >>>> upstream from the agent using the HDFS sink, and the upstream agents would >>>> not recover gracefully when the downstream agent disconnected them on >>>> encountering an out of memory error. >>>> >>>> >>>> On Sat, Aug 2, 2014 at 8:18 AM, Christopher Shannon < >>>> cshannon...@gmail.com> wrote: >>>> >>>>> Roshan, >>>>> >>>>> After more testing, it became plain that the Windows environment was >>>>> not the problem. The simultaneous JVMs were behaving well in the Windows >>>>> environment. There does appear to be a documented memory leak problem with >>>>> the HDFS sink for version 1.3.0. Our distro comes from IBM BigInsights, >>>>> and >>>>> yes, there do appear to be some differences between some dependencies from >>>>> the open source and what comes with the IBM distro. So we are looking into >>>>> working with them to fix the problem. >>>>> >>>>> In the meantime, if you can think of any threads on the list pointing >>>>> to HDFS sink and out of memory errors, I would be grateful if some can be >>>>> sent my way. >>>>> >>>>> All the best, >>>>> >>>>> Chris. >>>>> >>>>> >>>>> >>>>> On Fri, Aug 1, 2014 at 6:50 PM, Roshan Naik <ros...@hortonworks.com> >>>>> wrote: >>>>> >>>>>> Is that a custom Flume build or from some distro ? Hard to say >>>>>> without additional info. anything interesting in the logs when it >>>>>> crashes ? >>>>>> >>>>>> >>>>>> On Tue, Jul 29, 2014 at 10:15 AM, Christopher Shannon < >>>>>> cshannon...@gmail.com> wrote: >>>>>> >>>>>>> For development and testing, I sometimes have to run multiple agents >>>>>>> on the same server / workstation. Recently I had to load test a second >>>>>>> agent in a Windows environment with a near identical configuration >>>>>>> (except >>>>>>> for paths and more memory allocated to the JVM), but the agent in the >>>>>>> second JVM soon ground to a halt whereas the first JVM with less memory >>>>>>> worked without slowdown. >>>>>>> >>>>>>> The total physicsl memory on the machine is 32 gb, and the total >>>>>>> physical RAM used never exceds 14GB. Disk space is plentiful. Agent A >>>>>>> (started first) has a max JVM heap space of 4 gb, the second JVM (Agent >>>>>>> B) >>>>>>> has 8 gb allocated. Both use a spooling file source, a memory channel, >>>>>>> and >>>>>>> an Avro sink. >>>>>>> >>>>>>> Yet agent B is the one that slows down and evetually hangs. I have >>>>>>> stood up multiple Flume agents on Linux without this problem. >>>>>>> >>>>>>> What could account for the degrading performance in a Windows >>>>>>> environment for the second agent but not the first? >>>>>>> >>>>>>> All the best, Chris >>>>>>> >>>>>>> p.s. The agent version is 1.3.0, and I can't change it for >>>>>>> contractual reasons. >>>>>>> >>>>>> >>>>>> >>>>>> CONFIDENTIALITY NOTICE >>>>>> NOTICE: This message is intended for the use of the individual or >>>>>> entity to which it is addressed and may contain information that is >>>>>> confidential, privileged and exempt from disclosure under applicable law. >>>>>> If the reader of this message is not the intended recipient, you are >>>>>> hereby >>>>>> notified that any printing, copying, dissemination, distribution, >>>>>> disclosure or forwarding of this communication is strictly prohibited. If >>>>>> you have received this communication in error, please contact the sender >>>>>> immediately and delete it from your system. Thank You. >>>>> >>>>> >>>>> >>>> >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. >> >> >
