The one we are using is the bzip2 codec. That is something we could test. On Monday, August 4, 2014, Roshan Naik <ros...@hortonworks.com> wrote:
> One of the hdfs compression codecs had a memory leak. dont recall which > right now. if you are using one, try changing to a diff codec and see if > the leak goes away. > -roshan > > > On Sat, Aug 2, 2014 at 6:19 AM, Christopher Shannon <cshannon...@gmail.com > <javascript:_e(%7B%7D,'cvml','cshannon...@gmail.com');>> wrote: > >> I do want to add that the Windows agents in our configuration were >> upstream from the agent using the HDFS sink, and the upstream agents would >> not recover gracefully when the downstream agent disconnected them on >> encountering an out of memory error. >> >> >> On Sat, Aug 2, 2014 at 8:18 AM, Christopher Shannon < >> cshannon...@gmail.com >> <javascript:_e(%7B%7D,'cvml','cshannon...@gmail.com');>> wrote: >> >>> Roshan, >>> >>> After more testing, it became plain that the Windows environment was not >>> the problem. The simultaneous JVMs were behaving well in the Windows >>> environment. There does appear to be a documented memory leak problem with >>> the HDFS sink for version 1.3.0. Our distro comes from IBM BigInsights, and >>> yes, there do appear to be some differences between some dependencies from >>> the open source and what comes with the IBM distro. So we are looking into >>> working with them to fix the problem. >>> >>> In the meantime, if you can think of any threads on the list pointing to >>> HDFS sink and out of memory errors, I would be grateful if some can be sent >>> my way. >>> >>> All the best, >>> >>> Chris. >>> >>> >>> >>> On Fri, Aug 1, 2014 at 6:50 PM, Roshan Naik <ros...@hortonworks.com >>> <javascript:_e(%7B%7D,'cvml','ros...@hortonworks.com');>> wrote: >>> >>>> Is that a custom Flume build or from some distro ? Hard to say without >>>> additional info. anything interesting in the logs when it crashes ? >>>> >>>> >>>> On Tue, Jul 29, 2014 at 10:15 AM, Christopher Shannon < >>>> cshannon...@gmail.com >>>> <javascript:_e(%7B%7D,'cvml','cshannon...@gmail.com');>> wrote: >>>> >>>>> For development and testing, I sometimes have to run multiple agents >>>>> on the same server / workstation. Recently I had to load test a second >>>>> agent in a Windows environment with a near identical configuration (except >>>>> for paths and more memory allocated to the JVM), but the agent in the >>>>> second JVM soon ground to a halt whereas the first JVM with less memory >>>>> worked without slowdown. >>>>> >>>>> The total physicsl memory on the machine is 32 gb, and the total >>>>> physical RAM used never exceds 14GB. Disk space is plentiful. Agent A >>>>> (started first) has a max JVM heap space of 4 gb, the second JVM (Agent B) >>>>> has 8 gb allocated. Both use a spooling file source, a memory channel, and >>>>> an Avro sink. >>>>> >>>>> Yet agent B is the one that slows down and evetually hangs. I have >>>>> stood up multiple Flume agents on Linux without this problem. >>>>> >>>>> What could account for the degrading performance in a Windows >>>>> environment for the second agent but not the first? >>>>> >>>>> All the best, Chris >>>>> >>>>> p.s. The agent version is 1.3.0, and I can't change it for contractual >>>>> reasons. >>>>> >>>> >>>> >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or >>>> entity to which it is addressed and may contain information that is >>>> confidential, privileged and exempt from disclosure under applicable law. >>>> If the reader of this message is not the intended recipient, you are hereby >>>> notified that any printing, copying, dissemination, distribution, >>>> disclosure or forwarding of this communication is strictly prohibited. If >>>> you have received this communication in error, please contact the sender >>>> immediately and delete it from your system. Thank You. >>> >>> >>> >> > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
