Hi Claus, So I dont need to start/stop the template for every file?
Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template. Is this the right approach? Thanks. Chirag ________________________________ From: Claus Ibsen <claus.ib...@gmail.com> To: "users@camel.apache.org" <users@camel.apache.org> Sent: Monday, 17 February 2014 12:33 PM Subject: Re: OOM issue due to MemoryIdempotentRepository Hi You should likely reuse the consumer template instead. As this FAQ applies to it too http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <chirag.dewa...@yahoo.in> wrote: > Hi All, > > Any views on this? > > Actually my previous mail is not understandable. Let me rephrase the issue. > > I am using consumer template to consume single file using fileName property. > I have a single instance of consumerTemplate and I call doneUoW and stop > after consuming the file. > > I am running a test case where 10000 files are consumed by > consumerTemplate(one file at a time) . After a while JVM goes OOM and > crashes. In heap dump,I can see large number of MemoryIdempotentRepository > instances. I am not using idempotent=true. I figured out that > inProgressRepository also uses this cache. > > Is there something I am missing here? > > Thanks in advance. > > Chirag Dewan > > > > > ________________________________ > From: Chirag Dewan <chirag.dewa...@yahoo.in> > To: "users@camel.apache.org" <users@camel.apache.org>; Chirag Dewan > <chirag.dewa...@yahoo.in> > Sent: Friday, 14 February 2014 3:22 PM > Subject: Re: OOM issue due to MemoryIdempotentRepository > > > Hi All, > > I have been looking around why a large number of heap is being consumed by > MemoryIdempotentRepository. I saw one JIRA as well regarding > inProgressRepository not getting cleared. Can that be the reason here? > > Also,I am not using consumerTemplate to consume 1000o files. Does that also > use inProgressRepository? And when is it cleared from the memory? > > Any help is much appreciated. > > Thanks! > > Chirag Dewan > > > > > ________________________________ > > From: Chirag Dewan <chirag.dewa...@yahoo.in> > To: "users@camel.apache.org" <users@camel.apache.org>; Chirag Dewan > <chirag.dewa...@yahoo.in> > Sent: Wednesday, 12 February 2014 6:31 PM > Subject: Re: OOM issue due to MemoryIdempotentRepository > > > Hi, > > And I am using readLock=changed. So that can be a reason too? > > Chirag Dewan > > > > > ________________________________ > > From: Chirag Dewan <chirag.dewa...@yahoo.in> > To: "users@camel.apache.org" <users@camel.apache.org> > Sent: Wednesday, 12 February 2014 6:07 PM > Subject: Re: OOM issue due to MemoryIdempotentRepository > > > Hi Claus, > > Thanks for the quick reply. > > It might well be. But there is one thing I would like to get some insight > into. > > If idempotent=false(default) is used,will the LRU Cache still store the file > referrence? Because that is what I can see in my memory dump. > > org.apache.camel.util.LRUCache > >>org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > > I maybe doing something wrong here. Is there anything else I need to do to > make sure that file is deleted after it has been processed and not stored in > the cache? > > Thanks. > > Chirag Dewan > > > > > > ________________________________ > > From: Claus Ibsen <claus.ib...@gmail.com> > To: "users@camel.apache.org" <users@camel.apache.org> > Sent: Wednesday, 12 February 2014 5:42 PM > Subject: Re: OOM issue due to MemoryIdempotentRepository > > > Hi > > Some days ago someone else posted about a OOME issue when using Storm. > It smells like a Storm + Camel issue somewhere. > > > > On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <chirag.dewa...@yahoo.in> > wrote: >> Hi All, >> >> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my >> application. >> >> >> Now I intent to delete the file after consuming it. So I have delete=true in >> my route. After consuming around 6000 files(out of 10000,as my test case) >> JVM goes Out of memory. >> >> >> In my heap dump I can see the following trace as the major memory consumers: >> >> java.util.ArrayList >> >java.lang.Object >> >> >>>org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService >> >> >org.apache.camel.component.file.FileEndpoint >> >> >org.apache.camel.processor.idempotent.MemoryIdempotentRepository >> >org.apache.camel.util.LRUCache >> >>>org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap >> >> >> So if I am not using indempotent,should it still store the files in the >> memory? Is this memory repository used,even if I am deleting the files and >> not storing in memory? Is there a way I can avoid that? >> >> I am testing this scenario under Storm as execution environment and with 2GB >> max heap space. >> >> Thanks in advance >> >> Chirag Dewan > > > > -- > Claus Ibsen > ----------------- > Red Hat, Inc. > Email: cib...@redhat.com > Twitter: davsclaus > Blog: http://davsclaus.com > Author of Camel in Action: http://www.manning.com/ibsen > Make your Camel applications look hawt, try: http://hawt.io -- Claus Ibsen ----------------- Red Hat, Inc. Email: cib...@redhat.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen Make your Camel applications look hawt, try: http://hawt.io
