Hi Claus,
My route is running in a Storm cluster. And all I do in my processor is call a
listener to emit the File name and the produced file path to the Spout which
passes it on to the Bolts.
The thread name which I can see in my Profiler is "file:///destination_dir"
which gives an impression that its the File Producer threads that are
continously growing in number.
I will quickly run a simple test from ftp -> file without any processing and
post the results here.
Thanks a lot!
Chirag
________________________________
From: Claus Ibsen <claus.ib...@gmail.com>
To: "users@camel.apache.org" <users@camel.apache.org>
Sent: Wednesday, 5 February 2014 1:27 PM
Subject: Re: Large Number of threads getting spawned in File Producer
The ftp consumer in Camel is a single threaded scheduled thread pool.
So if threads is spawned it happens somewhere else. Maybe you do
something in that processor that causes threads to be spawned. You can
try to just do from ftp -> to file or from ftp -> to log and see if
there is some changes.
All the threads pool used by Camel are enlisted in JMX which you can
see. Maybe that can also give some pointers.
On Tue, Feb 4, 2014 at 12:33 PM, Chirag Dewan <chirag.dewa...@yahoo.in> wrote:
> Hi All,
>
> I am using Camel 2.12.1. I have a route which consumes files from a FTP
> Server and places them on the local disk.
>
> from("ftp:admin@ip:port/source_dir&throwExceptionOnConnectFailed=true&stepwise=false&readLock=changed&fastExistsCheck=true&localWorkDirectory=destination_dir/.temp").to("file:destination_dir").process(some
> processor);
>
> Now I have 10000 files(around 10kbs in size) on the FTP source directory.
> When I start my route,I see a lot of "file:destination_dir" threads created
> and in TIME_WAITING state during profiling. Roughly,there is a thread for
> each file that the FTP Consumer picks from the FTP server(2000 threads for
> around 2000 files) after which the JVM crashes due to OOM.
>
> It seems the file producer creates a new thread for every file that is
> consumed from its default thread pool.
>
> How can I restrict the number of threads? Or why are all the threads are in a
> TIME_WAIT state?
>
> Any help is highly appreciated.
>
> Thanks!
>
> Chirag
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cib...@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
