Should I raise a JIRA for that?
-----Original Message-----
From: Ayush Dixit
Sent: Wednesday, April 5, 2017 1:22 PM
To: [email protected]
Subject: RE: Performance issue in Camel SFTP component
Hi Claus,
Already tried that , See below endpoint configuration . No luck :(
to("sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser&stepwise=false";)
Thanks
Ayush
-----Original Message-----
From: Claus Ibsen [mailto:[email protected]]
Sent: Wednesday, April 5, 2017 1:08 PM
To: [email protected]
Subject: Re: Performance issue in Camel SFTP component
Try with maxMessagesPerPoll to set an upper limit.
On Tue, Apr 4, 2017 at 12:10 PM, Ayush Dixit <[email protected]>
wrote:
>
> Thanks Claus , I've tried stepwise=false in camel producer . No such
> luck
>
> Also, I have tried download=false and useList=false. Still the
> performance is same. I am referring the below link for camel
> documentation
>
> http://camel.apache.org/ftp2.html
>
> Is there is anything else you think I can try?
>
> Thanks
> Ayush
>
>
> -----Original Message-----
> From: Claus Ibsen [mailto:[email protected]]
> Sent: Tuesday, April 4, 2017 1:35 PM
> To: [email protected]
> Cc: [email protected]
> Subject: Re: Performance issue in Camel SFTP component
>
> Please spend more time to read the documentation and you can find
> options to tweak the options to use or not use FTP list etc, and there
> is also a stepwise option
>
> On Tue, Apr 4, 2017 at 9:16 AM, Ayush Dixit
> <[email protected]>
> wrote:
>
>>
>>
>> Hi ,
>>
>>
>>
>>
>>
>> We have implemented a camel route where we are having camel sftp
>> producer to transfer a files to remote SFTP location but on
>> performance testing on client environment and on our local
>> environment we have observed degradation in the time for transferring files
>> to remote SFTP location.
>> Please find the detailed analysis below. The we tried the various
>> test in our local environment. In each test we put around 22 files on
>> camel file consumer and each file took below time to write the file.
>> PFB details
>>
>>
>>
>>
>>
>> ============================================================
>> ============================
>>
>> When target directory having 20,000 files. Camel sftp producer took
>> around
>> 1 minute 43 second to a transfer file
>>
>> ============================================================
>> ============================
>>
>>
>>
>> DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.FileConsumer> About to process file:
>> GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/
>> 1 /ST30PERFORPMAJAR17020726.txt] using exchange:
>> Exchange[ST30PERFORPMAJAR17020726.txt]
>>
>> INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder> Picked IOP response file :
>> ST30PERFORPMAJAR17020726.txt from IOP FTP directory
>>
>> INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder> Sending IOP response file :
>> ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory
>>
>> DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.
>> 190/icoms/1?download=false&maxMessagesPerPoll=10&
>> password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[
>> ST30PERFORPMAJAR17020726.txt]
>>
>>
>>
>>
>>
>>
>>
>> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.GenericFileConverter> Read file
>> /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFOR
>> P
>> MAJAR17020726.txt
>> (no charset)
>>
>> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> About to store
>> file: QST30PERFORPMAJAR17020726.txt using stream:
>> java.io.BufferedInputStream@54a89ff5
>>
>> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> Took 0.658
>> seconds
>> (658 millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP
>> client
>> returned: true
>>
>>
>>
>> DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.RemoteFileProducer> Wrote
>> [icoms/1/QST30PERFORPMAJAR17020726.txt]
>> to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&;
>> maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser
>> ]
>> ]
>>
>> DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> Renaming file:
>> icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/
>> ST30PERFORPMAJAR17020726.txt
>>
>> INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder> IOP response file : ST30PERFORPMAJAR17020726.txt
>> successfully sent to ICOMS FTP directory
>>
>>
>>
>>
>>
>> ============================================================
>> ===========================
>>
>> When target directory having 40,000 files. Camel sftp producer took
>> around
>> 3 minute 17 second to transfer file
>>
>> ============================================================
>> ===========================
>>
>>
>>
>> DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.FileConsumer> About to process file:
>> GenericFile[/data/users/slvm02/smp53/domains/v
>>
>>
>> mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt]
>> using exchange: Exchange[SX30RPMAJAR1702483756.txt]
>>
>> INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder>
>> Picked IOP response file :
>>
>> SX30RPMAJAR1702483756.txt from IOP
>> FTP directory
>>
>> INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder>
>> Sending IOP response file
>>
>> : SX30RPMAJAR1702483756.txt to ICOMS
>> FTP directory
>>
>> DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.
>> 190/icoms/1?download=false&maxMessagesPer
>>
>> Poll=10&password=xxxxxx&
>> tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt]
>>
>>
>>
>>
>>
>> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.GenericFileConverter> Read file
>> /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJA
>> R
>> 1702483756.txt
>> (no charset)
>>
>> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> About to store
>> file: QSX30RPMAJAR1702483756.txt using stream:
>> java.io.BufferedInputStream@
>> 7954cfa8
>>
>> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> Took 0.659
>> seconds
>> (659 millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client
>> returned: true
>>
>> DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.RemoteFileProducer> Wrote
>> [icoms/1/QSX30RPMAJAR1702483756.txt]
>> to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&;
>> maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser
>> ]
>> ]
>>
>> DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> Renaming file:
>> icoms/1/QSX30RPMAJAR1702483756.txt to:
>> icoms/1/SX30RPMAJAR1702483756.txt
>>
>> INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder> IOP response file : SX30RPMAJAR1702483756.txt
>> successfully sent to ICOMS FTP directory
>>
>>
>>
>>
>>
>>
>>
>> ============================================================
>> ========================================
>>
>> Similarly when we achieved the files from target directory. It took
>> around
>> 6 sec.It seems like there is a performance issue with camel sftp component.
>> Does it list the files in target directory which is taking time.
>>
>> ============================================================
>> =======================================
>>
>>
>>
>> PFB the producer route which we set up
>>
>> sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&;
>> password=xxxxxx&tempPrefix=Q&username=sigmauser
>>
>>
>>
>>
>>
>> Please let us know if you need anything else.
>>
>>
>>
>>
>>
>> *Ayush Dixit*
>> *Software Engineer*
>> [email protected]
>>
>> O: +91.20.25661992 x 321 <+91%2020%202566%201992> | M:
>> +91-814-982-6260 <+91%2081498%2026260> | Skype: ayushdixit421
>>
>> [image: cid:[email protected]]
>> <http://www.sigma-systems.com/>
>>
>>
>>
>>
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
