Hi Claus, It was in the email I forwarded. So below my new email. But here it is:
from(file:///mnt/sl-nl/bij/outbox/?sortBy=ignoreCase:file:name&filter=#fileFilter&recursive=false&move=processed&moveFailed=failed&scheduler.cron=0+0/1+0-23+?+*+1,2,3,4,5,6,7&scheduler=quartz2&scheduler.triggerId=nl_bij-export-to-archive-276) to(file:///data/work/sl/work-archive/work/276/) But the same goes for any component used it looks like (ftp, sftp, etc) Thanks!! With kind regards, Hans Orbaan -----Oorspronkelijk bericht----- Van: Claus Ibsen [mailto:[email protected]] Verzonden: Monday 21 March 2016 16:37 Aan: [email protected] Onderwerp: Re: Quartz schedule too many workers in route On Mon, Mar 21, 2016 at 3:23 PM, Hans Orbaan <[email protected]> wrote: > Hi, > > Is there someone that can help a bit with the issue below? > > An endpoint (file) with a quartz cronschedule is not stateful. This means an > schedule that triggers every minute it will start processing with multiple > threads after 1 minute if it did not finish the job. My quess is this is > unwanted in almost all situations. > > This issue does not happen when spring is used as scheduler. Also this does > not happen when using a regular delay=1m. > > I tried stuff like job.stateful=true, but that is not an allowed option. > At this point I see no other solution then to start using the timer or quartz > component to trigger routes, but I would much rather keep using the file, > ftp, etc components with a schedule. > Ah can you provide an example of the uri you configure for this. We could look into what it would take to make scheduler.job.stateful=true or something being supported. > Any help would be appreciated. > > With kind regards, > > Hans Orbaan > > -----Oorspronkelijk bericht----- > Van: Hans Orbaan [mailto:[email protected]] > Verzonden: Wednesday 16 March 2016 10:18 > Aan: [email protected] > Onderwerp: Quartz schedule too many workers in route > > Hello all, > > We have a few very busy routes that are throwing FileNotFoundExceptions when > processing files (full stacktrace below). Searching for those filenames in > the log it is very clear that those files have already been processed in the > past minute and thus are not there. > After removing the 1 minute schedule and adding a 1 minute delay the problems > have disappeared. > > After testing it looks like a cron scheduler can only start 1 thread/worker > per route and a next scheduler ping will wait for the previous one to be > finished. Even if we manually trigger the route by its triggerId. This was > tested by holding a thread in a debugger for a few minutes, no new threads > passed the breakpoint until we let the first thread go. Still we are seeing > what we are seeing but I cannot create the volume of messages and routes in a > test/develop environment. > > During low volume processing we can see multiple workers busy, but always in > a different minute then the other. So 1 thread/worker is busy per minute but > the worker name can differ. > During high volume processing at some point multiple workers will be > processing, the same route, in the same second. Also there have been moments > we can find 5 camellock files at the same time, in the same directory. There > is only 1 route per directory so there should not be more than 1 camellock, I > think. > > Example from log (all the same routeId): > 2016-03-14 23:35:38,783 [DefaultQuartzScheduler-esbContext-1_Worker-1] > 2016-03-14 23:35:40,385 [DefaultQuartzScheduler-esbContext-1_Worker-7] > 2016-03-14 23:35:40,620 [DefaultQuartzScheduler-esbContext-1_Worker-6] > 2016-03-14 23:35:40,843 [DefaultQuartzScheduler-esbContext-1_Worker-8] > 2016-03-14 23:35:41,262 [DefaultQuartzScheduler-esbContext-1_Worker-1] > 2016-03-14 23:35:41,529 [DefaultQuartzScheduler-esbContext-1_Worker-2] > 2016-03-14 23:35:43,200 [DefaultQuartzScheduler-esbContext-1_Worker-7] > 2016-03-14 23:35:43,205 [DefaultQuartzScheduler-esbContext-1_Worker-6] > 2016-03-14 23:35:44,470 [DefaultQuartzScheduler-esbContext-1_Worker-8] > 2016-03-14 23:35:44,665 [DefaultQuartzScheduler-esbContext-1_Worker-1] > 2016-03-14 23:35:45,503 [DefaultQuartzScheduler-esbContext-1_Worker-2] > 2016-03-14 23:35:46,393 [DefaultQuartzScheduler-esbContext-1_Worker-7] > 2016-03-14 23:35:47,051 [DefaultQuartzScheduler-esbContext-1_Worker-6] > > The routes are very simple, from a mounted folder to a local directory with > nothing in between except for some onException handling for retrying and > sending to an error route (direct). Lowering maxMessagesPerPoll (250) did > seem to decrease the problem a bit but I am not convinced yet. This should be > tested with a number lower than 100. > > from(file:///mnt/sl-nl/bij/outbox/?sortBy=ignoreCase:file:name&filter= > #fileFilter&recursive=false&move=processed&moveFailed=failed&scheduler > .cron=0+/1+0-23+?+*+1,2,3,4,5,6,7&scheduler=quartz2&scheduler.triggerI > d=nl_bij-export-to-archive-276) > to(file:///data/work/sl/work-archive/work/276/) > > Is there anything we can change to make sure we have only one worker > processing at a time, per route? We seem to have 10 workers total that are > divided amongst the routes with schedulers. We have 250 routes total with a > schedule. Most schedules are every minute. Also there are 250 routes with > delay=5s, but they do not use those workers. > > org.quartz.threadPool.threadCount is now at 1. We might need to increase this > number as we have 250 routes with a 1 minute schedule. But the issue we have > is quite the opposite of what to expect if quartz cannot trigger all jobs it > should trigger. > > ----------------------------------------- > version.camel 2.16.2 > > file:quartz.properties > org.quartz.scheduler.instanceName = QuartzScheduler > org.quartz.scheduler.threadsInheritContextClassLoaderOfInitializer = > true org.quartz.scheduler.skipUpdateCheck = true > org.quartz.scheduler.jmx.export = true org.quartz.threadPool.class = > org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = > 1 org.quartz.jobStore.class = org.quartz.simpl.RAMJobStore > > server.info > Tomcat Version Apache Tomcat/8.0.26 > JVM Version 1.8.0_60-b27 > JVM Vendor Oracle Corporation > OS Name Linux > OS Version 3.8.13-98.2.2.el6uek.x86_64 > OS Architecture amd64 > > 2016-03-14 22:29:02,485 [DefaultQuartzScheduler-esbContext-1_Worker-10] > errors - WARN - Error stacktrace: > org.apache.camel.component.file.GenericFileOperationFailedException: Cannot > store file: /data/work/sl/work-archive/work/276/DD_SOS2016031422110816.xml > at > org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:292) > at > org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277) > at > org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165) > at > org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:79) > at > org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) > at > org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) > at > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:121) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190) > at > org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:442) > at > org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:214) > at > org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:178) > at > org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174) > at > org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101) > at > org.apache.camel.pollconsumer.quartz2.QuartzScheduledPollConsumerJob.execute(QuartzScheduledPollConsumerJob.java:59) > at org.quartz.core.JobRunShell.run(JobRunShell.java:202) > at > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja > va:573) Caused by: org.apache.camel.InvalidPayloadException: No body > available of type: java.io.InputStream but has value: > GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml] of type: > org.apache.camel.component.file.GenericFile on: DD_SOS2016031422110816.xml. > Caused by: Error during type conversion from type: java.lang.String to the > required type: byte[] with value [Body is file based: > GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]] due > java.io.FileNotFoundException: > /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory). > Exchange[ID-slesb101-60348-1457682250340-0-642502][DD_SOS2016031422110816.xml]. > Caused by: [org.apache.camel.TypeConversionException - Error during type > conversion from type: java.lang.String to the required type: byte[] with > value [Body is file based: > GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]] due > java.io.FileNotFoundException: > /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory)] > at > org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101) > at > org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:273) > ... 19 more > Caused by: org.apache.camel.TypeConversionException: Error during type > conversion from type: java.lang.String to the required type: byte[] with > value [Body is file based: > GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]] due > java.io.FileNotFoundException: > /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.createTypeConversionException(BaseTypeConverterRegistry.java:610) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:177) > at > org.apache.camel.component.file.FileBinding.loadContent(FileBinding.java:57) > at > org.apache.camel.component.file.GenericFileConverter.genericFileToInputStream(GenericFileConverter.java:123) > at sun.reflect.GeneratedMethodAccessor559.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243) > at > org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:293) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168) > at > org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:99) > ... 20 more > Caused by: org.apache.camel.RuntimeCamelException: > java.io.FileNotFoundException: > /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory) > at > org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1642) > at > org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1247) > at > org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59) > at > org.apache.camel.component.file.GenericFileConverter.convertTo(GenericFileConverter.java:97) > at sun.reflect.GeneratedMethodAccessor555.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243) > at > org.apache.camel.impl.converter.StaticMethodFallbackTypeConverter.convertTo(StaticMethodFallbackTypeConverter.java:62) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:333) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168) > ... 30 more > Caused by: java.io.FileNotFoundException: > /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory) > at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) > at java.io.FileInputStream.<init>(FileInputStream.java:138) > at > org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:78) > at > org.apache.camel.converter.IOConverter.toByteArray(IOConverter.java:266) > at sun.reflect.GeneratedMethodAccessor604.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243) > ... 39 more -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
