Hi Have not had time to look into this. You are welcome to log a JIRA so we can improve this in a future release http://camel.apache.org/support.html
On Wed, Mar 23, 2016 at 5:33 PM, Hans Orbaan <[email protected]> wrote: > Hi, > > The only way I have found to make sure a file or ftp component does not use > multiple threads while consuming very large batches is to annotate the > QuartzScheduledPollConsumerJob class with @DisallowConcurrentExecution. > > That’s it. It seems to work for us. I am not familiar enough with the > Camelcode to say what side effects it has and if this would prevent any > quartz job in camel to now be single threaded, even if the user does not want > it to be. > But to me it looks like an oversight when moving from quartz to quartz2. A > file or ftp consumer should be single threaded while retrieving. > > Hope this helps... > > With kind regards, > > Hans Orbaan > > > -----Oorspronkelijk bericht----- > Van: Hans Orbaan [mailto:[email protected]] > Verzonden: Wednesday 23 March 2016 9:52 > Aan: [email protected] > Onderwerp: RE: Quartz schedule too many workers in route > > Hi Claus, > > Did you have a chance to check this already? > I really think file and ftp consumers should never use multiple threads > unless explicitly defined. This means a from(file://) with a quartz schedule > should be stateful by default. Only then would it have the same behaviour as > spring scheduler or when using a delay. > > Thanks! > > With kind regards, > > Hans Orbaan > > > -----Oorspronkelijk bericht----- > Van: Hans Orbaan [mailto:[email protected]] > Verzonden: Monday 21 March 2016 16:52 > Aan: [email protected] > Onderwerp: RE: Quartz schedule too many workers in route > > Hi Claus, > > It was in the email I forwarded. So below my new email. But here it is: > > from(file:///mnt/sl-nl/bij/outbox/?sortBy=ignoreCase:file:name&filter=#fileFilter&recursive=false&move=processed&moveFailed=failed&scheduler.cron=0+0/1+0-23+?+*+1,2,3,4,5,6,7&scheduler=quartz2&scheduler.triggerId=nl_bij-export-to-archive-276) > to(file:///data/work/sl/work-archive/work/276/) > > But the same goes for any component used it looks like (ftp, sftp, etc) > > Thanks!! > > With kind regards, > > Hans Orbaan > > -----Oorspronkelijk bericht----- > Van: Claus Ibsen [mailto:[email protected]] > Verzonden: Monday 21 March 2016 16:37 > Aan: [email protected] > Onderwerp: Re: Quartz schedule too many workers in route > > On Mon, Mar 21, 2016 at 3:23 PM, Hans Orbaan <[email protected]> wrote: >> Hi, >> >> Is there someone that can help a bit with the issue below? >> >> An endpoint (file) with a quartz cronschedule is not stateful. This means an >> schedule that triggers every minute it will start processing with multiple >> threads after 1 minute if it did not finish the job. My quess is this is >> unwanted in almost all situations. >> >> This issue does not happen when spring is used as scheduler. Also this does >> not happen when using a regular delay=1m. >> >> I tried stuff like job.stateful=true, but that is not an allowed option. >> At this point I see no other solution then to start using the timer or >> quartz component to trigger routes, but I would much rather keep using the >> file, ftp, etc components with a schedule. >> > > Ah can you provide an example of the uri you configure for this. We could > look into what it would take to make scheduler.job.stateful=true or something > being supported. > > >> Any help would be appreciated. >> >> With kind regards, >> >> Hans Orbaan >> >> -----Oorspronkelijk bericht----- >> Van: Hans Orbaan [mailto:[email protected]] >> Verzonden: Wednesday 16 March 2016 10:18 >> Aan: [email protected] >> Onderwerp: Quartz schedule too many workers in route >> >> Hello all, >> >> We have a few very busy routes that are throwing FileNotFoundExceptions when >> processing files (full stacktrace below). Searching for those filenames in >> the log it is very clear that those files have already been processed in the >> past minute and thus are not there. >> After removing the 1 minute schedule and adding a 1 minute delay the >> problems have disappeared. >> >> After testing it looks like a cron scheduler can only start 1 thread/worker >> per route and a next scheduler ping will wait for the previous one to be >> finished. Even if we manually trigger the route by its triggerId. This was >> tested by holding a thread in a debugger for a few minutes, no new threads >> passed the breakpoint until we let the first thread go. Still we are seeing >> what we are seeing but I cannot create the volume of messages and routes in >> a test/develop environment. >> >> During low volume processing we can see multiple workers busy, but always in >> a different minute then the other. So 1 thread/worker is busy per minute but >> the worker name can differ. >> During high volume processing at some point multiple workers will be >> processing, the same route, in the same second. Also there have been moments >> we can find 5 camellock files at the same time, in the same directory. There >> is only 1 route per directory so there should not be more than 1 camellock, >> I think. >> >> Example from log (all the same routeId): >> 2016-03-14 23:35:38,783 [DefaultQuartzScheduler-esbContext-1_Worker-1] >> 2016-03-14 23:35:40,385 [DefaultQuartzScheduler-esbContext-1_Worker-7] >> 2016-03-14 23:35:40,620 [DefaultQuartzScheduler-esbContext-1_Worker-6] >> 2016-03-14 23:35:40,843 [DefaultQuartzScheduler-esbContext-1_Worker-8] >> 2016-03-14 23:35:41,262 [DefaultQuartzScheduler-esbContext-1_Worker-1] >> 2016-03-14 23:35:41,529 [DefaultQuartzScheduler-esbContext-1_Worker-2] >> 2016-03-14 23:35:43,200 [DefaultQuartzScheduler-esbContext-1_Worker-7] >> 2016-03-14 23:35:43,205 [DefaultQuartzScheduler-esbContext-1_Worker-6] >> 2016-03-14 23:35:44,470 [DefaultQuartzScheduler-esbContext-1_Worker-8] >> 2016-03-14 23:35:44,665 [DefaultQuartzScheduler-esbContext-1_Worker-1] >> 2016-03-14 23:35:45,503 [DefaultQuartzScheduler-esbContext-1_Worker-2] >> 2016-03-14 23:35:46,393 [DefaultQuartzScheduler-esbContext-1_Worker-7] >> 2016-03-14 23:35:47,051 [DefaultQuartzScheduler-esbContext-1_Worker-6] >> >> The routes are very simple, from a mounted folder to a local directory with >> nothing in between except for some onException handling for retrying and >> sending to an error route (direct). Lowering maxMessagesPerPoll (250) did >> seem to decrease the problem a bit but I am not convinced yet. This should >> be tested with a number lower than 100. >> >> from(file:///mnt/sl-nl/bij/outbox/?sortBy=ignoreCase:file:name&filter= >> #fileFilter&recursive=false&move=processed&moveFailed=failed&scheduler >> .cron=0+/1+0-23+?+*+1,2,3,4,5,6,7&scheduler=quartz2&scheduler.triggerI >> d=nl_bij-export-to-archive-276) >> to(file:///data/work/sl/work-archive/work/276/) >> >> Is there anything we can change to make sure we have only one worker >> processing at a time, per route? We seem to have 10 workers total that are >> divided amongst the routes with schedulers. We have 250 routes total with a >> schedule. Most schedules are every minute. Also there are 250 routes with >> delay=5s, but they do not use those workers. >> >> org.quartz.threadPool.threadCount is now at 1. We might need to increase >> this number as we have 250 routes with a 1 minute schedule. But the issue we >> have is quite the opposite of what to expect if quartz cannot trigger all >> jobs it should trigger. >> >> ----------------------------------------- >> version.camel 2.16.2 >> >> file:quartz.properties >> org.quartz.scheduler.instanceName = QuartzScheduler >> org.quartz.scheduler.threadsInheritContextClassLoaderOfInitializer = >> true org.quartz.scheduler.skipUpdateCheck = true >> org.quartz.scheduler.jmx.export = true org.quartz.threadPool.class = >> org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = >> 1 org.quartz.jobStore.class = org.quartz.simpl.RAMJobStore >> >> server.info >> Tomcat Version Apache Tomcat/8.0.26 >> JVM Version 1.8.0_60-b27 >> JVM Vendor Oracle Corporation >> OS Name Linux >> OS Version 3.8.13-98.2.2.el6uek.x86_64 >> OS Architecture amd64 >> >> 2016-03-14 22:29:02,485 [DefaultQuartzScheduler-esbContext-1_Worker-10] >> errors - WARN - Error stacktrace: >> org.apache.camel.component.file.GenericFileOperationFailedException: Cannot >> store file: /data/work/sl/work-archive/work/276/DD_SOS2016031422110816.xml >> at >> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:292) >> at >> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277) >> at >> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165) >> at >> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:79) >> at >> org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) >> at >> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141) >> at >> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) >> at >> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460) >> at >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190) >> at org.apache.camel.processor.Pipeline.process(Pipeline.java:121) >> at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) >> at >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190) >> at >> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:442) >> at >> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:214) >> at >> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:178) >> at >> org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174) >> at >> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101) >> at >> org.apache.camel.pollconsumer.quartz2.QuartzScheduledPollConsumerJob.execute(QuartzScheduledPollConsumerJob.java:59) >> at org.quartz.core.JobRunShell.run(JobRunShell.java:202) >> at >> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> va:573) Caused by: org.apache.camel.InvalidPayloadException: No body >> available of type: java.io.InputStream but has value: >> GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml] of type: >> org.apache.camel.component.file.GenericFile on: DD_SOS2016031422110816.xml. >> Caused by: Error during type conversion from type: java.lang.String to the >> required type: byte[] with value [Body is file based: >> GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]] due >> java.io.FileNotFoundException: >> /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or >> directory). >> Exchange[ID-slesb101-60348-1457682250340-0-642502][DD_SOS2016031422110816.xml]. >> Caused by: [org.apache.camel.TypeConversionException - Error during type >> conversion from type: java.lang.String to the required type: byte[] with >> value [Body is file based: >> GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]] due >> java.io.FileNotFoundException: >> /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory)] >> at >> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101) >> at >> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:273) >> ... 19 more >> Caused by: org.apache.camel.TypeConversionException: Error during type >> conversion from type: java.lang.String to the required type: byte[] with >> value [Body is file based: >> GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]] due >> java.io.FileNotFoundException: >> /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory) >> at >> org.apache.camel.impl.converter.BaseTypeConverterRegistry.createTypeConversionException(BaseTypeConverterRegistry.java:610) >> at >> org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:177) >> at >> org.apache.camel.component.file.FileBinding.loadContent(FileBinding.java:57) >> at >> org.apache.camel.component.file.GenericFileConverter.genericFileToInputStream(GenericFileConverter.java:123) >> at sun.reflect.GeneratedMethodAccessor559.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243) >> at >> org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59) >> at >> org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:293) >> at >> org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168) >> at >> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:99) >> ... 20 more >> Caused by: org.apache.camel.RuntimeCamelException: >> java.io.FileNotFoundException: >> /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory) >> at >> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1642) >> at >> org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1247) >> at >> org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59) >> at >> org.apache.camel.component.file.GenericFileConverter.convertTo(GenericFileConverter.java:97) >> at sun.reflect.GeneratedMethodAccessor555.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243) >> at >> org.apache.camel.impl.converter.StaticMethodFallbackTypeConverter.convertTo(StaticMethodFallbackTypeConverter.java:62) >> at >> org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:333) >> at >> org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168) >> ... 30 more >> Caused by: java.io.FileNotFoundException: >> /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory) >> at java.io.FileInputStream.open0(Native Method) >> at java.io.FileInputStream.open(FileInputStream.java:195) >> at java.io.FileInputStream.<init>(FileInputStream.java:138) >> at >> org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:78) >> at >> org.apache.camel.converter.IOConverter.toByteArray(IOConverter.java:266) >> at sun.reflect.GeneratedMethodAccessor604.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243) >> ... 39 more > > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
