Yeah The other sql components are likely a bit better at this, such as mybatis or the sql component. For the latter you can set useIterator=false to get the List. And you can set that maxMessagesPerPoll to 10 but that is after the poll, so try to set it on the jdbc driver, eg on template.maxRows=10 for SQL. MyBatis ought to have something as well to configure a max limit to the jdbc driver.
The JPA component was originally not intended for arbitrary SQL and thus was a 1:1 jpa entity/table thingy, and it could use a bit of love to make the consumer to poll X together. You are welcome to log a JIRA On Fri, Mar 24, 2017 at 1:02 PM, dvlato <[email protected]> wrote: > Hello, > I have just started working with Camel and I have what I believe is a pretty > common requirement: poll the database to fetch all the records with a > certain status, and process them in chunks (let's 10 rows at a time). I > though I could leverage the existing components to do this easily but I am > not finding the correct way: > > What I have tried so far is to usa the JPA component (as we already have JPA > entities in place) to poll the database and use the @Consumed annotation to > change the status of the record (some pointers about how to deal with > transactionality are also most welcome). The problem with this approach is: > 1) Here the rows are received one by one, and we can later merge them with > an aggregator, which seems suboptimal. > 2) Even if we ignored the performance problem with that approach, I'm not > sure of how to configure the aggregator to have fixed size blocks but taking > into account the whole size of the batch (I mean, if after chunking we only > have 5 rows in a message, just return that instead of waiting for the 10th - > maybe it works like that by default - I don't really know). > > I've seen that there is a class called "JpaPollingConsumer" which returns > all the rows instead of one by one, but I don't know if there is anyway to > use it with a "from()" clause, it seems that Camel calls "createConsumer" > and not "createPollingConsumer", right? Is there a way to use > from("jpa://...") and have Camel use the polling consumer? I have tried > using pollEnrich but in that case the @Consumed code is not executed. Is > there any other alternative? > > > I apologise for such basic questions, I am sure I could probably figure out > a "good enough" approach using my inexistent knowledge of Camel (we have > Camel in Action which has been great so far, but I don't see anything in > Chapter 6 regarding database polling) but, on the other hand, it's such a > common task that someone might have best practices already available. I > have browsed the myBatis and SQL docs and I think that they might be better > suited (if I am not using JPA incorrectly, which is quite likely the case), > but I will have the same problem of not knowing if I am using Camel > correctly. > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Best-way-to-poll-a-database-and-process-in-chunks-tp5796094.html > Sent from the Camel - Users mailing list archive at Nabble.com. -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
