Hello Claus. I've seen this post before. Using an aggregation indeed helps to reduce the open and close operations overhead, buffering lines before sends them to the file endpoint.
As we use only blueprints in our solution until now, I'd prefer not to use java code. Do you know about predefined aggregation strategies that just group body's objects. In camel 2.11 the abstract class "org.apache.camel.processor.aggregate.AbstractListAggregationStrategy" seems to help to make 'List<V>', but in our ServiceMix we only have a 2.10. Is there any library that provides some simple aggregator "out off the shelves" ? For example List<Object> should be good for our case ! Secondly, we're talking about the file endpoint solution, but what about streams ? According to my tests, streaming the splitted lines to the file doesn't open and close it for each resulting exchange and works rather well, except that the output file is not closed at the end of the main exchange. Some idea on this way ? Thanks for your help. Ephemeris Lappis Le 02/05/2014 06:55, claus.straube [via Camel] a écrit : > Hi, > > have a look at > http://www.catify.com/2012/07/09/parsing-large-files-with-apache-camel/ > - perhaps this helps (it's also about writing big files). > > Best regards - Claus > > On 01.05.2014 17:17, Ephemeris Lappis wrote: > > > Hello. > > > > We have to produce some rather big volumes of data and generate output > > files, in several steps, some of these steps using splitting to process > > inputs. I've been looking for two ways... > > > > The first way is writing a file, setting an "append" mode, that > could be a > > nice solution to write batches of lines, but I suppose that the file if > > closed and reopened for each exchange, and thus may be a quite bad > solution > > when writing millions of line one by one from a splitting loop. > > > > The second way is to use a stream, using "stream:file", but I've not > found > > any way to control actually how the file is closed. FYI, we use > Camel with > > Service Mix with an emdedded 2.10, and the "closeOnDone" option is not > > available. The "autoCloseCount" seemed to be the beginning of a > solution, > > but as the value can't be set dynamically, using a property for > example, > > this doesn't give a full control on the file closing, and outputs > might stay > > opened, and make fail the following tasks. > > > > Any idea to write big files and control how to close them ? > > > > Thanks in advance. > > > > Regards. > > > > > > > > -- > > View this message in context: > http://camel.465427.n5.nabble.com/Writing-big-files-stream-or-file-tp5750742.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > > > ------------------------------------------------------------------------ > If you reply to this email, your message will be added to the > discussion below: > http://camel.465427.n5.nabble.com/Writing-big-files-stream-or-file-tp5750742p5750764.html > > > To unsubscribe from Writing big files : stream or file ?, click here > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5750742&code=ZXBoZW1lcmlzLmxhcHBpc0BnbWFpbC5jb218NTc1MDc0Mnw0OTQyMjM2NDI=>. > NAML > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > -- View this message in context: http://camel.465427.n5.nabble.com/Writing-big-files-stream-or-file-tp5750742p5750771.html Sent from the Camel - Users mailing list archive at Nabble.com.
