Hello. In cases that produce about one million lines, grouping lines by 1000 lines lets the file opened and closed 1000 times. Perhaps this is an acceptable performance price...
As an idealist I am, the best solution should be to open the file once and close it at the end of the main exchange when all the output is done. This seems to be what is intended with the new stream options (closeOnDone) in Camel 2.11, but is not available in our customer version. Ephemeris Lappis Le 02/05/2014 09:15, Claus Ibsen-2 [via Camel] a écrit : > Hi > > I wonder if using groupLines 1000 etc to work on a bulk of lines at a > time, wont be fast enough with the fileExists=Append mode. > > > > On Thu, May 1, 2014 at 5:17 PM, Ephemeris Lappis > <[hidden email] </user/SendEmail.jtp?type=node&node=5750772&i=0>> wrote: > > > Hello. > > > > We have to produce some rather big volumes of data and generate output > > files, in several steps, some of these steps using splitting to process > > inputs. I've been looking for two ways... > > > > The first way is writing a file, setting an "append" mode, that > could be a > > nice solution to write batches of lines, but I suppose that the file if > > closed and reopened for each exchange, and thus may be a quite bad > solution > > when writing millions of line one by one from a splitting loop. > > > > The second way is to use a stream, using "stream:file", but I've not > found > > any way to control actually how the file is closed. FYI, we use > Camel with > > Service Mix with an emdedded 2.10, and the "closeOnDone" option is not > > available. The "autoCloseCount" seemed to be the beginning of a > solution, > > but as the value can't be set dynamically, using a property for > example, > > this doesn't give a full control on the file closing, and outputs > might stay > > opened, and make fail the following tasks. > > > > Any idea to write big files and control how to close them ? > > > > Thanks in advance. > > > > Regards. > > > > > > > > -- > > View this message in context: > http://camel.465427.n5.nabble.com/Writing-big-files-stream-or-file-tp5750742.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > > -- > Claus Ibsen > ----------------- > Red Hat, Inc. > Email: [hidden email] </user/SendEmail.jtp?type=node&node=5750772&i=1> > Twitter: davsclaus > Blog: http://davsclaus.com > Author of Camel in Action: http://www.manning.com/ibsen > hawtio: http://hawt.io/ > fabric8: http://fabric8.io/ > > > ------------------------------------------------------------------------ > If you reply to this email, your message will be added to the > discussion below: > http://camel.465427.n5.nabble.com/Writing-big-files-stream-or-file-tp5750742p5750772.html > > > To unsubscribe from Writing big files : stream or file ?, click here > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5750742&code=ZXBoZW1lcmlzLmxhcHBpc0BnbWFpbC5jb218NTc1MDc0Mnw0OTQyMjM2NDI=>. > NAML > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > -- View this message in context: http://camel.465427.n5.nabble.com/Writing-big-files-stream-or-file-tp5750742p5750773.html Sent from the Camel - Users mailing list archive at Nabble.com.
