Alright writing is pretty slow. Writing 10million records take two hours. But reading is pretty darn quick. Iterating through 10million rows and dumping each row to stdout (redirected to /dev/null) takes about 1 minute +- 5 seconds. Throughout which memory use is constant and low: less than 1.5%. That's got to be quicker than anything using jdbc which is obliged to push everything up and down the tcp/ip stack.
Not looking any more at this now - just wanted to see the numbers. Looks like a good addition to Knut's student project list. Refactor dhis14 file import to inject a jackcess backend in place of the hibernate one. Cheers Bob 2010/7/23 Bob Jolliffe <bobjolli...@gmail.com>: > 2010/7/22 Lars Helge Øverland <larshe...@gmail.com>: >> >> No doubt this looks much simpler. >> Would be interesting to do a test with a large table (>10 mill) and see how >> it performs in terms of memory usage. > > 10 million records is a lot of records! I have it whirring away in > the background as I get on with other stuff. Started 1.5 hours ago > and still writing records ... db file up to 250M ... hasn't started > reading back yet but thus far memory usage is low and constant. I'll > let you know when/if it finishes :-) > >> Lars >> On Wed, Jul 21, 2010 at 3:42 AM, Ime Asangansi <asanga...@yahoo.com> wrote: >>> >>> Impressive! >>> First time seeing that clean functionality! >>> I see potential there to move data between both systems :) >>> >>> Ime >>> >>> >>> --- On Tue, 7/20/10, Knut Staring <knu...@gmail.com> wrote: >>> >>> > From: Knut Staring <knu...@gmail.com> >>> > Subject: Re: [Dhis2-devs] dhis14 import >>> > To: "Bob Jolliffe" <bobjolli...@gmail.com> >>> > Cc: "dhis2-devs" <dhis2-devs@lists.launchpad.net> >>> > Date: Tuesday, July 20, 2010, 3:32 PM >>> > That sounds really great - it has >>> > been problematic to require Windows for this. >>> > >>> > k >>> > >>> > On Tue, Jul 20, 2010 at 3:23 PM, Bob Jolliffe <bobjolli...@gmail.com> >>> > wrote: >>> > > Just some throwaway code testing out jackcess for >>> > reading dhis14 (and >>> > > potentially modulo basico files): >>> > > >>> > > http://pastebin.com/wMv1SZqq >>> > > >>> > > I'm pretty impressed. It works well and I suspect >>> > also much faster >>> > > than accessing via odbc/ibatis or whatever it is. >>> > Never mind the >>> > > nonsense of what this code actually does - the point >>> > is that it can >>> > > iterate over access tables using java (on ubuntu). >>> > Kind of nice. >>> > > >>> > > Cheers >>> > > Bob >>> > > >>> > > _______________________________________________ >>> > > Mailing list: https://launchpad.net/~dhis2-devs >>> > > Post to : dhis2-devs@lists.launchpad.net >>> > > Unsubscribe : https://launchpad.net/~dhis2-devs >>> > > More help : https://help.launchpad.net/ListHelp >>> > > >>> > >>> > >>> > >>> > -- >>> > Cheers, >>> > Knut Staring >>> > >>> > _______________________________________________ >>> > Mailing list: https://launchpad.net/~dhis2-devs >>> > Post to : dhis2-devs@lists.launchpad.net >>> > Unsubscribe : https://launchpad.net/~dhis2-devs >>> > More help : https://help.launchpad.net/ListHelp >>> > >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~dhis2-devs >>> Post to : dhis2-devs@lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~dhis2-devs >>> More help : https://help.launchpad.net/ListHelp >> >> > _______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
