2010/7/23 Bob Jolliffe <bobjolli...@gmail.com> > Alright writing is pretty slow. Writing 10million records take two hours. > > But reading is pretty darn quick. Iterating through 10million rows > and dumping each row to stdout (redirected to /dev/null) takes about 1 > minute +- 5 seconds. Throughout which memory use is constant and > low: less than 1.5%. That's got to be quicker than anything using > jdbc which is obliged to push everything up and down the tcp/ip stack. > > Not looking any more at this now - just wanted to see the numbers. > Looks like a good addition to Knut's student project list. Refactor > dhis14 file import to inject a jackcess backend in place of the > hibernate one. > > Cheers > Bob >
Thanks for testing this. Looks promising. Actually what I was most curious about was how data is fetched from such large tables. By doing a "select * from DataValue" on a 10 mill table we will run out of memory one way or the other. Doing a "select * from DataValue order by DataValueID limit 0, 10000" and then increase the limit takes a long time due to the ordering. Ibatis (what we currenty use) has a RowHandler interface which deals with this and lets you handle each row individually with apparently quite effective memory management (I don't know how this is implemented though.) > > 2010/7/23 Bob Jolliffe <bobjolli...@gmail.com>: > > 2010/7/22 Lars Helge Øverland <larshe...@gmail.com>: > >> > >> No doubt this looks much simpler. > >> Would be interesting to do a test with a large table (>10 mill) and see > how > >> it performs in terms of memory usage. > > > > 10 million records is a lot of records! I have it whirring away in > > the background as I get on with other stuff. Started 1.5 hours ago > > and still writing records ... db file up to 250M ... hasn't started > > reading back yet but thus far memory usage is low and constant. I'll > > let you know when/if it finishes :-) > > > >> Lars > >> On Wed, Jul 21, 2010 at 3:42 AM, Ime Asangansi <asanga...@yahoo.com> > wrote: > >>> > >>> Impressive! > >>> First time seeing that clean functionality! > >>> I see potential there to move data between both systems :) > >>> > >>> Ime > >>> > >>> > >>> --- On Tue, 7/20/10, Knut Staring <knu...@gmail.com> wrote: > >>> > >>> > From: Knut Staring <knu...@gmail.com> > >>> > Subject: Re: [Dhis2-devs] dhis14 import > >>> > To: "Bob Jolliffe" <bobjolli...@gmail.com> > >>> > Cc: "dhis2-devs" <dhis2-devs@lists.launchpad.net> > >>> > Date: Tuesday, July 20, 2010, 3:32 PM > >>> > That sounds really great - it has > >>> > been problematic to require Windows for this. > >>> > > >>> > k > >>> > > >>> > On Tue, Jul 20, 2010 at 3:23 PM, Bob Jolliffe <bobjolli...@gmail.com > > > >>> > wrote: > >>> > > Just some throwaway code testing out jackcess for > >>> > reading dhis14 (and > >>> > > potentially modulo basico files): > >>> > > > >>> > > http://pastebin.com/wMv1SZqq > >>> > > > >>> > > I'm pretty impressed. It works well and I suspect > >>> > also much faster > >>> > > than accessing via odbc/ibatis or whatever it is. > >>> > Never mind the > >>> > > nonsense of what this code actually does - the point > >>> > is that it can > >>> > > iterate over access tables using java (on ubuntu). > >>> > Kind of nice. > >>> > > > >>> > > Cheers > >>> > > Bob > >>> > > > >>> > > _______________________________________________ > >>> > > Mailing list: https://launchpad.net/~dhis2-devs > >>> > > Post to : dhis2-devs@lists.launchpad.net > >>> > > Unsubscribe : https://launchpad.net/~dhis2-devs > >>> > > More help : https://help.launchpad.net/ListHelp > >>> > > > >>> > > >>> > > >>> > > >>> > -- > >>> > Cheers, > >>> > Knut Staring > >>> > > >>> > _______________________________________________ > >>> > Mailing list: https://launchpad.net/~dhis2-devs > >>> > Post to : dhis2-devs@lists.launchpad.net > >>> > Unsubscribe : https://launchpad.net/~dhis2-devs > >>> > More help : https://help.launchpad.net/ListHelp > >>> > > >>> > >>> _______________________________________________ > >>> Mailing list: https://launchpad.net/~dhis2-devs > >>> Post to : dhis2-devs@lists.launchpad.net > >>> Unsubscribe : https://launchpad.net/~dhis2-devs > >>> More help : https://help.launchpad.net/ListHelp > >> > >> > > >
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
