Hi Bob, Mmmm..interesting.
Just to be sure. Were you reading from 1.4 and writing to 2.0? mysql or postgres? Were you writing the other way (to access)? Thanks. Ime --- On Fri, 7/23/10, Bob Jolliffe <bobjolli...@gmail.com> wrote: > From: Bob Jolliffe <bobjolli...@gmail.com> > Subject: Re: [Dhis2-devs] dhis14 import > To: "Lars Helge Øverland" <larshe...@gmail.com> > Cc: "Ime Asangansi" <asanga...@yahoo.com>, "Knut Staring" <knu...@gmail.com>, > "dhis2-devs" <dhis2-devs@lists.launchpad.net> > Date: Friday, July 23, 2010, 3:53 PM > Alright writing is pretty slow. > Writing 10million records take two hours. > > But reading is pretty darn quick. Iterating through > 10million rows > and dumping each row to stdout (redirected to /dev/null) > takes about 1 > minute +- 5 seconds. Throughout which > memory use is constant and > low: less than 1.5%. That's got to be quicker than > anything using > jdbc which is obliged to push everything up and down the > tcp/ip stack. > > Not looking any more at this now - just wanted to see the > numbers. > Looks like a good addition to Knut's student project > list. Refactor > dhis14 file import to inject a jackcess backend in place of > the > hibernate one. > > Cheers > Bob > > 2010/7/23 Bob Jolliffe <bobjolli...@gmail.com>: > > 2010/7/22 Lars Helge Øverland <larshe...@gmail.com>: > >> > >> No doubt this looks much simpler. > >> Would be interesting to do a test with a large > table (>10 mill) and see how > >> it performs in terms of memory usage. > > > > 10 million records is a lot of records! I have it > whirring away in > > the background as I get on with other stuff. Started > 1.5 hours ago > > and still writing records ... db file up to 250M ... > hasn't started > > reading back yet but thus far memory usage is low and > constant. I'll > > let you know when/if it finishes :-) > > > >> Lars > >> On Wed, Jul 21, 2010 at 3:42 AM, Ime Asangansi > <asanga...@yahoo.com> > wrote: > >>> > >>> Impressive! > >>> First time seeing that clean functionality! > >>> I see potential there to move data between > both systems :) > >>> > >>> Ime > >>> > >>> > >>> --- On Tue, 7/20/10, Knut Staring <knu...@gmail.com> > wrote: > >>> > >>> > From: Knut Staring <knu...@gmail.com> > >>> > Subject: Re: [Dhis2-devs] dhis14 import > >>> > To: "Bob Jolliffe" <bobjolli...@gmail.com> > >>> > Cc: "dhis2-devs" <dhis2-devs@lists.launchpad.net> > >>> > Date: Tuesday, July 20, 2010, 3:32 PM > >>> > That sounds really great - it has > >>> > been problematic to require Windows for > this. > >>> > > >>> > k > >>> > > >>> > On Tue, Jul 20, 2010 at 3:23 PM, Bob > Jolliffe <bobjolli...@gmail.com> > >>> > wrote: > >>> > > Just some throwaway code testing out > jackcess for > >>> > reading dhis14 (and > >>> > > potentially modulo basico files): > >>> > > > >>> > > http://pastebin.com/wMv1SZqq > >>> > > > >>> > > I'm pretty impressed. It works > well and I suspect > >>> > also much faster > >>> > > than accessing via odbc/ibatis or > whatever it is. > >>> > Never mind the > >>> > > nonsense of what this code actually > does - the point > >>> > is that it can > >>> > > iterate over access tables using > java (on ubuntu). > >>> > Kind of nice. > >>> > > > >>> > > Cheers > >>> > > Bob > >>> > > > >>> > > > _______________________________________________ > >>> > > Mailing list: https://launchpad.net/~dhis2-devs > >>> > > Post to : dhis2-devs@lists.launchpad.net > >>> > > Unsubscribe : https://launchpad.net/~dhis2-devs > >>> > > More help : https://help.launchpad.net/ListHelp > >>> > > > >>> > > >>> > > >>> > > >>> > -- > >>> > Cheers, > >>> > Knut Staring > >>> > > >>> > > _______________________________________________ > >>> > Mailing list: https://launchpad.net/~dhis2-devs > >>> > Post to : dhis2-devs@lists.launchpad.net > >>> > Unsubscribe : https://launchpad.net/~dhis2-devs > >>> > More help : https://help.launchpad.net/ListHelp > >>> > > >>> > >>> > _______________________________________________ > >>> Mailing list: https://launchpad.net/~dhis2-devs > >>> Post to : dhis2-devs@lists.launchpad.net > >>> Unsubscribe : https://launchpad.net/~dhis2-devs > >>> More help : https://help.launchpad.net/ListHelp > >> > >> > > > _______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
