Hi Ime On 23 July 2010 19:23, Ime Asangansi <asanga...@yahoo.com> wrote: > Hi Bob, > > Mmmm..interesting. > > Just to be sure. > Were you reading from 1.4 and writing to 2.0? mysql or postgres? > > Were you writing the other way (to access)?
Nothing so grand as any of the above. Though I am reading and writing
access. Reading/writing mysql and postgres will be much more of a
bottleneck than reading jackcess 'coz of the tcp requirement of jdbc.
In this scenario I'm just running scratch code in the background to
keep my laptop warm while I try to write :-) See attached. jackcess
is very maven friendly.
mvn assembly:assembly
java -jar target/dhaccess-jar-with-dendencies.jar
Drink lots of coffee, take a walk, have lunch .... etc
Cheers.
Bob
package org.hisp.dhis;
import com.healthmarketscience.jackcess.ColumnBuilder;
import com.healthmarketscience.jackcess.DataType;
import java.io.File;
import java.io.IOException;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.healthmarketscience.jackcess.Database;
import com.healthmarketscience.jackcess.Table;
import com.healthmarketscience.jackcess.TableBuilder;
/**
* CPU warmer
*
*/
public class App
{
private static final Log log = LogFactory.getLog( App.class );
private static long MAXROWS = 10000000;
public static void main( String[] args )
{
try
{
File dbFile = new File( "/home/bobj/src/dhaccess/test.mdb" );
Database db = Database.create( dbFile );
log.info( "Opened " + dbFile.getName() );
Map<String, Object> row = null;
Table table = new TableBuilder( "test" ).addColumn( new
ColumnBuilder( "id", DataType.LONG ).toColumn() ).addColumn( new
ColumnBuilder( "value", DataType.TEXT ).toColumn() ).toTable( db );
long i = 0;
for ( i = 0; i < MAXROWS; i++ )
{
table.addRow( i, "some text" );
}
log.info( "Reading data" );
int count = 0;
while ( ( row = table.getNextRow() ) != null )
{
count++;
log.debug( row );
Integer id = (Integer) row.get( "id" );
String value = (String) row.get( "value" );
System.err.println( id + " : " + value );
}
log.info( count + " datavalues" );
} catch ( IOException ex )
{
log.info("Ouch: "+ex);
}
}
}
>
> Thanks.
>
> Ime
>
>
> --- On Fri, 7/23/10, Bob Jolliffe <bobjolli...@gmail.com> wrote:
>
>> From: Bob Jolliffe <bobjolli...@gmail.com>
>> Subject: Re: [Dhis2-devs] dhis14 import
>> To: "Lars Helge Øverland" <larshe...@gmail.com>
>> Cc: "Ime Asangansi" <asanga...@yahoo.com>, "Knut Staring"
>> <knu...@gmail.com>, "dhis2-devs" <dhis2-devs@lists.launchpad.net>
>> Date: Friday, July 23, 2010, 3:53 PM
>> Alright writing is pretty slow.
>> Writing 10million records take two hours.
>>
>> But reading is pretty darn quick. Iterating through
>> 10million rows
>> and dumping each row to stdout (redirected to /dev/null)
>> takes about 1
>> minute +- 5 seconds. Throughout which
>> memory use is constant and
>> low: less than 1.5%. That's got to be quicker than
>> anything using
>> jdbc which is obliged to push everything up and down the
>> tcp/ip stack.
>>
>> Not looking any more at this now - just wanted to see the
>> numbers.
>> Looks like a good addition to Knut's student project
>> list. Refactor
>> dhis14 file import to inject a jackcess backend in place of
>> the
>> hibernate one.
>>
>> Cheers
>> Bob
>>
>> 2010/7/23 Bob Jolliffe <bobjolli...@gmail.com>:
>> > 2010/7/22 Lars Helge Øverland <larshe...@gmail.com>:
>> >>
>> >> No doubt this looks much simpler.
>> >> Would be interesting to do a test with a large
>> table (>10 mill) and see how
>> >> it performs in terms of memory usage.
>> >
>> > 10 million records is a lot of records! I have it
>> whirring away in
>> > the background as I get on with other stuff. Started
>> 1.5 hours ago
>> > and still writing records ... db file up to 250M ...
>> hasn't started
>> > reading back yet but thus far memory usage is low and
>> constant. I'll
>> > let you know when/if it finishes :-)
>> >
>> >> Lars
>> >> On Wed, Jul 21, 2010 at 3:42 AM, Ime Asangansi
>> <asanga...@yahoo.com>
>> wrote:
>> >>>
>> >>> Impressive!
>> >>> First time seeing that clean functionality!
>> >>> I see potential there to move data between
>> both systems :)
>> >>>
>> >>> Ime
>> >>>
>> >>>
>> >>> --- On Tue, 7/20/10, Knut Staring <knu...@gmail.com>
>> wrote:
>> >>>
>> >>> > From: Knut Staring <knu...@gmail.com>
>> >>> > Subject: Re: [Dhis2-devs] dhis14 import
>> >>> > To: "Bob Jolliffe" <bobjolli...@gmail.com>
>> >>> > Cc: "dhis2-devs" <dhis2-devs@lists.launchpad.net>
>> >>> > Date: Tuesday, July 20, 2010, 3:32 PM
>> >>> > That sounds really great - it has
>> >>> > been problematic to require Windows for
>> this.
>> >>> >
>> >>> > k
>> >>> >
>> >>> > On Tue, Jul 20, 2010 at 3:23 PM, Bob
>> Jolliffe <bobjolli...@gmail.com>
>> >>> > wrote:
>> >>> > > Just some throwaway code testing out
>> jackcess for
>> >>> > reading dhis14 (and
>> >>> > > potentially modulo basico files):
>> >>> > >
>> >>> > > http://pastebin.com/wMv1SZqq
>> >>> > >
>> >>> > > I'm pretty impressed. It works
>> well and I suspect
>> >>> > also much faster
>> >>> > > than accessing via odbc/ibatis or
>> whatever it is.
>> >>> > Never mind the
>> >>> > > nonsense of what this code actually
>> does - the point
>> >>> > is that it can
>> >>> > > iterate over access tables using
>> java (on ubuntu).
>> >>> > Kind of nice.
>> >>> > >
>> >>> > > Cheers
>> >>> > > Bob
>> >>> > >
>> >>> > >
>> _______________________________________________
>> >>> > > Mailing list: https://launchpad.net/~dhis2-devs
>> >>> > > Post to : dhis2-devs@lists.launchpad.net
>> >>> > > Unsubscribe : https://launchpad.net/~dhis2-devs
>> >>> > > More help : https://help.launchpad.net/ListHelp
>> >>> > >
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Cheers,
>> >>> > Knut Staring
>> >>> >
>> >>> >
>> _______________________________________________
>> >>> > Mailing list: https://launchpad.net/~dhis2-devs
>> >>> > Post to : dhis2-devs@lists.launchpad.net
>> >>> > Unsubscribe : https://launchpad.net/~dhis2-devs
>> >>> > More help : https://help.launchpad.net/ListHelp
>> >>> >
>> >>>
>> >>>
>> _______________________________________________
>> >>> Mailing list: https://launchpad.net/~dhis2-devs
>> >>> Post to : dhis2-devs@lists.launchpad.net
>> >>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> >>> More help : https://help.launchpad.net/ListHelp
>> >>
>> >>
>> >
>>
>
dhaccess.tgz
Description: GNU Zip compressed data
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
