Dear James, Can you specify the system you are working on, e.g. hardware/os.
You are really beating the system. I don't know a system that can handle a TPC-H SF100 out of the box. This size often require a multistage process and careful setting of the system parameters. But, let's see what you have done and learn from your experience. James Laken wrote: > Dear MonetDB Developers and Users, > > My TPC-H SF100 test is still producing interesting problems. After > several hours of work I have managed to import near 420 million record > to the lineitem table (7x 60 million record slice). Accidentally I Ok, you perform a sliced base load. i.e. 7 x a SF-10 > killed the import process, and stopped the server process. I have tried Did you stop and restart the server between the loads? If not, then from a recovery point of view all 420M are stored in a single log file and become the target of a single reload. It behaves as if you loaded the 7x60M as a single batch. Killing a database process is of course hard. In that case, the recovery process has to reload the data and enters a really expensive part of TPC-H: ensure correctness of the integrity relationships. Protection against this is hard, because it requires that integrity rules enforcement should either be disabled (the method persued in MySQL). > to restart the server process but after three hours of intensive > processing the sql module still not started. Please note that the > initialization process allocated nearly all memory and swap. This is what we expect. Your tables require a lot of space, because MonetDB does not automatically partitioning it. (That's scheduled for an upcoming release ;-)) > > I have attached a gdb to the server process and the execution stacks > looks like this: > > Program received signal SIGINT, Interrupt. > 0x00002b240cb1c2a4 in file_read () from /usr/lib/libstream.so.0 > (gdb) where > #0 0x00002b240cb1c2a4 in file_read () from /usr/lib/libstream.so.0 > #1 0x00002b240cb1b744 in stream_readLngArray () from > /usr/lib/libstream.so.0 > #2 0x00002b240c67a27f in lngRead () from /usr/lib/libbat.so.0 > #3 0x00002b240c788e90 in logger_readlog () from /usr/lib/libbat.so.0 > #4 0x00002b240c789b0c in logger_create () from /usr/lib/libbat.so.0 > #5 0x00002aaaaab42658 in store_init () from > /usr/lib/MonetDB5/lib/lib_sql.so > #6 0x00002aaaaab179b3 in mvc_init () from /usr/lib/MonetDB5/lib/lib_sql.so > #7 0x00002aaaaaacf9f4 in SQLinit () from /usr/lib/MonetDB5/lib/lib_sql.so > #8 0x00002b240bf89e3a in initScenario () from /usr/lib/libmal.so.0 > #9 0x00002aaaaaacf968 in SQLsession () from > /usr/lib/MonetDB5/lib/lib_sql.so > #10 0x00002b240bf6536e in runMALsequence () from /usr/lib/libmal.so.0 > #11 0x00002b240bf6697b in runMAL () from /usr/lib/libmal.so.0 > #12 0x00002b240bf5f3f3 in MALengine () from /usr/lib/libmal.so.0 > #13 0x00002b240bf5e3ab in callString () from /usr/lib/libmal.so.0 > #14 0x0000000000402a65 in main () > (gdb) c > Continuing. > > Program received signal SIGINT, Interrupt. > 0x00002b240d6c31a0 in malloc () from /lib/libc.so.6 > (gdb) where > #0 0x00002b240d6c31a0 in malloc () from /lib/libc.so.6 > #1 0x00002b240c675c04 in GDKmallocmax () from /usr/lib/libbat.so.0 > #2 0x00002b240c675da9 in GDKmalloc () from /usr/lib/libbat.so.0 > #3 0x00002b240c6787ee in strRead () from /usr/lib/libbat.so.0 > #4 0x00002b240c788e90 in logger_readlog () from /usr/lib/libbat.so.0 > #5 0x00002b240c789b0c in logger_create () from /usr/lib/libbat.so.0 > #6 0x00002aaaaab42658 in store_init () from > /usr/lib/MonetDB5/lib/lib_sql.so > #7 0x00002aaaaab179b3 in mvc_init () from /usr/lib/MonetDB5/lib/lib_sql.so > #8 0x00002aaaaaacf9f4 in SQLinit () from /usr/lib/MonetDB5/lib/lib_sql.so > #9 0x00002b240bf89e3a in initScenario () from /usr/lib/libmal.so.0 > #10 0x00002aaaaaacf968 in SQLsession () from > /usr/lib/MonetDB5/lib/lib_sql.so > #11 0x00002b240bf6536e in runMALsequence () from /usr/lib/libmal.so.0 > #12 0x00002b240bf6697b in runMAL () from /usr/lib/libmal.so.0 > #13 0x00002b240bf5f3f3 in MALengine () from /usr/lib/libmal.so.0 > #14 0x00002b240bf5e3ab in callString () from /usr/lib/libmal.so.0 > #15 0x0000000000402a65 in main () > > Any idea? > > Regards, > J. > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
