Hi Joshua,
On 08/01/15 23:50, Joshua Dunham wrote:
When using the kiwiLoader, I’m finding it to be extremely slow. I started out
with a **very** large file which didn’t finish after 2 months and trying a
800KB file took 3mins 45s to import.
Has anyone seen similar performance and / or have tips to speed things up? My
exact command is,
java -cp "mysql-connector-java-5.1.34.jar:marmotta-loader-kiwi-3.3.0-SNAPSHOT.jar"
org.apache.marmotta.loader.core.MarmottaLoader -f /app/data/taxonomies/uniprot/enzyme.rdf.gz -z -c
"http://marmottaServer:8080/marmotta/context/uniprot"; -C
"jdbc:mysql://mySQLserver:3306/database?zeroDateTimeBehavior=round&useUnicode=true&characterEncoding=UTF-8"
-U dbuser -P dbpass
I have to admit I have no idea about the performance of the KiWi loader
on MySQL. But those figures are very very poor, I agree.
We always use PostgreSQL on production environment and is the database
more optimized. For instance, loading large datasets (2 billion triples)
we get an average of about 7.000 triples/sec.
Who is using MySQL out there? Because at some point we'ĺl have to
discuss in the community how to maintain that implementation.
Anyway, things like generally help:
* Performance hardware, memory is less important than a fast SSD disk
* Dropping indexes (using the '-I' option) gets something extra (it's
cheaper to re-build the indexes than the overhead on each insertion)
* Database general tuning (memory, buffers, etc)
Some of then are general common sense already described at
http://wiki.apache.org/marmotta/PerformanceTuning
Since this had been running for a while I was on a 330 snapshot that was
snagged from git at that time.
I don't think there are significant improvements in the release than in
the snapshopt from the previous weeks... anyway is always recommended to
use the release.
Hope that helps.
Cheers,
--
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 660 2747 925
e: sergio.fernan...@redlink.co
w: http://redlink.co
