hi,
i am trying to index a database table with appr. 1,5 million rows..
first i got OutOfMemory exception, then i used offsets, it works now till
50000 rows
lucene is new to me ..should i set sth else ?? below is the code that i use
private static void indexData(MysqlDataSource ds) throws SQLException,
Exception {
JdbcDirectory jdbcDir = new JdbcDirectory(ds, new
MySQLDialect(), "indexTable");
jdbcDir.create(); // creates the indexTable in the DB (test).
No need to
// create it manually
StandardAnalyzer analyzer = new StandardAnalyzer();
int offset=0;
int rowCount = 5000;
IndexWriter writer = new IndexWriter(jdbcDir, analyzer, true,
MaxFieldLength.UNLIMITED);
try{
Connection conn = ds.getConnection();
while (offset<1244800){
indexDocs(writer, conn, offset, rowCount);
offset += rowCount;
}
System.out.println("Optimizing...");
writer.optimize();
writer.close();
}catch (Exception e) {
System.out.println("exception " + e.getMessage());
}
}
static void indexDocs(IndexWriter writer, Connection conn, int offset,
int
rowCount) throws Exception {
String sql = "select id, content from monitoring_dk_test.entry
limit "+
offset + ","+ rowCount+";";
System.out.println(sql);
ResultSet rs = conn.createStatement().executeQuery(sql);
while (rs.next()) {
Document d = new Document();
d.add(new Field("id", rs.getString("id"),
Field.Store.YES,
Field.Index.NO));
d.add(new Field("content", rs.getString("content"),
Field.Store.YES,
Field.Index.TOKENIZED));
writer.addDocument(d);
}
System.out.println("indexing finished ...");
}
best regards
cem
--
View this message in context:
http://www.nabble.com/indexing-database-tp21592598p21592598.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
