This is not a lucene question, but a jdbc question. The code is not releasing the jdbc connection, statement, and resultset, and what's worse, the code is creating new connections when paginating the results.
-- Chris Lu ------------------------- Instant Scalable Full-Text Search On Any Database/Application site: http://www.dbsight.net demo: http://search.dbsight.com Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes DBSight customer, a shopping comparison site, (anonymous per request) got 2.6 Million Euro funding! On Wed, Jan 21, 2009 at 1:29 PM, cemsoft <bcemku...@yahoo.com> wrote: > > hi, > > i am trying to index a database table with appr. 1,5 million rows.. > first i got OutOfMemory exception, then i used offsets, it works now till > 50000 rows > > lucene is new to me ..should i set sth else ?? below is the code that i use > > private static void indexData(MysqlDataSource ds) throws SQLException, > Exception { > JdbcDirectory jdbcDir = new JdbcDirectory(ds, new > MySQLDialect(), "indexTable"); > jdbcDir.create(); // creates the indexTable in the DB > (test). No need to > // create it > manually > StandardAnalyzer analyzer = new StandardAnalyzer(); > > int offset=0; > int rowCount = 5000; > > IndexWriter writer = new IndexWriter(jdbcDir, analyzer, > true, > MaxFieldLength.UNLIMITED); > try{ > Connection conn = ds.getConnection(); > while (offset<1244800){ > indexDocs(writer, conn, offset, rowCount); > offset += rowCount; > } > System.out.println("Optimizing..."); > writer.optimize(); > writer.close(); > }catch (Exception e) { > System.out.println("exception " + e.getMessage()); > } > > } > > static void indexDocs(IndexWriter writer, Connection conn, int > offset, int > rowCount) throws Exception { > > String sql = "select id, content from > monitoring_dk_test.entry limit "+ > offset + ","+ rowCount+";"; > System.out.println(sql); > ResultSet rs = conn.createStatement().executeQuery(sql); > while (rs.next()) { > Document d = new Document(); > d.add(new Field("id", rs.getString("id"), > Field.Store.YES, > Field.Index.NO)); > d.add(new Field("content", rs.getString("content"), > Field.Store.YES, > Field.Index.TOKENIZED)); > writer.addDocument(d); > } > System.out.println("indexing finished ..."); > } > > > > best regards > cem > -- > View this message in context: > http://www.nabble.com/indexing-database-tp21592598p21592598.html > Sent from the Lucene - Java Users mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
