thnx for your reply, can you give me an example, that opens only one connection while paginating resultset
(maybe because of alotof connections, it does not work ...) regards cem chrislusf wrote: > > This is not a lucene question, but a jdbc question. > The code is not releasing the jdbc connection, statement, and resultset, > and > what's worse, the code is creating new connections when paginating the > results. > > -- > Chris Lu > ------------------------- > Instant Scalable Full-Text Search On Any Database/Application > site: http://www.dbsight.net > demo: http://search.dbsight.com > Lucene Database Search in 3 minutes: > http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes > DBSight customer, a shopping comparison site, (anonymous per request) got > 2.6 Million Euro funding! > > On Wed, Jan 21, 2009 at 1:29 PM, cemsoft <bcemku...@yahoo.com> wrote: > >> >> hi, >> >> i am trying to index a database table with appr. 1,5 million rows.. >> first i got OutOfMemory exception, then i used offsets, it works now till >> 50000 rows >> >> lucene is new to me ..should i set sth else ?? below is the code that i >> use >> >> private static void indexData(MysqlDataSource ds) throws SQLException, >> Exception { >> JdbcDirectory jdbcDir = new JdbcDirectory(ds, new >> MySQLDialect(), "indexTable"); >> jdbcDir.create(); // creates the indexTable in the DB >> (test). No need to >> // create it >> manually >> StandardAnalyzer analyzer = new StandardAnalyzer(); >> >> int offset=0; >> int rowCount = 5000; >> >> IndexWriter writer = new IndexWriter(jdbcDir, analyzer, >> true, >> MaxFieldLength.UNLIMITED); >> try{ >> Connection conn = ds.getConnection(); >> while (offset<1244800){ >> indexDocs(writer, conn, offset, rowCount); >> offset += rowCount; >> } >> System.out.println("Optimizing..."); >> writer.optimize(); >> writer.close(); >> }catch (Exception e) { >> System.out.println("exception " + e.getMessage()); >> } >> >> } >> >> static void indexDocs(IndexWriter writer, Connection conn, int >> offset, int >> rowCount) throws Exception { >> >> String sql = "select id, content from >> monitoring_dk_test.entry limit "+ >> offset + ","+ rowCount+";"; >> System.out.println(sql); >> ResultSet rs = conn.createStatement().executeQuery(sql); >> while (rs.next()) { >> Document d = new Document(); >> d.add(new Field("id", rs.getString("id"), >> Field.Store.YES, >> Field.Index.NO)); >> d.add(new Field("content", >> rs.getString("content"), >> Field.Store.YES, >> Field.Index.TOKENIZED)); >> writer.addDocument(d); >> } >> System.out.println("indexing finished ..."); >> } >> >> >> >> best regards >> cem >> -- >> View this message in context: >> http://www.nabble.com/indexing-database-tp21592598p21592598.html >> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> > > -- View this message in context: http://www.nabble.com/indexing-database-tp21592598p21600256.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
