Thank you for your reply,it is very import to me.
1.I agree with you by i read solr's source code,i found that it can resolve
this problem by config db-data-config.xml,like this(my database's
Sqlserver2005,other database will unavailable):
<dataSource name="dsSqlServer" type="JdbcDataSource"
driver="com.microsoft.sqlserver.jdbc.SQLServerDriver" batchSize="3000"
url="jdbc:sqlserver://192.168.1.5:1433;
DatabaseName=testDatabase;responseBuffering=adaptive;selectMethod=cursor"
user="sa" password="12345" />
add responseBuffering=adaptive;selectMethod=cursor into url attribute,and Solr
will set these parameters by itself:
c.createStatement(ResultSet.TYPE_FORWARD_ONLY,ResultSet.CONCUR_READ_ONLY);
By these configs,Solr can import big table's datas to index dir.
2.But there are some problems:
if the table is very big,solr will spend a long time to import and index,may be
one day and more.so once occurred network problems and others during this
time,maybe solr can not remember what documents had processed,and if we
continue data import ,we do not know where to start.
3.i am sorry for my bad English.i wish you can know what i mean.
2010-09-08
郭芸
发件人： Alexey Serba
发送时间： 2010-09-07 16:07:49
收件人： dev
抄送：
主题： Re: About Solr DataImportHandler
> i found that Solr import the datas to memory first,then write them to index
> dir.
That's not really true. DataImportHandler streams the result from
database query and adding documents into index. So it shouldn't load
all database data into memory. Disabling autoCommit, warming queries
and spellcheckers usually decreases required amount of memory during
indexing process.
Please share your hardware details, jvm options, solrconfig and schema
configuration, etc.
2010/9/7 郭芸 <[email protected]>:
> Dear all:
> I use Solr DataImportHandler's JdbcDataSource to import the Sqlsever 2005's
> datas to Solr,but My table is versy big,about 300G.and i found that Solr
> import the datas to memory first,then write them to index dir.So if the
> datas are too big,there will trigger an OutOfMemoryException.
> I want to solve this problem,and how can ti do it?anybody can help me?Thank
> you.
>
> 2010-09-07
> ________________________________
> 郭芸
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
