Migration to Lucene 6.x

[krish mohan]

Hi
   I'm using Lucene 4.10.4. QueryParser in LUCENE_30 forms phrase query for
input with special characers ($,/,-,...)

   For eg:
 For input *google-chrome-stable*, query formed as "google chrome
stable".

 Using QueryParser(Version matchVersion, String f, Analyzer a)

,
I pass the version as  LUCENE_30 and achieved this behavior.

But in Lucene 6.x this constructor is removed. Is there any way to
achieve this in Lucene 6.x ?

Re: Migration to Lucene 6.x

[krish mohan]

Correction in link. QueryParser(Version matchVersion, String f, Analyzer a)


On Mon, Jul 31, 2017 at 2:50 PM, krish mohan 
wrote:

> Hi
>I'm using Lucene 4.10.4. QueryParser in LUCENE_30 forms phrase query
> for input with special characers ($,/,-,...)
>
>For eg:
>  For input *google-chrome-stable*, query formed as "google chrome
> stable".
>
>  Using QueryParser(Version matchVersion, String f, Analyzer a)
> 
>  ,
> I pass the version as  LUCENE_30 and achieved this behavior.
>
> But in Lucene 6.x this constructor is removed. Is there any way to
> achieve this in Lucene 6.x ?
>

lucene Input and Output format

[Ranganath B N]

Hi All,

 Can you point me to some of the implementations  of lucene Input and 
Output format? I wanted to know them to  understand the distributed 
implementation approach.


Thanks,
Ranganath B. N.

Lucene 6.6: "Too many open files"

[Nawab Zada Asad Iqbal]

Hi,

I am upgrading from solr4.5 to solr6.6 and hitting this issue during
complete reindexing scenario.  We have some batch indexing scripts, which
flood the solr servers with indexing requests (while keeping open-searcher
false) for many hours and then perform one commit. This used to work fine
with 4.5, but with 6.6, i get 'Too many open files' within a couple of
minutes. I have checked that "ulimit" is same between old and new servers.

Has something fundamentally changed in recent lucene versions, which keeps
file descriptors around for a longer time?


Here is a sample error message:
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:763)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3206)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:644)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1894)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1871)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:62)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException:
/local/var/solr/shard2/filesearch/data/index/_34w5.fdx: Too many open files
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixExcepti

Re: Lucene 6.6: "Too many open files"

[Erick Erickson]

No, nothing's changed fundamentally. But you say:

"We have some batch indexing scripts, which
flood the solr servers with indexing requests (while keeping open-searcher
false)"

What is your commit interval? Regardless of whether openSearcher is false
or not, background merging continues apace with every commit. By any chance
did you change your merge policy (or not copy the one from 4x to 6x)? Shot
in the dark...

Best,
Erick

On Mon, Jul 31, 2017 at 7:15 PM, Nawab Zada Asad Iqbal  wrote:
> Hi,
>
> I am upgrading from solr4.5 to solr6.6 and hitting this issue during
> complete reindexing scenario.  We have some batch indexing scripts, which
> flood the solr servers with indexing requests (while keeping open-searcher
> false) for many hours and then perform one commit. This used to work fine
> with 4.5, but with 6.6, i get 'Too many open files' within a couple of
> minutes. I have checked that "ulimit" is same between old and new servers.
>
> Has something fundamentally changed in recent lucene versions, which keeps
> file descriptors around for a longer time?
>
>
> Here is a sample error message:
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:763)
> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3206)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:644)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1894)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1871)
> at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
> at
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:62)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

Re: Lucene 6.6: "Too many open files"

[Jigar Shah]

I faced such problem when I used nomergepolicy, and did some code to manual
merging segments which had bug and I had same issue. Make sure you have
default AFAIR ConcurrentMergeStrategy enabled. And its is configured with
appropriate settings.

On Jul 31, 2017 11:21 PM, "Erick Erickson"  wrote:

> No, nothing's changed fundamentally. But you say:
>
> "We have some batch indexing scripts, which
> flood the solr servers with indexing requests (while keeping open-searcher
> false)"
>
> What is your commit interval? Regardless of whether openSearcher is false
> or not, background merging continues apace with every commit. By any chance
> did you change your merge policy (or not copy the one from 4x to 6x)? Shot
> in the dark...
>
> Best,
> Erick
>
> On Mon, Jul 31, 2017 at 7:15 PM, Nawab Zada Asad Iqbal 
> wrote:
> > Hi,
> >
> > I am upgrading from solr4.5 to solr6.6 and hitting this issue during
> > complete reindexing scenario.  We have some batch indexing scripts, which
> > flood the solr servers with indexing requests (while keeping
> open-searcher
> > false) for many hours and then perform one commit. This used to work fine
> > with 4.5, but with 6.6, i get 'Too many open files' within a couple of
> > minutes. I have checked that "ulimit" is same between old and new
> servers.
> >
> > Has something fundamentally changed in recent lucene versions, which
> keeps
> > file descriptors around for a longer time?
> >
> >
> > Here is a sample error message:
> > at org.apache.lucene.index.IndexWriter.ensureOpen(
> IndexWriter.java:749)
> > at org.apache.lucene.index.IndexWriter.ensureOpen(
> IndexWriter.java:763)
> > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3206)
> > at
> > org.apache.solr.update.DirectUpdateHandler2.commit(
> DirectUpdateHandler2.java:644)
> > at
> > org.apache.solr.update.processor.RunUpdateProcessor.processCommit(
> RunUpdateProcessorFactory.java:93)
> > at
> > org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(
> UpdateRequestProcessor.java:68)
> > at
> > org.apache.solr.update.processor.DistributedUpdateProcessor.
> doLocalCommit(DistributedUpdateProcessor.java:1894)
> > at
> > org.apache.solr.update.processor.DistributedUpdateProcessor.
> processCommit(DistributedUpdateProcessor.java:1871)
> > at
> > org.apache.solr.update.processor.LogUpdateProcessorFactory$
> LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
> > at
> > org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(
> UpdateRequestProcessor.java:68)
> > at
> > org.apache.solr.handler.RequestHandlerUtils.handleCommit(
> RequestHandlerUtils.java:68)
> > at
> > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(
> ContentStreamHandlerBase.java:62)
> > at
> > org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:173)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> > at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:723)
> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> > at
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:361)
> > at
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:305)
> > at
> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1691)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:582)
> > at
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
> > at
> > org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
> > at
> > org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
> > at
> > org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1180)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:512)
> > at
> > org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
> > at
> > org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1112)
> > at
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
> > at
> > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:213)
> > at
> > org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
> > at
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
> > at
> > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
> RewriteHandler.java:335)
> > at
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
> > at org.eclipse.jetty.server.Server.handle(Server.java:534)
> > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> > at
> > org.eclipse