I'm migrating code from Lucene 3.5 to 4.0. I have the following code which is supposed to highlight text. I get the exception InvalidTokenOffsetsException. I have no idea what that means. I am using a custom analyzer which seems to work for searching/indexing, so I assume it will work here (even though it took a couple of "minor" changes to get it to compile in 4.0 This code used to work in 3.5.
Anyone have any ideas? Scott Code fragment: try { ctf = new CachingTokenFilter(myCustomAnalyzer .tokenStream(MyFieldName, new StringReader(myText))); } catch (IOException e1) { s_oLog.error("Search:markCommon: Exception creating CachingTokenFilter: " + e1.getMessage()); return null; } String markedString; SimpleHTMLFormatter formatter; try { formatter = new SimpleHTMLFormatter(_zBeginHighlight, _zEndHighlight); Scorer score = new QueryScorer(q); ht = new Highlighter(formatter, score); ht.setTextFragmenter(new NullFragmenter()); markedString = ht.getBestFragment(ctf, myText); } catch (IOException e) { s_oLog.error("Search:markCommon: Unable to highlight string: " + e.getMessage()); return null; } catch(InvalidTokenOffsetsException e2) { s_oLog.error("Search:markCommon: Unable to highlight string2: " + e2.getMessage()); return null; }