somandal opened a new pull request, #16094: URL: https://github.com/apache/pinot/pull/16094
This PR adds handling for potential bad data for some of the `IndexCreator::add()` functions: - LuceneFSTIndexCreator - adds handling in the constructor and add() to catch exceptions and skip - NativeTextIndexCreator - adds handling in add() to catch exceptions and skip - BaseJsonIndexCreator - adds more handling for JSON processing exceptions and reuses the skip flag for returning a default SKIPPED record Also added some test files for the following (these indexes haven't been changed): - HnswVectorIndexCreatorTest - LuceneTextIndexCreatorTest Testing: - Added some tests for LuceneFSTIndexCreator, NativeTextIndexCreator, and ran the existing tests for JSON (hard to add tests for the exact scenarios) Some complications to address skip or add default value for `HnswVectorIndexCreator` and `LuceneTextIndexCreator`: - They use: https://lucene.apache.org/core/9_0_0/core/org/apache/lucene/index/IndexWriter.html#addDocument(java.lang.Iterable) which can throw exceptions for some system level issues, which may not be easy to handle - The index readers need a correct (smaller or equal to) numDocs as input, otherwise its creation fails due to overflow in number of docs. Thus we cannot just skip these records -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
