rinkako created FLINK-18359:
-------------------------------
Summary: Improve error-log strategy for Elasticsearch sink for
large data documentId conflict when using create mode for `insert ignore`
semantics
Key: FLINK-18359
URL: https://issues.apache.org/jira/browse/FLINK-18359
Project: Flink
Issue Type: Improvement
Reporter: rinkako
The story is: when a flink job for ingesting large number of records from data
source, processing and indexing to ES with Elasticsearch sink fault, we may
restart it from a specific data set which contains lots of data which already
sink into ES.
At this case, a `INSERT IGNORE` semantics is necessary, and we use `public
IndexRequest create(boolean create)` with `true` args and ignore the 409
restStatusCode at a customized ActionRequestFailureHandler to make it work.
But, the `BulkProcessorListener` always log a error event before it calls the
`failureHandler` in its `afterBulk` method, and will produce tons of error log
for document id conflict, which we already know and handle them in customized
ActionRequestFailureHandler.
Therefore, it seems that the error log action at the
ActionRequestFailureHandler (either the default IgnoringFailureHandler or a
custom handler) is more flexible ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
