Tzu-Li (Gordon) Tai created FLINK-5487:
------------------------------------------

             Summary: Proper at-least-once support for ElasticsearchSink
                 Key: FLINK-5487
                 URL: https://issues.apache.org/jira/browse/FLINK-5487
             Project: Flink
          Issue Type: Bug
          Components: Streaming Connectors
            Reporter: Tzu-Li (Gordon) Tai
            Priority: Critical


Discussion in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html

Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
message delivery.

For proper support of at-least-once, the sink will need to participate in 
Flink's checkpointing: when snapshotting is triggered at the 
`ElasticsearchSink`, we need to synchronize on the pending `ActionRequest`s by 
flushing the `BulkProcessor`. For temporary ES failures (see FLINK-5122) that 
may happen on the flush, we should retry them before returning from 
snapshotting and acking the checkpoint. If there are non-temporary ES failures 
on the flush, the current snapshot should fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to