[
https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510356#comment-17510356
]
Konstantin Knauf edited comment on FLINK-26281 at 3/22/22, 9:20 AM:
--------------------------------------------------------------------
I've done a couple of tests with ES7, none with ES6: DataStream & Table API,
NONE/AT_LEAST_ONCE, Elastic Search Index, Elastic Search Data Stream [1].
Overall, no blockers. Here are some suggestions for improvements to the
documentation:
* explain how end-to-end exactly-once can be achieved with the ES connector
(AT_LEAST_ONCE + upserts based on a deterministic id). In my understanding it
is not recommended anymore in Elastic to set the id manually, so this might be
worth highlighting.
* remove the `type` from the IndexRequest, because it is deprected
* Add documentation for usage of es datastreams (e.g. requires a @timestamp
field, only OpType.CREATE, what it means for end-to-end delivery guarantees)
I don't think, I'll have the time to look into ES6. So, [~alexanderpreuss],
I'll leave it up to you to either move this to DONE or to unassign myself.
[1]
[https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html]
was (Author: knaufk):
I've done a couple of tests with ES7, none with ES6: DataStream & Table API,
NONE/AT_LEAST_ONCE, Elastic Search Index, Elastic Search Data Stream [1].
Overall, no blockers. Here are some suggestions for improvements to the
documentation:
* explain how end-to-end exactly-once can be achieved with the ES connector
(AT_LEAST_ONCE + upserts based on a deterministic id). In my understanding it
is not recommended anymore in Elastic to set the id manually, so this might be
worth highlighting.
* remove the `type` from the IndexRequest, because it is deprected
* Add documentation for usage of es datastreams (e.g. requires a @timestamp
field, only OpType.CREATE, what it means for end-to-end delivery guarantees)
[1]
https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html
> Test Elasticsearch connector End2End
> ------------------------------------
>
> Key: FLINK-26281
> URL: https://issues.apache.org/jira/browse/FLINK-26281
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / ElasticSearch
> Affects Versions: 1.15.0
> Reporter: Alexander Preuss
> Assignee: Konstantin Knauf
> Priority: Blocker
> Labels: pull-request-available, release-testing
> Fix For: 1.15.0
>
>
> Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323
> Documentation for [datastream
> api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/]
> Documentation for [table
> api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/]
> As 1.15 deprecated the SinkFunction-based Elasticsearch connector and
> introduces the new connector based on the Sink interface we should test it
> behaves correctly and as the user expects.
>
> Some suggestions what to test:
> * Test delivery guarantees (none, at-least-once) (exactly-once should not
> run)
> * Write a simple job that is inserting/upserting data into Elasticsearch
> * Write a simple job that is inserting/upserting data into Elasticsearch and
> use a non-default parallelism
> * Write a simple job in both datastream api and table api
> * Test restarting jobs and scaling up/down
> * Test failure of a simple job that is inserting data with exactly-once
> delivery guarantee by terminating and restarting Elasticsearch
> * Test against Elasticsearch 6.X and 7.X with the respective connectors
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
