Hi Rex, As I said in my previous email the documentation for sink.bulk-flush.max-actions is wrong. You should be able to disable it with -1. I've just checked it on the 1.11.2 tag and it seems to be working just fine with:
CREATE TABLE esTable ( a BIGINT NOT NULL, b TIME, c STRING NOT NULL, d FLOAT, e TINYINT NOT NULL, f DATE, g TIMESTAMP NOT NULL, h as a + 2, PRIMARY KEY (a, g) NOT ENFORCED ) WITH ( 'connector'='elasticsearch-6', 'index'='table-api', 'document-type'='MyType', 'hosts'='http://127.0.0.1:9200', 'sink.flush-on-checkpoint'='false', 'sink.bulk-flush.max-actions'='-1', 'sink.bulk-flush.max-size'='0' ) If it still does not work for you with -1 could you share an example how I can reproduce the problem. Best, Dawid On 14/01/2021 18:08, Rex Fenley wrote: > Flink 1.11.2 > > CREATE TABLE sink_es ( > ... > ) WITH ( > 'connector'= 'elasticsearch-7', > 'hosts'= '${sys:proxyEnv.ELASTICSEARCH_HOSTS}', > 'index'= '${sys:graph.flink.index_name}', > 'format'= 'json', > 'sink.bulk-flush.max-actions'= '0', > 'sink.bulk-flush.max-size'= '0', > 'sink.bulk-flush.interval'= '1s', > 'sink.bulk-flush.backoff.delay'= '1s', > 'sink.bulk-flush.backoff.max-retries'= '4', > 'sink.bulk-flush.backoff.strategy'= 'CONSTANT' > ) > > On Thu, Jan 14, 2021 at 4:16 AM Dawid Wysakowicz > <dwysakow...@apache.org <mailto:dwysakow...@apache.org>> wrote: > > Hi, > > First of all, what Flink versions are you using? > > You are right it is a mistake in the documentation of the > sink.bulk-flush.max-actions. It should say: Can be set to |'-1'| > to disable it. I created a ticket[1] to track that. And as far as > I can tell and I quickly checked that it should work. As for the > sink.bulk-flush.max-size you should be able to disable it with a > value of '0'. > > Could you share with us how do you use the connector? Could you > also share the full stack trace for the exception you're getting? > Are you creating the table with a CREATE statement? > > Best, > > Dawid > > [1] https://issues.apache.org/jira/browse/FLINK-20979 > > On 13/01/2021 20:10, Rex Fenley wrote: >> Hello, >> >> It doesn't seem like we can disable max actions and max size for >> Elasticsearch connector. >> >> Docs: >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/connectors/elasticsearch.html#sink-bulk-flush-max-actions >> >> >> sink.bulk-flush.max-actions >> >> optional 1000 Integer Maximum number of buffered >> actions per >> bulk request. Can be set to |'0'| to disable it. >> >> >> sink.bulk-flush.max-size >> >> optional 2mb MemorySize Maximum size in memory of >> buffered >> actions per bulk request. Must be in MB granularity. Can be set >> to |'0'| to disable it. >> >> Reality: >> >> org.apache.flink.client.program.ProgramInvocationException: The >> main method caused an error: Max number of buffered actions must >> be larger than 0. >> >> ES code looks like -1 is actually the value for disabling, but >> when I use -1: >> Caused by: java.lang.IllegalArgumentException: Could not parse >> value '-1' for key 'sink.bulk-flush.max-size'. >> >> How can I disable these two settings? >> >> Thanks! >> >> -- >> >> Rex Fenley | Software Engineer - Mobile and Backend >> >> >> Remind.com <https://www.remind.com/>| BLOG >> <http://blog.remind.com/> | FOLLOW US >> <https://twitter.com/remindhq> | LIKE US >> <https://www.facebook.com/remindhq> >> > > > -- > > Rex Fenley | Software Engineer - Mobile and Backend > > > Remind.com <https://www.remind.com/>| BLOG > <http://blog.remind.com/> | FOLLOW US > <https://twitter.com/remindhq> | LIKE US > <https://www.facebook.com/remindhq> >
signature.asc
Description: OpenPGP digital signature
