Will do. Thank you for digging in and finding this! On Mon, Jan 18, 2021 at 3:01 AM Dawid Wysakowicz <dwysakow...@apache.org> wrote:
> I've checked it again, with ES 7 this time (I missed that previously and > checked with ES 6) and actually you're right it fails for value -1. I > created a JIRA[1] to fix it. Until this is merged unfortunately, I can't > think of a way to actually disable it. > > For a future reference. It would be really really nice if you could > provide us with a full stack trace instead of just the message, as it is > much much easier to track back the problem that way. > > Best, > > Dawid > > [1] https://issues.apache.org/jira/browse/FLINK-21009 > On 15/01/2021 21:28, Rex Fenley wrote: > > Yes, use the same SQL and change '0's to '-1'. We received "Caused by: > java.lang.IllegalArgumentException: Could not parse value '-1' for key > 'sink.bulk-flush.max-size'." > > On Fri, Jan 15, 2021 at 6:04 AM Dawid Wysakowicz <dwysakow...@apache.org> > wrote: > >> Hi Rex, >> >> As I said in my previous email the documentation for >> sink.bulk-flush.max-actions is wrong. You should be able to disable it with >> -1. I've just checked it on the 1.11.2 tag and it seems to be working just >> fine with: >> >> CREATE TABLE esTable ( >> >> a BIGINT NOT NULL, >> b TIME, >> c STRING NOT NULL, >> d FLOAT, >> e TINYINT NOT NULL, >> f DATE, >> g TIMESTAMP NOT NULL, >> h as a + 2, >> PRIMARY KEY (a, g) NOT ENFORCED >> ) >> WITH ( >> 'connector'='elasticsearch-6', >> 'index'='table-api', >> 'document-type'='MyType', >> 'hosts'='http://127.0.0.1:9200', >> 'sink.flush-on-checkpoint'='false', >> 'sink.bulk-flush.max-actions'='-1', >> 'sink.bulk-flush.max-size'='0' >> ) >> >> If it still does not work for you with -1 could you share an example how >> I can reproduce the problem. >> >> Best, >> >> Dawid >> On 14/01/2021 18:08, Rex Fenley wrote: >> >> Flink 1.11.2 >> >> CREATE TABLE sink_es ( >> ... >> ) WITH ( >> 'connector' = 'elasticsearch-7', >> 'hosts' = '${sys:proxyEnv.ELASTICSEARCH_HOSTS}', >> 'index' = '${sys:graph.flink.index_name}', >> 'format' = 'json', >> 'sink.bulk-flush.max-actions' = '0', >> 'sink.bulk-flush.max-size' = '0', >> 'sink.bulk-flush.interval' = '1s', >> 'sink.bulk-flush.backoff.delay' = '1s', >> 'sink.bulk-flush.backoff.max-retries' = '4', >> 'sink.bulk-flush.backoff.strategy' = 'CONSTANT' >> ) >> >> On Thu, Jan 14, 2021 at 4:16 AM Dawid Wysakowicz <dwysakow...@apache.org> >> wrote: >> >>> Hi, >>> >>> First of all, what Flink versions are you using? >>> >>> You are right it is a mistake in the documentation of the >>> sink.bulk-flush.max-actions. It should say: Can be set to '-1' to >>> disable it. I created a ticket[1] to track that. And as far as I can tell >>> and I quickly checked that it should work. As for the >>> sink.bulk-flush.max-size you should be able to disable it with a value of >>> '0'. >>> >>> Could you share with us how do you use the connector? Could you also >>> share the full stack trace for the exception you're getting? Are you >>> creating the table with a CREATE statement? >>> >>> Best, >>> >>> Dawid >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-20979 >>> On 13/01/2021 20:10, Rex Fenley wrote: >>> >>> Hello, >>> >>> It doesn't seem like we can disable max actions and max size for >>> Elasticsearch connector. >>> >>> Docs: >>> >>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/connectors/elasticsearch.html#sink-bulk-flush-max-actions >>> sink.bulk-flush.max-actions optional 1000 Integer Maximum number of >>> buffered actions per bulk request. Can be set to '0' to disable it. >>> sink.bulk-flush.max-size optional 2mb MemorySize Maximum size in memory >>> of buffered actions per bulk request. Must be in MB granularity. Can be set >>> to '0' to disable it. >>> Reality: >>> >>> org.apache.flink.client.program.ProgramInvocationException: The main >>> method caused an error: Max number of buffered actions must be larger than >>> 0. >>> >>> ES code looks like -1 is actually the value for disabling, but when I >>> use -1: >>> Caused by: java.lang.IllegalArgumentException: Could not parse value >>> '-1' for key 'sink.bulk-flush.max-size'. >>> >>> How can I disable these two settings? >>> >>> Thanks! >>> >>> -- >>> >>> Rex Fenley | Software Engineer - Mobile and Backend >>> >>> >>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> >>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US >>> <https://www.facebook.com/remindhq> >>> >>> >> >> -- >> >> Rex Fenley | Software Engineer - Mobile and Backend >> >> >> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> >> | FOLLOW US <https://twitter.com/remindhq> | LIKE US >> <https://www.facebook.com/remindhq> >> >> > > -- > > Rex Fenley | Software Engineer - Mobile and Backend > > > Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> | > FOLLOW US <https://twitter.com/remindhq> | LIKE US > <https://www.facebook.com/remindhq> > > -- Rex Fenley | Software Engineer - Mobile and Backend Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> | FOLLOW US <https://twitter.com/remindhq> | LIKE US <https://www.facebook.com/remindhq>
