Re: Flink 1.11 Sql client environment yaml

2020-07-18 Thread godfrey he
hi GenericInMemoryCatalog does not support settings now, or you can refer to [1] for supported catalog details and you can refer to [2] to supported types details. "Kafka schema registry for schema" is under discussion [3], which can be ready in 1.12. sql client supports DDL to create a table wi

Re: How do I trigger clear custom state in ProcessWindowsFunction

2020-07-18 Thread David Anderson
ProcessWindowFunction#process is passed a Context object that contains public abstract KeyedStateStore windowState(); public abstract KeyedStateStore globalState(); which are available for you to use for custom state. Whatever you store in windowState is scoped to a window, and is cleared whe

How do I trigger clear custom state in ProcessWindowsFunction

2020-07-18 Thread ?g???U?[????
Dear all: How do I clear custom state in ProcessWindowsFunction? Because there is no onTimer method in ProcessAllWindowFunction. Thanks Jiazhi

Re: Flink Sinks

2020-07-18 Thread David Anderson
Prasanna, The Flink project does not have an SQS connector, and a quick google search hasn't found one. Nor does Flink have an HTTP sink, but with a bit of googling you can find that various folks have implemented this themselves. As for implementing SQS as a custom sink, if you need exactly once

Re: Flink FsStatebackend is giving better performance than RocksDB

2020-07-18 Thread David Anderson
You should be able to tune your setup to avoid the OOM problem you have run into with RocksDB. It will grow to use all of the memory available to it, but shouldn't leak. Perhaps something is misconfigured. As for performance, with the FSStateBackend you can expect: * much better throughput and av

Re: Backpressure on Window step

2020-07-18 Thread David Anderson
Steve, Your approach to debugging this sounds reasonable, but keep in mind that the backpressure detection built into the WebUI is not infallible. You could have backpressure that it doesn't detect. FWIW, keyBy isn't an operator -- it's a declaration of how the operators before and after the keyB