Re: Flink operator throttle

2020-05-16 Thread Benchao Li
Hi, > If I want to use the rate limiter in other connectors, such as Kafka sink, ES sink, I need to do some more work on these connectors. Yes, you can do this by changing Kafka/ES sink, actually, this is how we did internally. > I'd like to know if the community has a plan to make a lower-level

Re: Flink operator throttle

2020-05-16 Thread 王雷
Hi Benchao Thanks for your answer! According to your answer, I found `GuavaFlinkConnectorRateLimiter` which is the implementation of the `FlinkConnectorRateLimiter`. If I want to use the rate limiter in other connectors, such as Kafka sink, ES sink, I need to do some more work on these connector

Properly using ConnectorDescriptor instead of registerTableSource

2020-05-16 Thread Nikola Hrusov
Hello, I am trying to update my cluster from flink 1.9.2 to 1.10.1 We are running batch jobs in it. So far everything has been smooth and I have seen one of my method calls being deprecated. Our code is selecting some ORC files' contents and running query on them. It is something of this nature:

Re: Protection against huge values in RocksDB List State

2020-05-16 Thread Congxian Qiu
Hi As you described, I'm not sure whether MapState can help you in such case. MapState will serializer each separately, so it would not encounter such the problem as ListState. When using MapState, you may need to handle how to set the mapKey, if the whole state will be cleared after processed,