Flink operator throttle

hi, All Does Flink support rate limitation? How to limit the rate when the external database connected by the sink operator has throughput limitation. Instead of passive back pressure after reaching the limit of the external database, we want to limit rate actively. Thanks Ray

Re: Flink operator throttle

AFAIK, `FlinkKafkaConsumer010#setRateLimiter` can configure the kafka source to have a rate limiter. (I assume you uses Kafka) However it only exists in Kafka 0.10 DataStream Connector, not in other versions nor table api. 王雷 于2020年5月14日周四 下午5:31写道： > hi, All > > Does Flink support rate limitati

Re: How To subscribe a Kinesis Stream using enhance fanout?

Hi Xiaolong, You are right, the way the Kinesis connector is implemented / the way the AWS APIs are used, does not allow it to consume Kinesis streams with enhanced fan-out enabled consumers [1]. Could you open a JIRA ticket for this? As far as I can tell, this could be a valuable contribution to

Re: How To subscribe a Kinesis Stream using enhance fanout?

Thanks, I'll check it out. On Thu, May 14, 2020 at 6:26 PM Tzu-Li (Gordon) Tai wrote: > Hi Xiaolong, > > You are right, the way the Kinesis connector is implemented / the way the > AWS APIs are used, does not allow it to consume Kinesis streams with > enhanced fan-out enabled consumers [1]. > Co

Re: ProducerRecord with Kafka Sink for 1.8.0

> > Its because the flink distribution of the cluster is 1.7.2. We use a > standalone cluster , so in the lib directory in flink the artifact is > flink-core-1.7.2.jar . I need to pack flink-core-1.9.0.jar from application > and use child first class loading to use newer version of flink-core. > D

Re: Statefun 2.0 questions

Hi, I'm glad things are getting clearer, looking forward to seeing how statefun is working out for you :-) To change the parallelism you can simply set the "parallelism.default" [1] key in flink-conf.yaml. It is located in the statefun container at /opt/flink/conf/flink-conf.yaml. To avoid rebuild

Re: ProducerRecord with Kafka Sink for 1.8.0

Hi Gary, I have used this technique before. I deleted flink-avro jar from lib and packed it into the application jar and there are no problems. Best, Nick On Thu, May 14, 2020 at 6:11 AM Gary Yao wrote: > Its because the flink distribution of the cluster is 1.7.2. We use a >> standalone cluste

[Announce] Flink Forward Global 2020 - Call for Proposals

Hi Everyone! After a successful Virtual Flink Forward in April, we have decided to present our October edition in the same way. In these uncertain times, we are conscious of everyone's health and safety and want to make sure our events are accessible for everyone. Flink Forward Global Conferenc

Re: Watermarks and parallelism

Hi Gnana, 1. No, watermarks are generated independently per subtask. I think this section of the docs might make things more clear - [1] . 2. The same watermark from the

Protection against huge values in RocksDB List State

Hi all! I cannot seem to find any setting to limit the number of records appended in a RocksDBListState that is used when we use SessionWindows with a ProcessFunction. It seems that, for each incoming element, the new element will be appended to the value with the RocksDB `merge` operator, without

How to read UUID out of a JDBC table source

Dear list, I need to use a Table Source to extract data from a PostgreSQL table that includes a column of type uuid. Data in the column is converted to java.util.UUID by the postgresql jdbc driver (I guess) however I was not able to find a way to define a Table schema for correctly reading tha

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hi Yang, I am able to resolve the issue by removing Hadoop dependency as you mentioned. 1. Removed hadoop-common dependency and org.apache.flink flink-streaming-java_${scala.binary.version} ${flink.version} org.apache.flink flink-hadoop-f

Re: Protection against huge values in RocksDB List State

Hi Robin First of all, the root cause is not RocksDB cannot store large list state when you merge but the JNI limitation of 2^31 bytes [1]. Moreover, RocksDB java would not return anything when you call merge [2] operator. Did you merge too many elements or just merge too big-size elements? Las

Flink performance tuning on operators

Hi, We have a Flink job that reads data from an input stream, then converts each event from JSON string Avro object, finally writes to parquet files using StreamingFileSink with OnCheckPointRollingPolicy of 5 mins. Basically a stateless job. Initially, we use one map operator to convert Json st

Flink suggestions;

Hello Guys, I am a beginner in this field of real-time streaming and i am working with apache flink, and i ignore a lot of features of it, and actually I am building an application, in which i receive some sensors data in this format {"status": "Alerte", "classe": " ", "value": {"temperature": 15.7

Re: Flink restart strategy on specific exception

Ticket FLINK-17714 is created to track this requirement. Thanks, Zhu Zhu Till Rohrmann 于2020年5月13日周三 下午8:30写道： > Yes, you are right Zhu Zhu. Extending > the RestartBackoffTimeStrategyFactoryLoader to also load custom > RestartBackoffTimeStrategies sound like a good improvement for the future. >