Re: Approaches to customize the parallelism in SQL generated operators

2021-03-20 Thread David Anderson
No, there is no mechanism available for individually tuning the parallelism of the generated operators in a SQL job. Moreover, such fine-tuning is often counter-productive. In most cases you are better off simply setting the overall parallelism to whatever is needed by the busiest operator(s). Unne

Re: Flink application has slightly data loss using Processing Time

2021-03-20 Thread David Anderson
You should increase the kafka transaction timeout -- transaction.max.timeout.ms -- to something much larger than the default, which I believe is 15 minutes. Suitable values are more on the order of a few hours to a few days -- long enough to allow for any conceivable outage. This way, if a request

DataDog and Flink

2021-03-20 Thread Vishal Santoshi
Hello folks, This is quite strange. We see a TM stop reporting metrics to DataDog .The logs from that specific TM for every DataDog dispatch time out with* java.net.SocketTimeoutException: timeout *and that seems to repeat over every dispatch to DataDog. It seems it is on a 10 se