Hi Roman, Thanks for working on this! I deployed the change and it appears to be working as expected.
Will monitor over a period of time to compare the checkpoint counts and get back to you if there are still issues. Thomas On Thu, Aug 13, 2020 at 3:41 AM Roman Khachatryan <ro...@data-artisans.com> wrote: > Hi Thomas, > > The fix is now merged to master and to release-1.11. > So if you'd like you can check if it solves your problem (it would be > helpful for us too). > > On Sat, Aug 8, 2020 at 9:26 AM Roman Khachatryan <ro...@data-artisans.com> > wrote: > >> Hi Thomas, >> >> Thanks a lot for the detailed information. >> >> I think the problem is in CheckpointCoordinator. It stores the last >> checkpoint completion time after checking queued requests. >> I've created a ticket to fix this: >> https://issues.apache.org/jira/browse/FLINK-18856 >> >> >> On Sat, Aug 8, 2020 at 5:25 AM Thomas Weise <t...@apache.org> wrote: >> >>> Just another update: >>> >>> The duration of snapshotState is capped by the Kinesis >>> producer's "RecordTtl" setting (default 30s). The sleep time in flushSync >>> does not contribute to the observed behavior. >>> >>> I guess the open question is why, with the same settings, is 1.11 since >>> commit 355184d69a8519d29937725c8d85e8465d7e3a90 processing more checkpoints? >>> >>> >>> On Fri, Aug 7, 2020 at 9:15 AM Thomas Weise <t...@apache.org> wrote: >>> >>>> Hi Roman, >>>> >>>> Here are the checkpoint summaries for both commits: >>>> >>>> >>>> https://docs.google.com/presentation/d/159IVXQGXabjnYJk3oVm3UP2UW_5G-TGs_u9yzYb030I/edit#slide=id.g86d15b2fc7_0_0 >>>> >>>> The config: >>>> >>>> CheckpointConfig checkpointConfig = env.getCheckpointConfig(); >>>> >>>> checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); >>>> checkpointConfig.setCheckpointInterval(*10_000*); >>>> checkpointConfig.setMinPauseBetweenCheckpoints(*10_000*); >>>> >>>> checkpointConfig.enableExternalizedCheckpoints(DELETE_ON_CANCELLATION); >>>> checkpointConfig.setCheckpointTimeout(600_000); >>>> checkpointConfig.setMaxConcurrentCheckpoints(1); >>>> checkpointConfig.setFailOnCheckpointingErrors(true); >>>> >>>> The values marked bold when changed to *60_000* make the symptom >>>> disappear. I meanwhile also verified that with the 1.11.0 release commit. >>>> >>>> I will take a look at the sleep time issue. >>>> >>>> Thanks, >>>> Thomas >>>> >>>> >>>> On Fri, Aug 7, 2020 at 1:44 AM Roman Khachatryan < >>>> ro...@data-artisans.com> wrote: >>>> >>>>> Hi Thomas, >>>>> >>>>> Thanks for your reply! >>>>> >>>>> I think you are right, we can remove this sleep and improve >>>>> KinesisProducer. >>>>> Probably, it's snapshotState can also be sped up by forcing records >>>>> flush more often. >>>>> Do you see that 30s checkpointing duration is caused >>>>> by KinesisProducer (or maybe other operators)? >>>>> >>>>> I'd also like to understand the reason behind this increase in >>>>> checkpoint frequency. >>>>> Can you please share these values: >>>>> - execution.checkpointing.min-pause >>>>> - execution.checkpointing.max-concurrent-checkpoints >>>>> - execution.checkpointing.timeout >>>>> >>>>> And what is the "new" observed checkpoint frequency (or how many >>>>> checkpoints are created) compared to older versions? >>>>> >>>>> >>>>> On Fri, Aug 7, 2020 at 4:49 AM Thomas Weise <t...@apache.org> wrote: >>>>> >>>>>> Hi Roman, >>>>>> >>>>>> Indeed there are more frequent checkpoints with this change! The >>>>>> application was configured to checkpoint every 10s. With 1.10 ("good >>>>>> commit"), that leads to fewer completed checkpoints compared to 1.11 >>>>>> ("bad >>>>>> commit"). Just to be clear, the only difference between the two runs >>>>>> was >>>>>> the commit 355184d69a8519d29937725c8d85e8465d7e3a90 >>>>>> >>>>>> Since the sync part of checkpoints with the Kinesis producer always >>>>>> takes >>>>>> ~30 seconds, the 10s configured checkpoint frequency really had no >>>>>> effect >>>>>> before 1.11. I confirmed that both commits perform comparably by >>>>>> setting >>>>>> the checkpoint frequency and min pause to 60s. >>>>>> >>>>>> I still have to verify with the final 1.11.0 release commit. >>>>>> >>>>>> It's probably good to take a look at the Kinesis producer. Is it >>>>>> really >>>>>> necessary to have 500ms sleep time? What's responsible for the ~30s >>>>>> duration in snapshotState? >>>>>> >>>>>> As things stand it doesn't make sense to use checkpoint intervals < >>>>>> 30s >>>>>> when using the Kinesis producer. >>>>>> >>>>>> Thanks, >>>>>> Thomas >>>>>> >>>>>> On Sat, Aug 1, 2020 at 2:53 PM Roman Khachatryan < >>>>>> ro...@data-artisans.com> >>>>>> wrote: >>>>>> >>>>>> > Hi Thomas, >>>>>> > >>>>>> > Thanks a lot for the analysis. >>>>>> > >>>>>> > The first thing that I'd check is whether checkpoints became more >>>>>> frequent >>>>>> > with this commit (as each of them adds at least 500ms if there is >>>>>> at least >>>>>> > one not sent record, according to >>>>>> FlinkKinesisProducer.snapshotState). >>>>>> > >>>>>> > Can you share checkpointing statistics (1.10 vs 1.11 or last "good" >>>>>> vs >>>>>> > first "bad" commits)? >>>>>> > >>>>>> > On Fri, Jul 31, 2020 at 5:29 AM Thomas Weise < >>>>>> thomas.we...@gmail.com> >>>>>> > wrote: >>>>>> > >>>>>> > > I run git bisect and the first commit that shows the regression >>>>>> is: >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > >>>>>> https://github.com/apache/flink/commit/355184d69a8519d29937725c8d85e8465d7e3a90 >>>>>> > > >>>>>> > > >>>>>> > > On Thu, Jul 23, 2020 at 6:46 PM Kurt Young <ykt...@gmail.com> >>>>>> wrote: >>>>>> > > >>>>>> > > > From my experience, java profilers are sometimes not accurate >>>>>> enough to >>>>>> > > > find out the performance regression >>>>>> > > > root cause. In this case, I would suggest you try out intel >>>>>> vtune >>>>>> > > amplifier >>>>>> > > > to watch more detailed metrics. >>>>>> > > > >>>>>> > > > Best, >>>>>> > > > Kurt >>>>>> > > > >>>>>> > > > >>>>>> > > > On Fri, Jul 24, 2020 at 8:51 AM Thomas Weise <t...@apache.org> >>>>>> wrote: >>>>>> > > > >>>>>> > > > > The cause of the issue is all but clear. >>>>>> > > > > >>>>>> > > > > Previously I had mentioned that there is no suspect change to >>>>>> the >>>>>> > > Kinesis >>>>>> > > > > connector and that I had reverted the AWS SDK change to no >>>>>> effect. >>>>>> > > > > >>>>>> > > > > https://issues.apache.org/jira/browse/FLINK-17496 actually >>>>>> fixed >>>>>> > > another >>>>>> > > > > regression in the previous release and is present before and >>>>>> after. >>>>>> > > > > >>>>>> > > > > I repeated the run with 1.11.0 core and downgraded the entire >>>>>> Kinesis >>>>>> > > > > connector to 1.10.1: Nothing changes, i.e. the regression is >>>>>> still >>>>>> > > > present. >>>>>> > > > > Therefore we will need to look elsewhere for the root cause. >>>>>> > > > > >>>>>> > > > > Regarding the time spent in snapshotState, repeat runs reveal >>>>>> a wide >>>>>> > > > range >>>>>> > > > > for both versions, 1.10 and 1.11. So again this is nothing >>>>>> pointing >>>>>> > to >>>>>> > > a >>>>>> > > > > root cause. >>>>>> > > > > >>>>>> > > > > At this point, I have no ideas remaining other than doing a >>>>>> bisect to >>>>>> > > > find >>>>>> > > > > the culprit. Any other suggestions? >>>>>> > > > > >>>>>> > > > > Thomas >>>>>> > > > > >>>>>> > > > > >>>>>> > > > > On Thu, Jul 16, 2020 at 9:19 PM Zhijiang < >>>>>> wangzhijiang...@aliyun.com >>>>>> > > > > .invalid> >>>>>> > > > > wrote: >>>>>> > > > > >>>>>> > > > > > Hi Thomas, >>>>>> > > > > > >>>>>> > > > > > Thanks for your further profiling information and glad to >>>>>> see we >>>>>> > > > already >>>>>> > > > > > finalized the location to cause the regression. >>>>>> > > > > > Actually I was also suspicious of the point of >>>>>> #snapshotState in >>>>>> > > > previous >>>>>> > > > > > discussions since it indeed cost much time to block normal >>>>>> operator >>>>>> > > > > > processing. >>>>>> > > > > > >>>>>> > > > > > Based on your below feedback, the sleep time during >>>>>> #snapshotState >>>>>> > > > might >>>>>> > > > > > be the main concern, and I also digged into the >>>>>> implementation of >>>>>> > > > > > FlinkKinesisProducer#snapshotState. >>>>>> > > > > > while (producer.getOutstandingRecordsCount() > 0) { >>>>>> > > > > > producer.flush(); >>>>>> > > > > > try { >>>>>> > > > > > Thread.sleep(500); >>>>>> > > > > > } catch (InterruptedException e) { >>>>>> > > > > > LOG.warn("Flushing was interrupted."); >>>>>> > > > > > break; >>>>>> > > > > > } >>>>>> > > > > > } >>>>>> > > > > > It seems that the sleep time is mainly affected by the >>>>>> internal >>>>>> > > > > operations >>>>>> > > > > > inside KinesisProducer implementation provided by >>>>>> amazonaws, which >>>>>> > I >>>>>> > > am >>>>>> > > > > not >>>>>> > > > > > quite familiar with. >>>>>> > > > > > But I noticed there were two upgrades related to it in >>>>>> > > release-1.11.0. >>>>>> > > > > One >>>>>> > > > > > is for upgrading amazon-kinesis-producer to 0.14.0 [1] and >>>>>> another >>>>>> > is >>>>>> > > > for >>>>>> > > > > > upgrading aws-sdk-version to 1.11.754 [2]. >>>>>> > > > > > You mentioned that you already reverted the SDK upgrade to >>>>>> verify >>>>>> > no >>>>>> > > > > > changes. Did you also revert the [1] to verify? >>>>>> > > > > > [1] https://issues.apache.org/jira/browse/FLINK-17496 >>>>>> > > > > > [2] https://issues.apache.org/jira/browse/FLINK-14881 >>>>>> > > > > > >>>>>> > > > > > Best, >>>>>> > > > > > Zhijiang >>>>>> > > > > > >>>>>> ------------------------------------------------------------------ >>>>>> > > > > > From:Thomas Weise <t...@apache.org> >>>>>> > > > > > Send Time:2020年7月17日(星期五) 05:29 >>>>>> > > > > > To:dev <dev@flink.apache.org> >>>>>> > > > > > Cc:Zhijiang <wangzhijiang...@aliyun.com>; Stephan Ewen < >>>>>> > > > se...@apache.org >>>>>> > > > > >; >>>>>> > > > > > Arvid Heise <ar...@ververica.com>; Aljoscha Krettek < >>>>>> > > > aljos...@apache.org >>>>>> > > > > > >>>>>> > > > > > Subject:Re: Kinesis Performance Issue (was [VOTE] Release >>>>>> 1.11.0, >>>>>> > > > release >>>>>> > > > > > candidate #4) >>>>>> > > > > > >>>>>> > > > > > Sorry for the delay. >>>>>> > > > > > >>>>>> > > > > > I confirmed that the regression is due to the sink >>>>>> (unsurprising, >>>>>> > > since >>>>>> > > > > > another job with the same consumer, but not the producer, >>>>>> runs as >>>>>> > > > > > expected). >>>>>> > > > > > >>>>>> > > > > > As promised I did CPU profiling on the problematic >>>>>> application, >>>>>> > which >>>>>> > > > > gives >>>>>> > > > > > more insight into the regression [1] >>>>>> > > > > > >>>>>> > > > > > The screenshots show that the average time for snapshotState >>>>>> > > increases >>>>>> > > > > from >>>>>> > > > > > ~9s to ~28s. The data also shows the increase in sleep time >>>>>> during >>>>>> > > > > > snapshotState. >>>>>> > > > > > >>>>>> > > > > > Does anyone, based on changes made in 1.11, have a theory >>>>>> why? >>>>>> > > > > > >>>>>> > > > > > I had previously looked at the changes to the Kinesis >>>>>> connector and >>>>>> > > > also >>>>>> > > > > > reverted the SDK upgrade, which did not change the >>>>>> situation. >>>>>> > > > > > >>>>>> > > > > > It will likely be necessary to drill into the sink / >>>>>> checkpointing >>>>>> > > > > details >>>>>> > > > > > to understand the cause of the problem. >>>>>> > > > > > >>>>>> > > > > > Let me know if anyone has specific questions that I can >>>>>> answer from >>>>>> > > the >>>>>> > > > > > profiling results. >>>>>> > > > > > >>>>>> > > > > > Thomas >>>>>> > > > > > >>>>>> > > > > > [1] >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > >>>>>> > >>>>>> https://docs.google.com/presentation/d/159IVXQGXabjnYJk3oVm3UP2UW_5G-TGs_u9yzYb030I/edit?usp=sharing >>>>>> > > > > > >>>>>> > > > > > On Mon, Jul 13, 2020 at 11:14 AM Thomas Weise < >>>>>> t...@apache.org> >>>>>> > > wrote: >>>>>> > > > > > >>>>>> > > > > > > + dev@ for visibility >>>>>> > > > > > > >>>>>> > > > > > > I will investigate further today. >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > On Wed, Jul 8, 2020 at 4:42 AM Aljoscha Krettek < >>>>>> > > aljos...@apache.org >>>>>> > > > > >>>>>> > > > > > > wrote: >>>>>> > > > > > > >>>>>> > > > > > >> On 06.07.20 20:39, Stephan Ewen wrote: >>>>>> > > > > > >> > - Did sink checkpoint notifications change in a >>>>>> relevant >>>>>> > way, >>>>>> > > > for >>>>>> > > > > > >> example >>>>>> > > > > > >> > due to some Kafka issues we addressed in 1.11 >>>>>> (@Aljoscha >>>>>> > maybe?) >>>>>> > > > > > >> >>>>>> > > > > > >> I think that's unrelated: the Kafka fixes were isolated >>>>>> in Kafka >>>>>> > > and >>>>>> > > > > the >>>>>> > > > > > >> one bug I discovered on the way was about the Task >>>>>> reaper. >>>>>> > > > > > >> >>>>>> > > > > > >> >>>>>> > > > > > >> On 07.07.20 17:51, Zhijiang wrote: >>>>>> > > > > > >> > Sorry for my misunderstood of the previous information, >>>>>> > Thomas. >>>>>> > > I >>>>>> > > > > was >>>>>> > > > > > >> assuming that the sync checkpoint duration increased >>>>>> after >>>>>> > upgrade >>>>>> > > > as >>>>>> > > > > it >>>>>> > > > > > >> was mentioned before. >>>>>> > > > > > >> > >>>>>> > > > > > >> > If I remembered correctly, the memory state backend >>>>>> also has >>>>>> > the >>>>>> > > > > same >>>>>> > > > > > >> issue? If so, we can dismiss the rocksDB state changes. >>>>>> As the >>>>>> > > slot >>>>>> > > > > > sharing >>>>>> > > > > > >> enabled, the downstream and upstream should >>>>>> > > > > > >> > probably deployed into the same slot, then no network >>>>>> shuffle >>>>>> > > > > effect. >>>>>> > > > > > >> > >>>>>> > > > > > >> > I think we need to find out whether it has other >>>>>> symptoms >>>>>> > > changed >>>>>> > > > > > >> besides the performance regression to further figure out >>>>>> the >>>>>> > > scope. >>>>>> > > > > > >> > E.g. any metrics changes, the number of TaskManager >>>>>> and the >>>>>> > > number >>>>>> > > > > of >>>>>> > > > > > >> slots per TaskManager from deployment changes. >>>>>> > > > > > >> > 40% regression is really big, I guess the changes >>>>>> should also >>>>>> > be >>>>>> > > > > > >> reflected in other places. >>>>>> > > > > > >> > >>>>>> > > > > > >> > I am not sure whether we can reproduce the regression >>>>>> in our >>>>>> > AWS >>>>>> > > > > > >> environment by writing any Kinesis jobs, since there are >>>>>> also >>>>>> > > normal >>>>>> > > > > > >> Kinesis jobs as Thomas mentioned after upgrade. >>>>>> > > > > > >> > So it probably looks like to touch some corner case. I >>>>>> am very >>>>>> > > > > willing >>>>>> > > > > > >> to provide any help for debugging if possible. >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> > Best, >>>>>> > > > > > >> > Zhijiang >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > ------------------------------------------------------------------ >>>>>> > > > > > >> > From:Thomas Weise <t...@apache.org> >>>>>> > > > > > >> > Send Time:2020年7月7日(星期二) 23:01 >>>>>> > > > > > >> > To:Stephan Ewen <se...@apache.org> >>>>>> > > > > > >> > Cc:Aljoscha Krettek <aljos...@apache.org>; Arvid >>>>>> Heise < >>>>>> > > > > > >> ar...@ververica.com>; Zhijiang < >>>>>> wangzhijiang...@aliyun.com> >>>>>> > > > > > >> > Subject:Re: Kinesis Performance Issue (was [VOTE] >>>>>> Release >>>>>> > > 1.11.0, >>>>>> > > > > > >> release candidate #4) >>>>>> > > > > > >> > >>>>>> > > > > > >> > We are deploying our apps with FlinkK8sOperator. We >>>>>> have one >>>>>> > job >>>>>> > > > > that >>>>>> > > > > > >> works as expected after the upgrade and the one >>>>>> discussed here >>>>>> > > that >>>>>> > > > > has >>>>>> > > > > > the >>>>>> > > > > > >> performance regression. >>>>>> > > > > > >> > >>>>>> > > > > > >> > "The performance regression is obvious caused by long >>>>>> duration >>>>>> > > of >>>>>> > > > > sync >>>>>> > > > > > >> checkpoint process in Kinesis sink operator, which would >>>>>> block >>>>>> > the >>>>>> > > > > > normal >>>>>> > > > > > >> data processing until back pressure the source." >>>>>> > > > > > >> > >>>>>> > > > > > >> > That's a constant. Before (1.10) and upgrade have the >>>>>> same >>>>>> > sync >>>>>> > > > > > >> checkpointing time. The question is what change came in >>>>>> with the >>>>>> > > > > > upgrade. >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> > On Tue, Jul 7, 2020 at 7:33 AM Stephan Ewen < >>>>>> se...@apache.org >>>>>> > > >>>>>> > > > > wrote: >>>>>> > > > > > >> > >>>>>> > > > > > >> > @Thomas Just one thing real quick: Are you using the >>>>>> > standalone >>>>>> > > > > setup >>>>>> > > > > > >> scripts (like start-cluster.sh, and the former "slaves" >>>>>> file) ? >>>>>> > > > > > >> > Be aware that this is now called "workers" because of >>>>>> avoiding >>>>>> > > > > > >> sensitive names. >>>>>> > > > > > >> > In one internal benchmark we saw quite a lot of >>>>>> slowdown >>>>>> > > > initially, >>>>>> > > > > > >> before seeing that the cluster was not a distributed >>>>>> cluster any >>>>>> > > > more >>>>>> > > > > > ;-) >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> > On Tue, Jul 7, 2020 at 9:08 AM Zhijiang < >>>>>> > > > wangzhijiang...@aliyun.com >>>>>> > > > > > >>>>>> > > > > > >> wrote: >>>>>> > > > > > >> > Thanks for this kickoff and help analysis, Stephan! >>>>>> > > > > > >> > Thanks for the further feedback and investigation, >>>>>> Thomas! >>>>>> > > > > > >> > >>>>>> > > > > > >> > The performance regression is obvious caused by long >>>>>> duration >>>>>> > of >>>>>> > > > > sync >>>>>> > > > > > >> checkpoint process in Kinesis sink operator, which would >>>>>> block >>>>>> > the >>>>>> > > > > > normal >>>>>> > > > > > >> data processing until back pressure the source. >>>>>> > > > > > >> > Maybe we could dig into the process of sync execution >>>>>> in >>>>>> > > > checkpoint. >>>>>> > > > > > >> E.g. break down the steps inside respective >>>>>> > operator#snapshotState >>>>>> > > > to >>>>>> > > > > > >> statistic which operation cost most of the time, then >>>>>> > > > > > >> > we might probably find the root cause to bring such >>>>>> cost. >>>>>> > > > > > >> > >>>>>> > > > > > >> > Look forward to the further progress. :) >>>>>> > > > > > >> > >>>>>> > > > > > >> > Best, >>>>>> > > > > > >> > Zhijiang >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > ------------------------------------------------------------------ >>>>>> > > > > > >> > From:Stephan Ewen <se...@apache.org> >>>>>> > > > > > >> > Send Time:2020年7月7日(星期二) 14:52 >>>>>> > > > > > >> > To:Thomas Weise <t...@apache.org> >>>>>> > > > > > >> > Cc:Stephan Ewen <se...@apache.org>; Zhijiang < >>>>>> > > > > > >> wangzhijiang...@aliyun.com>; Aljoscha Krettek < >>>>>> > > aljos...@apache.org >>>>>> > > > >; >>>>>> > > > > > >> Arvid Heise <ar...@ververica.com> >>>>>> > > > > > >> > Subject:Re: Kinesis Performance Issue (was [VOTE] >>>>>> Release >>>>>> > > 1.11.0, >>>>>> > > > > > >> release candidate #4) >>>>>> > > > > > >> > >>>>>> > > > > > >> > Thank you for the digging so deeply. >>>>>> > > > > > >> > Mysterious think this regression. >>>>>> > > > > > >> > >>>>>> > > > > > >> > On Mon, Jul 6, 2020, 22:56 Thomas Weise < >>>>>> t...@apache.org> >>>>>> > wrote: >>>>>> > > > > > >> > @Stephan: yes, I refer to sync time in the web UI (it >>>>>> is >>>>>> > > unchanged >>>>>> > > > > > >> between 1.10 and 1.11 for the specific pipeline). >>>>>> > > > > > >> > >>>>>> > > > > > >> > I verified that increasing the checkpointing interval >>>>>> does not >>>>>> > > > make >>>>>> > > > > a >>>>>> > > > > > >> difference. >>>>>> > > > > > >> > >>>>>> > > > > > >> > I looked at the Kinesis connector changes since 1.10.1 >>>>>> and >>>>>> > don't >>>>>> > > > see >>>>>> > > > > > >> anything that could cause this. >>>>>> > > > > > >> > >>>>>> > > > > > >> > Another pipeline that is using the Kinesis consumer >>>>>> (but not >>>>>> > the >>>>>> > > > > > >> producer) performs as expected. >>>>>> > > > > > >> > >>>>>> > > > > > >> > I tried reverting the AWS SDK version change, symptoms >>>>>> remain >>>>>> > > > > > unchanged: >>>>>> > > > > > >> > >>>>>> > > > > > >> > diff --git >>>>>> a/flink-connectors/flink-connector-kinesis/pom.xml >>>>>> > > > > > >> b/flink-connectors/flink-connector-kinesis/pom.xml >>>>>> > > > > > >> > index a6abce23ba..741743a05e 100644 >>>>>> > > > > > >> > --- a/flink-connectors/flink-connector-kinesis/pom.xml >>>>>> > > > > > >> > +++ b/flink-connectors/flink-connector-kinesis/pom.xml >>>>>> > > > > > >> > @@ -33,7 +33,7 @@ under the License. >>>>>> > > > > > >> > >>>>>> > > > > > >> >>>>>> > > > > >>>>>> > > >>>>>> <artifactId>flink-connector-kinesis_${scala.binary.version}</artifactId> >>>>>> > > > > > >> > <name>flink-connector-kinesis</name> >>>>>> > > > > > >> > <properties> >>>>>> > > > > > >> > - >>>>>> <aws.sdk.version>1.11.754</aws.sdk.version> >>>>>> > > > > > >> > + >>>>>> <aws.sdk.version>1.11.603</aws.sdk.version> >>>>>> > > > > > >> > >>>>>> > > > > > >> <aws.kinesis-kcl.version>1.11.2</aws.kinesis-kcl.version> >>>>>> > > > > > >> > >>>>>> > > > > > >> <aws.kinesis-kpl.version>0.14.0</aws.kinesis-kpl.version> >>>>>> > > > > > >> > >>>>>> > > > > > >> >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > >>>>>> > >>>>>> <aws.dynamodbstreams-kinesis-adapter.version>1.5.0</aws.dynamodbstreams-kinesis-adapter.version> >>>>>> > > > > > >> > >>>>>> > > > > > >> > I'm planning to take a look with a profiler next. >>>>>> > > > > > >> > >>>>>> > > > > > >> > Thomas >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> > On Mon, Jul 6, 2020 at 11:40 AM Stephan Ewen < >>>>>> > se...@apache.org> >>>>>> > > > > > wrote: >>>>>> > > > > > >> > Hi all! >>>>>> > > > > > >> > >>>>>> > > > > > >> > Forking this thread out of the release vote thread. >>>>>> > > > > > >> > From what Thomas describes, it really sounds like a >>>>>> > > sink-specific >>>>>> > > > > > >> issue. >>>>>> > > > > > >> > >>>>>> > > > > > >> > @Thomas: When you say sink has a long synchronous >>>>>> checkpoint >>>>>> > > time, >>>>>> > > > > you >>>>>> > > > > > >> mean the time that is shown as "sync time" on the >>>>>> metrics and >>>>>> > web >>>>>> > > > UI? >>>>>> > > > > > That >>>>>> > > > > > >> is not including any network buffer related operations. >>>>>> It is >>>>>> > > purely >>>>>> > > > > the >>>>>> > > > > > >> operator's time. >>>>>> > > > > > >> > >>>>>> > > > > > >> > Can we dig into the changes we did in sinks: >>>>>> > > > > > >> > - Kinesis version upgrade, AWS library updates >>>>>> > > > > > >> > >>>>>> > > > > > >> > - Could it be that some call (checkpoint complete) >>>>>> that was >>>>>> > > > > > >> previously (1.10) in a separate thread is not in the >>>>>> mailbox and >>>>>> > > > this >>>>>> > > > > > >> simply reduces the number of threads that do the work? >>>>>> > > > > > >> > >>>>>> > > > > > >> > - Did sink checkpoint notifications change in a >>>>>> relevant >>>>>> > way, >>>>>> > > > for >>>>>> > > > > > >> example due to some Kafka issues we addressed in 1.11 >>>>>> (@Aljoscha >>>>>> > > > > maybe?) >>>>>> > > > > > >> > >>>>>> > > > > > >> > Best, >>>>>> > > > > > >> > Stephan >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> > On Sun, Jul 5, 2020 at 7:10 AM Zhijiang < >>>>>> > > > wangzhijiang...@aliyun.com >>>>>> > > > > > .invalid> >>>>>> > > > > > >> wrote: >>>>>> > > > > > >> > Hi Thomas, >>>>>> > > > > > >> > >>>>>> > > > > > >> > Regarding [2], it has more detail infos in the Jira >>>>>> > > description >>>>>> > > > ( >>>>>> > > > > > >> https://issues.apache.org/jira/browse/FLINK-16404). >>>>>> > > > > > >> > >>>>>> > > > > > >> > I can also give some basic explanations here to >>>>>> dismiss the >>>>>> > > > > concern. >>>>>> > > > > > >> > 1. In the past, the following buffers after the >>>>>> barrier will >>>>>> > > be >>>>>> > > > > > >> cached on downstream side before alignment. >>>>>> > > > > > >> > 2. In 1.11, the upstream would not send the buffers >>>>>> after >>>>>> > the >>>>>> > > > > > >> barrier. When the downstream finishes the alignment, it >>>>>> will >>>>>> > > notify >>>>>> > > > > the >>>>>> > > > > > >> downstream of continuing sending following buffers, >>>>>> since it can >>>>>> > > > > process >>>>>> > > > > > >> them after alignment. >>>>>> > > > > > >> > 3. The only difference is that the temporary blocked >>>>>> buffers >>>>>> > > are >>>>>> > > > > > >> cached either on downstream side or on upstream side >>>>>> before >>>>>> > > > alignment. >>>>>> > > > > > >> > 4. The side effect would be the additional >>>>>> notification cost >>>>>> > > for >>>>>> > > > > > >> every barrier alignment. If the downstream and upstream >>>>>> are >>>>>> > > deployed >>>>>> > > > > in >>>>>> > > > > > >> separate TaskManager, the cost is network transport >>>>>> delay (the >>>>>> > > > effect >>>>>> > > > > > can >>>>>> > > > > > >> be ignored based on our testing with 1s checkpoint >>>>>> interval). >>>>>> > For >>>>>> > > > > > sharing >>>>>> > > > > > >> slot in your case, the cost is only one method call in >>>>>> > processor, >>>>>> > > > can >>>>>> > > > > be >>>>>> > > > > > >> ignored also. >>>>>> > > > > > >> > >>>>>> > > > > > >> > You mentioned "In this case, the downstream task has >>>>>> a high >>>>>> > > > > average >>>>>> > > > > > >> checkpoint duration(~30s, sync part)." This duration is >>>>>> not >>>>>> > > > reflecting >>>>>> > > > > > the >>>>>> > > > > > >> changes above, and it is only indicating the duration for >>>>>> > calling >>>>>> > > > > > >> `Operation.snapshotState`. >>>>>> > > > > > >> > If this duration is beyond your expectation, you can >>>>>> check >>>>>> > or >>>>>> > > > > debug >>>>>> > > > > > >> whether the source/sink operations might take more time >>>>>> to >>>>>> > finish >>>>>> > > > > > >> `snapshotState` in practice. E.g. you can >>>>>> > > > > > >> > make the implementation of this method as empty to >>>>>> further >>>>>> > > > verify >>>>>> > > > > > the >>>>>> > > > > > >> effect. >>>>>> > > > > > >> > >>>>>> > > > > > >> > Best, >>>>>> > > > > > >> > Zhijiang >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > >>>>>> ------------------------------------------------------------------ >>>>>> > > > > > >> > From:Thomas Weise <t...@apache.org> >>>>>> > > > > > >> > Send Time:2020年7月5日(星期日) 12:22 >>>>>> > > > > > >> > To:dev <dev@flink.apache.org>; Zhijiang < >>>>>> > > > > wangzhijiang...@aliyun.com >>>>>> > > > > > > >>>>>> > > > > > >> > Cc:Yingjie Cao <kevin.ying...@gmail.com> >>>>>> > > > > > >> > Subject:Re: [VOTE] Release 1.11.0, release candidate >>>>>> #4 >>>>>> > > > > > >> > >>>>>> > > > > > >> > Hi Zhijiang, >>>>>> > > > > > >> > >>>>>> > > > > > >> > Could you please point me to more details regarding: >>>>>> "[2]: >>>>>> > > Delay >>>>>> > > > > > send >>>>>> > > > > > >> the >>>>>> > > > > > >> > following buffers after checkpoint barrier on >>>>>> upstream side >>>>>> > > > until >>>>>> > > > > > >> barrier >>>>>> > > > > > >> > alignment on downstream side." >>>>>> > > > > > >> > >>>>>> > > > > > >> > In this case, the downstream task has a high average >>>>>> > > checkpoint >>>>>> > > > > > >> duration >>>>>> > > > > > >> > (~30s, sync part). If there was a change to hold >>>>>> buffers >>>>>> > > > depending >>>>>> > > > > > on >>>>>> > > > > > >> > downstream performance, could this possibly apply to >>>>>> this >>>>>> > case >>>>>> > > > > (even >>>>>> > > > > > >> when >>>>>> > > > > > >> > there is no shuffle that would require alignment)? >>>>>> > > > > > >> > >>>>>> > > > > > >> > Thanks, >>>>>> > > > > > >> > Thomas >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> > On Sat, Jul 4, 2020 at 7:39 AM Zhijiang < >>>>>> > > > > wangzhijiang...@aliyun.com >>>>>> > > > > > >> .invalid> >>>>>> > > > > > >> > wrote: >>>>>> > > > > > >> > >>>>>> > > > > > >> > > Hi Thomas, >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > Thanks for the further update information. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > I guess we can dismiss the network stack changes, >>>>>> since in >>>>>> > > > your >>>>>> > > > > > >> case the >>>>>> > > > > > >> > > downstream and upstream would probably be deployed >>>>>> in the >>>>>> > > same >>>>>> > > > > > slot >>>>>> > > > > > >> > > bypassing the network data shuffle. >>>>>> > > > > > >> > > Also I guess release-1.11 will not bring general >>>>>> > performance >>>>>> > > > > > >> regression in >>>>>> > > > > > >> > > runtime engine, as we also did the performance >>>>>> testing for >>>>>> > > all >>>>>> > > > > > >> general >>>>>> > > > > > >> > > cases by [1] in real cluster before and the testing >>>>>> > results >>>>>> > > > > should >>>>>> > > > > > >> fit the >>>>>> > > > > > >> > > expectation. But we indeed did not test the >>>>>> specific >>>>>> > source >>>>>> > > > and >>>>>> > > > > > sink >>>>>> > > > > > >> > > connectors yet as I known. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > Regarding your performance regression with 40%, I >>>>>> wonder >>>>>> > it >>>>>> > > is >>>>>> > > > > > >> probably >>>>>> > > > > > >> > > related to specific source/sink changes (e.g. >>>>>> kinesis) or >>>>>> > > > > > >> environment >>>>>> > > > > > >> > > issues with corner case. >>>>>> > > > > > >> > > If possible, it would be helpful to further locate >>>>>> whether >>>>>> > > the >>>>>> > > > > > >> regression >>>>>> > > > > > >> > > is caused by kinesis, by replacing the kinesis >>>>>> source & >>>>>> > sink >>>>>> > > > and >>>>>> > > > > > >> keeping >>>>>> > > > > > >> > > the others same. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > As you said, it would be efficient to contact with >>>>>> you >>>>>> > > > directly >>>>>> > > > > > >> next week >>>>>> > > > > > >> > > to further discuss this issue. And we are >>>>>> willing/eager to >>>>>> > > > > provide >>>>>> > > > > > >> any help >>>>>> > > > > > >> > > to resolve this issue soon. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > Besides that, I guess this issue should not be the >>>>>> blocker >>>>>> > > for >>>>>> > > > > the >>>>>> > > > > > >> > > release, since it is probably a corner case based >>>>>> on the >>>>>> > > > current >>>>>> > > > > > >> analysis. >>>>>> > > > > > >> > > If we really conclude anything need to be resolved >>>>>> after >>>>>> > the >>>>>> > > > > final >>>>>> > > > > > >> > > release, then we can also make the next minor >>>>>> > release-1.11.1 >>>>>> > > > > come >>>>>> > > > > > >> soon. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > [1] >>>>>> https://issues.apache.org/jira/browse/FLINK-18433 >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > Best, >>>>>> > > > > > >> > > Zhijiang >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > >>>>>> ------------------------------------------------------------------ >>>>>> > > > > > >> > > From:Thomas Weise <t...@apache.org> >>>>>> > > > > > >> > > Send Time:2020年7月4日(星期六) 12:26 >>>>>> > > > > > >> > > To:dev <dev@flink.apache.org>; Zhijiang < >>>>>> > > > > > wangzhijiang...@aliyun.com >>>>>> > > > > > >> > >>>>>> > > > > > >> > > Cc:Yingjie Cao <kevin.ying...@gmail.com> >>>>>> > > > > > >> > > Subject:Re: [VOTE] Release 1.11.0, release >>>>>> candidate #4 >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > Hi Zhijiang, >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > It will probably be best if we connect next week >>>>>> and >>>>>> > discuss >>>>>> > > > the >>>>>> > > > > > >> issue >>>>>> > > > > > >> > > directly since this could be quite difficult to >>>>>> reproduce. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > Before the testing result on our side comes out >>>>>> for your >>>>>> > > > > > respective >>>>>> > > > > > >> job >>>>>> > > > > > >> > > case, I have some other questions to confirm for >>>>>> further >>>>>> > > > > analysis: >>>>>> > > > > > >> > > - How much percentage regression you found >>>>>> after >>>>>> > > > switching >>>>>> > > > > to >>>>>> > > > > > >> 1.11? >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > ~40% throughput decline >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > - Are there any network bottleneck in your >>>>>> cluster? >>>>>> > > E.g. >>>>>> > > > > the >>>>>> > > > > > >> network >>>>>> > > > > > >> > > bandwidth is full caused by other jobs? If so, it >>>>>> might >>>>>> > have >>>>>> > > > > more >>>>>> > > > > > >> effects >>>>>> > > > > > >> > > by above [2] >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > The test runs on a k8s cluster that is also used >>>>>> for other >>>>>> > > > > > >> production jobs. >>>>>> > > > > > >> > > There is no reason be believe network is the >>>>>> bottleneck. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > - Did you adjust the default network buffer >>>>>> setting? >>>>>> > > E.g. >>>>>> > > > > > >> > > >>>>>> "taskmanager.network.memory.floating-buffers-per-gate" or >>>>>> > > > > > >> > > "taskmanager.network.memory.buffers-per-channel" >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > The job is using the defaults, i.e we don't >>>>>> configure the >>>>>> > > > > > settings. >>>>>> > > > > > >> If you >>>>>> > > > > > >> > > want me to try specific settings in the hope that >>>>>> it will >>>>>> > > help >>>>>> > > > > to >>>>>> > > > > > >> isolate >>>>>> > > > > > >> > > the issue please let me know. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > - I guess the topology has three vertexes >>>>>> > > > "KinesisConsumer >>>>>> > > > > -> >>>>>> > > > > > >> Chained >>>>>> > > > > > >> > > FlatMap -> KinesisProducer", and the partition >>>>>> mode for >>>>>> > > > > > >> "KinesisConsumer -> >>>>>> > > > > > >> > > FlatMap" and "FlatMap->KinesisProducer" are both >>>>>> > "forward"? >>>>>> > > If >>>>>> > > > > so, >>>>>> > > > > > >> the edge >>>>>> > > > > > >> > > connection is one-to-one, not all-to-all, then the >>>>>> above >>>>>> > > > [1][2] >>>>>> > > > > > >> should no >>>>>> > > > > > >> > > effects in theory with default network buffer >>>>>> setting. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > There are only 2 vertices and the edge is >>>>>> "forward". >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > - By slot sharing, I guess these three vertex >>>>>> > > parallelism >>>>>> > > > > task >>>>>> > > > > > >> would >>>>>> > > > > > >> > > probably be deployed into the same slot, then the >>>>>> data >>>>>> > > shuffle >>>>>> > > > > is >>>>>> > > > > > >> by memory >>>>>> > > > > > >> > > queue, not network stack. If so, the above [2] >>>>>> should no >>>>>> > > > effect. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > Yes, vertices share slots. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > - I also saw some Jira changes for kinesis in >>>>>> this >>>>>> > > > release, >>>>>> > > > > > >> could you >>>>>> > > > > > >> > > confirm that these changes would not effect the >>>>>> > performance? >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > I will need to take a look. 1.10 already had a >>>>>> regression >>>>>> > > > > > >> introduced by the >>>>>> > > > > > >> > > Kinesis producer update. >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > Thanks, >>>>>> > > > > > >> > > Thomas >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > On Thu, Jul 2, 2020 at 11:46 PM Zhijiang < >>>>>> > > > > > >> wangzhijiang...@aliyun.com >>>>>> > > > > > >> > > .invalid> >>>>>> > > > > > >> > > wrote: >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > > Hi Thomas, >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Thanks for your reply with rich information! >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > We are trying to reproduce your case in our >>>>>> cluster to >>>>>> > > > further >>>>>> > > > > > >> verify it, >>>>>> > > > > > >> > > > and @Yingjie Cao is working on it now. >>>>>> > > > > > >> > > > As we have not kinesis consumer and producer >>>>>> > internally, >>>>>> > > so >>>>>> > > > > we >>>>>> > > > > > >> will >>>>>> > > > > > >> > > > construct the common source and sink instead in >>>>>> the case >>>>>> > > of >>>>>> > > > > > >> backpressure. >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Firstly, we can dismiss the rockdb factor in this >>>>>> > release, >>>>>> > > > > since >>>>>> > > > > > >> you also >>>>>> > > > > > >> > > > mentioned that "filesystem leads to same >>>>>> symptoms". >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Secondly, if my understanding is right, you >>>>>> emphasis >>>>>> > that >>>>>> > > > the >>>>>> > > > > > >> regression >>>>>> > > > > > >> > > > only exists for the jobs with low checkpoint >>>>>> interval >>>>>> > > (10s). >>>>>> > > > > > >> > > > Based on that, I have two suspicions with the >>>>>> network >>>>>> > > > related >>>>>> > > > > > >> changes in >>>>>> > > > > > >> > > > this release: >>>>>> > > > > > >> > > > - [1]: Limited the maximum backlog value >>>>>> (default >>>>>> > 10) >>>>>> > > in >>>>>> > > > > > >> subpartition >>>>>> > > > > > >> > > > queue. >>>>>> > > > > > >> > > > - [2]: Delay send the following buffers after >>>>>> > > checkpoint >>>>>> > > > > > >> barrier on >>>>>> > > > > > >> > > > upstream side until barrier alignment on >>>>>> downstream >>>>>> > side. >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > These changes are motivated for reducing the >>>>>> in-flight >>>>>> > > > buffers >>>>>> > > > > > to >>>>>> > > > > > >> speedup >>>>>> > > > > > >> > > > checkpoint especially in the case of >>>>>> backpressure. >>>>>> > > > > > >> > > > In theory they should have very minor >>>>>> performance effect >>>>>> > > and >>>>>> > > > > > >> actually we >>>>>> > > > > > >> > > > also tested in cluster to verify within >>>>>> expectation >>>>>> > before >>>>>> > > > > > >> merging them, >>>>>> > > > > > >> > > > but maybe there are other corner cases we have >>>>>> not >>>>>> > > thought >>>>>> > > > of >>>>>> > > > > > >> before. >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Before the testing result on our side comes out >>>>>> for your >>>>>> > > > > > >> respective job >>>>>> > > > > > >> > > > case, I have some other questions to confirm for >>>>>> further >>>>>> > > > > > analysis: >>>>>> > > > > > >> > > > - How much percentage regression you found >>>>>> after >>>>>> > > > > switching >>>>>> > > > > > >> to 1.11? >>>>>> > > > > > >> > > > - Are there any network bottleneck in your >>>>>> cluster? >>>>>> > > > E.g. >>>>>> > > > > > the >>>>>> > > > > > >> network >>>>>> > > > > > >> > > > bandwidth is full caused by other jobs? If so, >>>>>> it might >>>>>> > > have >>>>>> > > > > > more >>>>>> > > > > > >> effects >>>>>> > > > > > >> > > > by above [2] >>>>>> > > > > > >> > > > - Did you adjust the default network buffer >>>>>> > setting? >>>>>> > > > E.g. >>>>>> > > > > > >> > > > >>>>>> "taskmanager.network.memory.floating-buffers-per-gate" >>>>>> > or >>>>>> > > > > > >> > > > "taskmanager.network.memory.buffers-per-channel" >>>>>> > > > > > >> > > > - I guess the topology has three vertexes >>>>>> > > > > "KinesisConsumer >>>>>> > > > > > -> >>>>>> > > > > > >> > > Chained >>>>>> > > > > > >> > > > FlatMap -> KinesisProducer", and the partition >>>>>> mode for >>>>>> > > > > > >> "KinesisConsumer >>>>>> > > > > > >> > > -> >>>>>> > > > > > >> > > > FlatMap" and "FlatMap->KinesisProducer" are both >>>>>> > > "forward"? >>>>>> > > > If >>>>>> > > > > > >> so, the >>>>>> > > > > > >> > > edge >>>>>> > > > > > >> > > > connection is one-to-one, not all-to-all, then >>>>>> the above >>>>>> > > > > [1][2] >>>>>> > > > > > >> should no >>>>>> > > > > > >> > > > effects in theory with default network buffer >>>>>> setting. >>>>>> > > > > > >> > > > - By slot sharing, I guess these three vertex >>>>>> > > > parallelism >>>>>> > > > > > >> task would >>>>>> > > > > > >> > > > probably be deployed into the same slot, then >>>>>> the data >>>>>> > > > shuffle >>>>>> > > > > > is >>>>>> > > > > > >> by >>>>>> > > > > > >> > > memory >>>>>> > > > > > >> > > > queue, not network stack. If so, the above [2] >>>>>> should no >>>>>> > > > > effect. >>>>>> > > > > > >> > > > - I also saw some Jira changes for kinesis >>>>>> in this >>>>>> > > > > release, >>>>>> > > > > > >> could you >>>>>> > > > > > >> > > > confirm that these changes would not effect the >>>>>> > > performance? >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Best, >>>>>> > > > > > >> > > > Zhijiang >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >>>>>> ------------------------------------------------------------------ >>>>>> > > > > > >> > > > From:Thomas Weise <t...@apache.org> >>>>>> > > > > > >> > > > Send Time:2020年7月3日(星期五) 01:07 >>>>>> > > > > > >> > > > To:dev <dev@flink.apache.org>; Zhijiang < >>>>>> > > > > > >> wangzhijiang...@aliyun.com> >>>>>> > > > > > >> > > > Subject:Re: [VOTE] Release 1.11.0, release >>>>>> candidate #4 >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Hi Zhijiang, >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > The performance degradation manifests in >>>>>> backpressure >>>>>> > > which >>>>>> > > > > > leads >>>>>> > > > > > >> to >>>>>> > > > > > >> > > > growing backlog in the source. I switched a few >>>>>> times >>>>>> > > > between >>>>>> > > > > > >> 1.10 and >>>>>> > > > > > >> > > 1.11 >>>>>> > > > > > >> > > > and the behavior is consistent. >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > The DAG is: >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > KinesisConsumer -> (Flat Map, Flat Map, Flat Map) >>>>>> > > -------- >>>>>> > > > > > >> forward >>>>>> > > > > > >> > > > ---------> KinesisProducer >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Parallelism: 160 >>>>>> > > > > > >> > > > No shuffle/rebalance. >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Checkpointing config: >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Checkpointing Mode Exactly Once >>>>>> > > > > > >> > > > Interval 10s >>>>>> > > > > > >> > > > Timeout 10m 0s >>>>>> > > > > > >> > > > Minimum Pause Between Checkpoints 10s >>>>>> > > > > > >> > > > Maximum Concurrent Checkpoints 1 >>>>>> > > > > > >> > > > Persist Checkpoints Externally Enabled (delete on >>>>>> > > > > cancellation) >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > State backend: rocksdb (filesystem leads to same >>>>>> > > symptoms) >>>>>> > > > > > >> > > > Checkpoint size is tiny (500KB) >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > An interesting difference to another job that I >>>>>> had >>>>>> > > upgraded >>>>>> > > > > > >> successfully >>>>>> > > > > > >> > > > is the low checkpointing interval. >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Thanks, >>>>>> > > > > > >> > > > Thomas >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > On Wed, Jul 1, 2020 at 9:02 PM Zhijiang < >>>>>> > > > > > >> wangzhijiang...@aliyun.com >>>>>> > > > > > >> > > > .invalid> >>>>>> > > > > > >> > > > wrote: >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > > Hi Thomas, >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > Thanks for the efficient feedback. >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > Regarding the suggestion of adding the release >>>>>> notes >>>>>> > > > > document, >>>>>> > > > > > >> I agree >>>>>> > > > > > >> > > > > with your point. Maybe we should adjust the >>>>>> vote >>>>>> > > template >>>>>> > > > > > >> accordingly >>>>>> > > > > > >> > > in >>>>>> > > > > > >> > > > > the respective wiki to guide the following >>>>>> release >>>>>> > > > > processes. >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > Regarding the performance regression, could you >>>>>> > provide >>>>>> > > > some >>>>>> > > > > > >> more >>>>>> > > > > > >> > > details >>>>>> > > > > > >> > > > > for our better measurement or reproducing on >>>>>> our >>>>>> > sides? >>>>>> > > > > > >> > > > > E.g. I guess the topology only includes two >>>>>> vertexes >>>>>> > > > source >>>>>> > > > > > and >>>>>> > > > > > >> sink? >>>>>> > > > > > >> > > > > What is the parallelism for every vertex? >>>>>> > > > > > >> > > > > The upstream shuffles data to the downstream >>>>>> via >>>>>> > > rebalance >>>>>> > > > > > >> partitioner >>>>>> > > > > > >> > > or >>>>>> > > > > > >> > > > > other? >>>>>> > > > > > >> > > > > The checkpoint mode is exactly-once with >>>>>> rocksDB state >>>>>> > > > > > backend? >>>>>> > > > > > >> > > > > The backpressure happened in this case? >>>>>> > > > > > >> > > > > How much percentage regression in this case? >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > Best, >>>>>> > > > > > >> > > > > Zhijiang >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> >>>>>> > ------------------------------------------------------------------ >>>>>> > > > > > >> > > > > From:Thomas Weise <t...@apache.org> >>>>>> > > > > > >> > > > > Send Time:2020年7月2日(星期四) 09:54 >>>>>> > > > > > >> > > > > To:dev <dev@flink.apache.org> >>>>>> > > > > > >> > > > > Subject:Re: [VOTE] Release 1.11.0, release >>>>>> candidate >>>>>> > #4 >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > Hi Till, >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > Yes, we don't have the setting in >>>>>> flink-conf.yaml. >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > Generally, we carry forward the existing >>>>>> configuration >>>>>> > > and >>>>>> > > > > any >>>>>> > > > > > >> change >>>>>> > > > > > >> > > to >>>>>> > > > > > >> > > > > default configuration values would impact the >>>>>> upgrade. >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > Yes, since it is an incompatible change I >>>>>> would state >>>>>> > it >>>>>> > > > in >>>>>> > > > > > the >>>>>> > > > > > >> release >>>>>> > > > > > >> > > > > notes. >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > Thanks, >>>>>> > > > > > >> > > > > Thomas >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > BTW I found a performance regression while >>>>>> trying to >>>>>> > > > upgrade >>>>>> > > > > > >> another >>>>>> > > > > > >> > > > > pipeline with this RC. It is a simple Kinesis >>>>>> to >>>>>> > Kinesis >>>>>> > > > > job. >>>>>> > > > > > >> Wasn't >>>>>> > > > > > >> > > able >>>>>> > > > > > >> > > > > to pin it down yet, symptoms include increased >>>>>> > > checkpoint >>>>>> > > > > > >> alignment >>>>>> > > > > > >> > > time. >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann < >>>>>> > > > > > >> trohrm...@apache.org> >>>>>> > > > > > >> > > > > wrote: >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > > Hi Thomas, >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > > just to confirm: When starting the image in >>>>>> local >>>>>> > > mode, >>>>>> > > > > then >>>>>> > > > > > >> you >>>>>> > > > > > >> > > don't >>>>>> > > > > > >> > > > > have >>>>>> > > > > > >> > > > > > any of the JobManager memory configuration >>>>>> settings >>>>>> > > > > > >> configured in the >>>>>> > > > > > >> > > > > > effective flink-conf.yaml, right? Does this >>>>>> mean >>>>>> > that >>>>>> > > > you >>>>>> > > > > > have >>>>>> > > > > > >> > > > explicitly >>>>>> > > > > > >> > > > > > removed `jobmanager.heap.size: 1024m` from >>>>>> the >>>>>> > default >>>>>> > > > > > >> configuration? >>>>>> > > > > > >> > > > If >>>>>> > > > > > >> > > > > > this is the case, then I believe it was more >>>>>> of an >>>>>> > > > > > >> unintentional >>>>>> > > > > > >> > > > artifact >>>>>> > > > > > >> > > > > > that it worked before and it has been >>>>>> corrected now >>>>>> > so >>>>>> > > > > that >>>>>> > > > > > >> one needs >>>>>> > > > > > >> > > > to >>>>>> > > > > > >> > > > > > specify the memory of the JM process >>>>>> explicitly. Do >>>>>> > > you >>>>>> > > > > > think >>>>>> > > > > > >> it >>>>>> > > > > > >> > > would >>>>>> > > > > > >> > > > > help >>>>>> > > > > > >> > > > > > to explicitly state this in the release >>>>>> notes? >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > > Cheers, >>>>>> > > > > > >> > > > > > Till >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise < >>>>>> > > > > t...@apache.org >>>>>> > > > > > > >>>>>> > > > > > >> wrote: >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > > > Thanks for preparing another RC! >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > > As mentioned in the previous RC thread, it >>>>>> would >>>>>> > be >>>>>> > > > > super >>>>>> > > > > > >> helpful >>>>>> > > > > > >> > > if >>>>>> > > > > > >> > > > > the >>>>>> > > > > > >> > > > > > > release notes that are part of the >>>>>> documentation >>>>>> > can >>>>>> > > > be >>>>>> > > > > > >> included >>>>>> > > > > > >> > > [1]. >>>>>> > > > > > >> > > > > > It's >>>>>> > > > > > >> > > > > > > a significant time-saver to have read >>>>>> those first. >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > > I found one more non-backward compatible >>>>>> change >>>>>> > that >>>>>> > > > > would >>>>>> > > > > > >> be worth >>>>>> > > > > > >> > > > > > > addressing/mentioning: >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > > It is now necessary to configure the >>>>>> jobmanager >>>>>> > heap >>>>>> > > > > size >>>>>> > > > > > in >>>>>> > > > > > >> > > > > > > flink-conf.yaml (with either >>>>>> jobmanager.heap.size >>>>>> > > > > > >> > > > > > > or jobmanager.memory.heap.size). Why would >>>>>> I not >>>>>> > > want >>>>>> > > > to >>>>>> > > > > > do >>>>>> > > > > > >> that >>>>>> > > > > > >> > > > > anyways? >>>>>> > > > > > >> > > > > > > Well, we set it dynamically for a cluster >>>>>> > deployment >>>>>> > > > via >>>>>> > > > > > the >>>>>> > > > > > >> > > > > > > flinkk8soperator, but the container image >>>>>> can also >>>>>> > > be >>>>>> > > > > used >>>>>> > > > > > >> for >>>>>> > > > > > >> > > > testing >>>>>> > > > > > >> > > > > > with >>>>>> > > > > > >> > > > > > > local mode (./bin/jobmanager.sh >>>>>> start-foreground >>>>>> > > > local). >>>>>> > > > > > >> That will >>>>>> > > > > > >> > > > fail >>>>>> > > > > > >> > > > > > if >>>>>> > > > > > >> > > > > > > the heap wasn't configured and that's how I >>>>>> > noticed >>>>>> > > > it. >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > > Thanks, >>>>>> > > > > > >> > > > > > > Thomas >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > > [1] >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > >>>>>> > >>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang < >>>>>> > > > > > >> > > wangzhijiang...@aliyun.com >>>>>> > > > > > >> > > > > > > .invalid> >>>>>> > > > > > >> > > > > > > wrote: >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > > > Hi everyone, >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > > Please review and vote on the release >>>>>> candidate >>>>>> > #4 >>>>>> > > > for >>>>>> > > > > > the >>>>>> > > > > > >> > > version >>>>>> > > > > > >> > > > > > > 1.11.0, >>>>>> > > > > > >> > > > > > > > as follows: >>>>>> > > > > > >> > > > > > > > [ ] +1, Approve the release >>>>>> > > > > > >> > > > > > > > [ ] -1, Do not approve the release >>>>>> (please >>>>>> > provide >>>>>> > > > > > >> specific >>>>>> > > > > > >> > > > comments) >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > > The complete staging area is available >>>>>> for your >>>>>> > > > > review, >>>>>> > > > > > >> which >>>>>> > > > > > >> > > > > includes: >>>>>> > > > > > >> > > > > > > > * JIRA release notes [1], >>>>>> > > > > > >> > > > > > > > * the official Apache source release and >>>>>> binary >>>>>> > > > > > >> convenience >>>>>> > > > > > >> > > > releases >>>>>> > > > > > >> > > > > to >>>>>> > > > > > >> > > > > > > be >>>>>> > > > > > >> > > > > > > > deployed to dist.apache.org [2], which >>>>>> are >>>>>> > signed >>>>>> > > > > with >>>>>> > > > > > >> the key >>>>>> > > > > > >> > > > with >>>>>> > > > > > >> > > > > > > > fingerprint >>>>>> > > 2DA85B93244FDFA19A6244500653C0A2CEA00D0E >>>>>> > > > > > [3], >>>>>> > > > > > >> > > > > > > > * all artifacts to be deployed to the >>>>>> Maven >>>>>> > > Central >>>>>> > > > > > >> Repository >>>>>> > > > > > >> > > [4], >>>>>> > > > > > >> > > > > > > > * source code tag "release-1.11.0-rc4" >>>>>> [5], >>>>>> > > > > > >> > > > > > > > * website pull request listing the new >>>>>> release >>>>>> > and >>>>>> > > > > > adding >>>>>> > > > > > >> > > > > announcement >>>>>> > > > > > >> > > > > > > > blog post [6]. >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > > The vote will be open for at least 72 >>>>>> hours. It >>>>>> > is >>>>>> > > > > > >> adopted by >>>>>> > > > > > >> > > > > majority >>>>>> > > > > > >> > > > > > > > approval, with at least 3 PMC >>>>>> affirmative votes. >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > > Thanks, >>>>>> > > > > > >> > > > > > > > Release Manager >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > > [1] >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > >>>>>> > >>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364 >>>>>> > > > > > >> > > > > > > > [2] >>>>>> > > > > > >> > > >>>>>> > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/ >>>>>> > > > > > >> > > > > > > > [3] >>>>>> > > > > > https://dist.apache.org/repos/dist/release/flink/KEYS >>>>>> > > > > > >> > > > > > > > [4] >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> >>>>>> > > > > >>>>>> > > >>>>>> https://repository.apache.org/content/repositories/orgapacheflink-1377/ >>>>>> > > > > > >> > > > > > > > [5] >>>>>> > > > > > >> > > > >>>>>> > > > > >>>>>> https://github.com/apache/flink/releases/tag/release-1.11.0-rc4 >>>>>> > > > > > >> > > > > > > > [6] >>>>>> > https://github.com/apache/flink-web/pull/352 >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> >>>>>> > > > > > >> >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Regards, >>>>>> > Roman >>>>>> > >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Roman >>>>> >>>> >> >> -- >> Regards, >> Roman >> > > > -- > Regards, > Roman >
