o verify?
> >>>>>> > > > > > [1] https://issues.apache.org/jira/browse/FLINK-17496
> >>>>>> > > > > > [2] https://issues.apache.org/jira/browse/FLINK-14881
> >>>>>> > > > > >
> >>>&g
---
>>>>>> > > > > > From:Thomas Weise
>>>>>> > > > > > Send Time:2020年7月17日(星期五) 05:29
>>>>>> > > > > > To:dev
>>>>>> > > > >
;> > > > > > > I will investigate further today.
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > On Wed, Jul 8, 2020 at 4:42 AM Aljoscha Krettek <
>>>>> >
;>> wangzhijiang...@aliyun.com
>>>> > > > > .invalid>
>>>> > > > > wrote:
>>>> > > > >
>>>> > > > > > Hi Thomas,
>>>> > > > > >
>>>> > > > > &g
ctually I was also suspicious of the point of #snapshotState
>>> in
>>> > > > previous
>>> > > > > > discussions since it indeed cost much time to block normal
>>> operator
>>> > > > > > processing.
>>> >
fect.
>> > > > > > >> >
>> > > > > > >> > I think we need to find out whether it has other symptoms
>> > > changed
>> > > > > > >> besides the performance regression to further fig
> > > >
> > > > > > > + dev@ for visibility
> > > > > > >
> > > > > > > I will investigate further today.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jul
t; > > operations
> > > > > inside KinesisProducer implementation provided by amazonaws, which
> I
> > am
> > > > not
> > > > > quite familiar with.
> > > > > But I noticed there were two upgrades related to it in
>
upgrading amazon-kinesis-producer to 0.14.0 [1] and another is
> > for
> > > > upgrading aws-sdk-version to 1.11.754 [2].
> > > > You mentioned that you already reverted the SDK upgrade to verify no
> > > > changes. Did you also revert the [1] to verify?
&
the [1] to verify?
> > > [1] https://issues.apache.org/jira/browse/FLINK-17496
> > > [2] https://issues.apache.org/jira/browse/FLINK-14881
> > >
> > > Best,
> > > Zhijiang
> > > --
; You mentioned that you already reverted the SDK upgrade to verify no
> > changes. Did you also revert the [1] to verify?
> > [1] https://issues.apache.org/jira/browse/FLINK-17496
> > [2] https://issues.apache.org/jira/browse/FLINK-14881
> >
> > Best,
> > Zhijiang
> > --
>
From:Thomas Weise
> Send Time:2020年7月17日(星期五) 05:29
> To:dev
> Cc:Zhijiang ; Stephan Ewen ;
> Arvid Heise ; Aljoscha Krettek
> Subject:Re: Kinesis Performance Issue (was [VOTE] Release 1.11.0, release
> candidate #4)
>
> Sorry for the delay.
>
> I confirmed that the r
t:Re: Kinesis Performance Issue (was [VOTE] Release 1.11.0, release
candidate #4)
Sorry for the delay.
I confirmed that the regression is due to the sink (unsurprising, since
another job with the same consumer, but not the producer, runs as expected).
As promised I did CPU profiling on th
> Zhijiang
>> >
>> >
>> > --
>> > From:Thomas Weise
>> > Send Time:2020年7月7日(星期二) 23:01
>> > To:Stephan Ewen
>> > Cc:Aljoscha Krettek ; Arvid Heise <
>> ar...@v
Best,
> > Zhijiang
> >
> >
> > --
> > From:Thomas Weise
> > Send Time:2020年7月7日(星期二) 23:01
> > To:Stephan Ewen
> > Cc:Aljoscha Krettek ; Arvid Heise <
> ar...@ververica.com>; Zhijiang
&
I'm happy to announce that we have unanimously approved the 1.11.0 release.
There are 17 approving votes, 8 of which are binding:
- Stephan (binding)
- Till (binding)
- Aljoscha (binding)
- Robert (binding)
- Chesnay (binding)
- Dawid (binding)
- Jark (binding)
- Jincheng (binding)
- Leonard Xu (
result soon in a separate email.
>
> Best,
> Zhijiang
>
>
> --
> From:Jingsong Li
> Send Time:2020年7月6日(星期一) 12:11
> To:dev
> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>
> +1
期一) 12:11
To:dev
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
+1 (non-binding)
- verified signature and checksum
- build from source
- checked webui and log sanity
- played with filesystem and new connectors
- played with Hive connector
Best,
Jingsonga
On Mon, Jul 6, 2020 at 9:50 AM
; the
> > > changes above, and it is only indicating the duration for calling
> > > `Operation.snapshotState`.
> > > If this duration is beyond your expectation, you can check or debug
> > > whether the source/sink operations might take more time to finish
> > > `snapshotState` in practice. E.g. you can
> > >
the implementation of this method as empty to further verify the
> > effect.
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Thomas Weise
> > Send Time:2020年7月5日(星期日) 12:22
might take more time to finish
> `snapshotState` in practice. E.g. you can
> make the implementation of this method as empty to further verify the
> effect.
>
> Best,
> Zhijiang
>
>
> ----------
> From:Thomas Wei
ntation of this method as empty to further verify the effect.
Best,
Zhijiang
--
From:Thomas Weise
Send Time:2020年7月5日(星期日) 12:22
To:dev ; Zhijiang
Cc:Yingjie Cao
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
Hi Zhijia
r case based on the current
> analysis.
> > If we really conclude anything need to be resolved after the final
> > release, then we can also make the next minor release-1.11.1 come soon.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18433
> >
> > Bes
Best,
> Zhijiang
>
>
> ------
> From:Thomas Weise
> Send Time:2020年7月4日(星期六) 12:26
> To:dev ; Zhijiang
> Cc:Yingjie Cao
> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>
> Hi Zhijiang,
>
&g
elease-1.11.1 come soon.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18433
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Thomas Weise
> > Send Time:2020年7月4日(星期
e the next minor release-1.11.1 come soon.
>
> [1] https://issues.apache.org/jira/browse/FLINK-18433
>
> Best,
> Zhijiang
>
>
> ------
> From:Thomas Weise
> Send Time:2020年7月4日(星期六) 12:26
> To:dev ; Zhiji
iang
--
From:Thomas Weise
Send Time:2020年7月4日(星期六) 12:26
To:dev ; Zhijiang
Cc:Yingjie Cao
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
Hi Zhijiang,
It will probably be best if we connect next week and discuss the issue
directly since this could be quite difficult to reproduce.
Before
sharing, I guess these three vertex parallelism task would
> probably be deployed into the same slot, then the data shuffle is by memory
> queue, not network stack. If so, the above [2] should no effect.
> - I also saw some Jira changes for kinesis in this release, could you
> confi
; > other?
> > The checkpoint mode is exactly-once with rocksDB state backend?
> > The backpressure happened in this case?
> > How much percentage regression in this case?
> >
> > Best,
> > Zhijiang
> >
> >
> >
> > ---
+1 (non-binding)
- check wheel package consistency with the built from the source code
- test the built from the wheel package in mac os with python 3.6
- verify the performance for PyFlink UDFs including Python General UDF and
Pandas UDF
- test Python UDTF
Best,
Xingbo
Dian Fu 于2020年7月3日周五 下午8
+1 (non-binding)
- built from source with scala 2.11 successfully
- checked the signature and checksum of the binary packages
- installed PyFlink on MacOS, Windows and Linux successfully
- tested the functionality of Pandas UDF and the conversion between PyFlink
Table and Pandas DataFrame
- verif
+1(binding)
Checks:
- check wheel package consistency
- test the built from the wheel package
- checked the signature and checksum
- pip installed the Python package
`apache_flink-1.11.0-cp37-cp37m-macosx_10_9_x86_64.whl` successfully and
run a simple word count example successfully
+1 (non-binding)
- checked/verified signatures and hashes
- built from source sing scala 2.11 succeeded
- go through all issues which "fixVersion" property is 1.11.0, there is no
blocker.
- checked that there are no missing artifacts
- test SQL connector Elasticsearch7/JDBC/HBase/Kafka (new conne
; > other?
> > The checkpoint mode is exactly-once with rocksDB state backend?
> > The backpressure happened in this case?
> > How much percentage regression in this case?
> >
> > Best,
> > Zhijiang
> >
> >
> >
> > ---
By slot sharing, I guess these three vertex parallelism task would
>> probably be deployed into the same slot, then the data shuffle is by memory
>> queue, not network stack. If so, the above [2] should no effect.
>> - I also saw some Jira changes for kinesis in this release, could y
y be deployed into the same slot, then the data shuffle is by memory
> queue, not network stack. If so, the above [2] should no effect.
> - I also saw some Jira changes for kinesis in this release, could you
> confirm that these changes would not effect the performance?
>
> Bes
ack. If so, the above [2] should no effect.
- I also saw some Jira changes for kinesis in this release, could you
confirm that these changes would not effect the performance?
Best,
Zhijiang
--
From:Thomas Weise
Send Time:2020年7月3日(
oint. Maybe we should adjust the vote template accordingly in
>>>> the respective wiki to guide the following release processes.
>>>>
>>>> Regarding the performance regression, could you provide some more details
>>>> for our better measurement
n in this case?
Best,
Zhijiang
--
From:Thomas Weise
Send Time:2020年7月2日(星期四) 09:54
To:dev
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
Hi Till,
Yes, we don't have the setting in flink-conf.yaml.
Generally, we carry forward the existing configuration and any change to
default configura
gt; E.g. I guess the topology only includes two vertexes source and sink?
> > > What is the parallelism for every vertex?
> > > The upstream shuffles data to the downstream via rebalance partitioner or
> > > other?
> > > The checkpoint mode is exactly-once with rocksDB state
> > The upstream shuffles data to the downstream via rebalance partitioner or
> > other?
> > The checkpoint mode is exactly-once with rocksDB state backend?
> > The backpressure happened in this case?
> > How much percentage regression in this case?
> >
> >
int mode is exactly-once with rocksDB state backend?
> The backpressure happened in this case?
> How much percentage regression in this case?
>
> Best,
> Zhijiang
>
>
>
> --
> From:Thomas Weise
> S
egarding the performance regression, if possible we can reproduce to
>> >>> analysis the reason based on Thomas's feedback, then we can evaluate
>> its
>> >>> effect.
>> >>>
>> >>> Regarding the FLINK-18461, after syncing with Jark offline, the bug
>> would
>> >>> effect one o
rio is actually the most commonly used way by users.
> >>> My suggestion is to merge it into release-1.11 ATM since the PR already
> >>> open for review, then let's further finalize the conclusion later. If
> >> this
> >>> issue is the only one after RC4 goi
necessary to be resolved soon, then it is no doubt to cover
all
of them in next RC5.
Best,
Zhijiang
--
From:Till Rohrmann
Send Time:2020年7月2日(星期四) 16:46
To:dev
Cc:Zhijiang
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
ver it in next release-1.11.1 as Robert suggested, as we can prepare
> for
> > the next minor release soon. If there are other blockers issues during
> > voting and necessary to be resolved soon, then it is no doubt to cover
> all
> > of them in next RC5.
> >
> > Best,
> > Zh
other blockers issues during
voting and necessary to be resolved soon, then it is no doubt to cover all
of them in next RC5.
Best,
Zhijiang
--
From:Till Rohrmann
Send Time:2020年7月2日(星期四) 16:46
To:dev
Cc:Zhijiang
Subject:Re: [VOTE]
g and necessary to be resolved soon, then it is no doubt to cover all
> of them in next RC5.
>
> Best,
> Zhijiang
>
>
> ----------
> From:Till Rohrmann
> Send Time:2020年7月2日(星期四) 16:46
> To:dev
> Cc:Zhijiang
>
g and necessary to
be resolved soon, then it is no doubt to cover all of them in next RC5.
Best,
Zhijiang
--
From:Till Rohrmann
Send Time:2020年7月2日(星期四) 16:46
To:dev
Cc:Zhijiang
Subject:Re: [VOTE] Release 1.11.0, release candid
vertexes source and sink?
> > > What is the parallelism for every vertex?
> > > The upstream shuffles data to the downstream via rebalance partitioner
> or
> > > other?
> > > The checkpoint mode is exactly-once with rocksDB state backend?
> >
rebalance partitioner or
> > other?
> > The checkpoint mode is exactly-once with rocksDB state backend?
> > The backpressure happened in this case?
> > How much percentage regression in this case?
> >
> > Best,
> > Zhijiang
> >
> >
>
---
> From:Thomas Weise
> Send Time:2020年7月2日(星期四) 09:54
> To:dev
> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>
> Hi Till,
>
> Yes, we don't have the setting in flink-conf.yaml.
>
> Generally, we carry forward the existi
] Release 1.11.0, release candidate #4
Hi Till,
Yes, we don't have the setting in flink-conf.yaml.
Generally, we carry forward the existing configuration and any change to
default configuration values would impact the upgrade.
Yes, since it is an incompatible change I would state it in the re
Hi Till,
Yes, we don't have the setting in flink-conf.yaml.
Generally, we carry forward the existing configuration and any change to
default configuration values would impact the upgrade.
Yes, since it is an incompatible change I would state it in the release
notes.
Thanks,
Thomas
BTW I found
Hi Thomas,
just to confirm: When starting the image in local mode, then you don't have
any of the JobManager memory configuration settings configured in the
effective flink-conf.yaml, right? Does this mean that you have explicitly
removed `jobmanager.heap.size: 1024m` from the default configuratio
Thanks for preparing another RC!
As mentioned in the previous RC thread, it would be super helpful if the
release notes that are part of the documentation can be included [1]. It's
a significant time-saver to have read those first.
I found one more non-backward compatible change that would be wor
- source does not contain binaries
- started a local cluster, logs are fine, examples run
- web submission works _in general_
However, a number of batch examples fail when submitted through the
WebUI with the following error:
Caused by: org.apache.flink.api.common.InvalidProgramException:
Job
Hi everyone,
Please review and vote on the release candidate #4 for the version 1.11.0, as
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)
The complete staging area is available for your review, which includes:
* JIRA release notes [1],
58 matches
Mail list logo
