date:20201228

Hi Yufei,

I cannot remember exactly the changes in this area between Flink 1.10.0 and
Flink 1.12.0. It sounds a bit as if we were not releasing memory segments
fast enough or had a memory leak. One thing to try out is to increase the
restart delay to see whether it is the first problem. Alternatively, you
can also try to bisect the commits in between these versions. If you have a
test failing reliably, this shouldn't take too long. Maybe Piotr knows
about a fix which could have solved this problem.

Cheers,
Till

On Fri, Dec 25, 2020 at 3:05 AM Yangze Guo  wrote:

> Hi, Yufei.
>
> Can you reproduce this issue in 1.10.0? The deterministic slot sharing
> introduced in 1.12.0 is one possible reason. Before 1.12.0, the
> distribution of tasks in slots is not determined. Even if the network
> buffers are enough from the perspective of the cluster. Bad
> distribution of tasks can lead to the "insufficient network buffer" as
> well.
>
> Best,
> Yangze Guo
>
> On Fri, Dec 25, 2020 at 12:54 AM Yufei Liu  wrote:
> >
> > Hey,
> > I’ve found that job will throw “java.io.IOException: Insufficient number
> of network buffers: required 51, but only 1 available” after job retstart,
> and I’ve observed TM use much more network buffers than before.
> > My internal branch is under 1.10.0 can easily  reproduce, but I use
> 1.12.0 doesn’t have this issue. I Think maybe was already fixed after some
> PR, I'm curious about what can lead to this problem?
> >
> > Best.
> > YuFei.
>

Re: [DISCUSS] Releasing Apache Flink 1.12.1

Hi Xintong,

quick question, what about FLINK-20654? Previously it was listed as a
release blocker but has not been fixed yet.

Cheers,
Till

On Fri, Dec 25, 2020 at 10:52 AM Xintong Song  wrote:

> Hi devs,
>
> I'm very glad to announce that all known blocker issues for release-1.12.1
> have been resolved.
>
> I'm creating our first release candidate now and will start a separate
> voting thread as soon as RC1 is created.
>
> Thanks everyone, and Merry Christmas.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Wed, Dec 23, 2020 at 6:07 PM Xintong Song 
> wrote:
>
> > Hi devs,
> >
> > Updates on the progress of release.
> >
> > In the past week, more than 20 issues were resolved for release 1.12.1.
> > Thanks for the efforts.
> >
> > We still have 3 unresolved release blockers at the moment.
> >
> >- [FLINK-20648] Unable to restore from savepoints with Kubernetes HA.
> >Consensus has been reached on the solution. @Yang Wang is working on a
> >PR.
> >- [FLINK-20654] Unaligned checkpoint recovery may lead to corrupted
> >data stream.
> >@Roman Khachatryan is still investigating the problem.
> >- [FLINK-20664] Support setting service account for TaskManager pod.
> >Boris Lublinsky has opened a PR, which is already reviewed and close
> >to mergeable.
> >
> > Since we are targeting a swift release, I'm not intended to further delay
> > the release for other non-blocker issues, unless there's a good reason.
> > If there's anything that you believe is absolutely necessary for release
> > 1.12.1, please reach out to me.
> > Otherwise, the voting process will be started as soon as the above
> > blockers are addressed.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Mon, Dec 21, 2020 at 10:05 AM Xingbo Huang 
> wrote:
> >
> >> Hi Xintong,
> >>
> >> Thanks a lot for driving this.
> >>
> >> I'd like to bring one more issue to your attention:
> >> https://issues.apache.org/jira/browse/FLINK-20389.
> >> This issue occurs quite frequently. Arvid and Kezhu have done some
> >> investigations of this issue and it may indicate a bug of the new Source
> >> API. It would be great to figure out the root cause of this issue.
> >>
> >> Best,
> >> Xingbo
> >>
> >> Xintong Song  于2020年12月18日周五 下午7:49写道：
> >>
> >> > Thanks for the replies so far.
> >> >
> >> > I've been reaching out to the owners of the reported issues. It seems
> >> most
> >> > of the blockers are likely resolved in the next few days.
> >> >
> >> > Since some of the issues are quite critical, I'd like to aim for a
> >> *feature
> >> > freeze on Dec. 23rd*, and start the release voting process by the end
> of
> >> > this week.
> >> >
> >> > If there's anything you might need more time for, please reach out to
> >> me.
> >> >
> >> > Thank you~
> >> >
> >> > Xintong Song
> >> >
> >> >
> >> >
> >> > On Fri, Dec 18, 2020 at 3:19 PM Tzu-Li (Gordon) Tai <
> >> tzuli...@apache.org>
> >> > wrote:
> >> >
> >> > > Thanks Xintong for driving this.
> >> > >
> >> > > I'd like to make two more issues related to the Kinesis connector
> >> changes
> >> > > in 1.12.0 a blocker for 1.12.1:
> >> > > https://issues.apache.org/jira/browse/FLINK-20630
> >> > > https://issues.apache.org/jira/browse/FLINK-20629
> >> > >
> >> > > There are already PRs for these issues from @Cranmer, Danny
> >> > > , will try to merge these very soon.
> >> > >
> >> > > Cheers,
> >> > > Gordon
> >> > >
> >> > > On Fri, Dec 18, 2020 at 1:19 PM Guowei Ma 
> >> wrote:
> >> > >
> >> > >> Thanks for driving this release Xintong.
> >> > >> I think https://issues.apache.org/jira/browse/FLINK-20652 should
> be
> >> > >> addressed.
> >> > >>
> >> > >> Best,
> >> > >> Guowei
> >> > >>
> >> > >>
> >> > >> On Fri, Dec 18, 2020 at 11:53 AM Jingsong Li <
> jingsongl...@gmail.com
> >> >
> >> > >> wrote:
> >> > >>
> >> > >> > Thanks for volunteering as our release manager Xintong. +1 for
> >> > releasing
> >> > >> > Flink 1.12.1 soon.
> >> > >> >
> >> > >> > I think https://issues.apache.org/jira/browse/FLINK-20665 should
> >> be
> >> > >> > addressed, I marked it as a Blocker.
> >> > >> >
> >> > >> > Best,
> >> > >> > Jingsong
> >> > >> >
> >> > >> > On Fri, Dec 18, 2020 at 11:16 AM Yang Wang <
> danrtsey...@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > > Hi David,
> >> > >> > >
> >> > >> > > I will take a look this ticket FLINK-20648 and try to get it
> >> > resolved
> >> > >> in
> >> > >> > > this release cycle.
> >> > >> > >
> >> > >> > > @Xintong Song 
> >> > >> > > One more Kubernetes HA related issue. We need to support
> setting
> >> > >> service
> >> > >> > > account for TaskManager pod[1]. Even though we have a work
> around
> >> > for
> >> > >> > this
> >> > >> > > issue, but it is not acceptable to always let the default
> service
> >> > >> account
> >> > >> > > with enough permissions.
> >> > >> > >
> >> > >> > > [1]. https://issues.apache.org/jira/browse/FLINK-20664
> >> > >> > >
> >> > >> > > Best,
> >> > >> > > Yang
> >> > >> > >
> >> > >> > >
> >> > >> > > Davi

Re: [DISCUSS] Releasing Apache Flink 1.12.1

Hi Till,

@Piotr and @Roman mentioned offline that FLINK-20648 is not a new bug and
they don't think we should block a release on it.


I guess we should have made the conversation public visible. Sorry for the
confusion.


Thank you~

Xintong Song



On Mon, Dec 28, 2020 at 4:59 PM Till Rohrmann  wrote:

> Hi Xintong,
>
> quick question, what about FLINK-20654? Previously it was listed as a
> release blocker but has not been fixed yet.
>
> Cheers,
> Till
>
> On Fri, Dec 25, 2020 at 10:52 AM Xintong Song 
> wrote:
>
> > Hi devs,
> >
> > I'm very glad to announce that all known blocker issues for
> release-1.12.1
> > have been resolved.
> >
> > I'm creating our first release candidate now and will start a separate
> > voting thread as soon as RC1 is created.
> >
> > Thanks everyone, and Merry Christmas.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Wed, Dec 23, 2020 at 6:07 PM Xintong Song 
> > wrote:
> >
> > > Hi devs,
> > >
> > > Updates on the progress of release.
> > >
> > > In the past week, more than 20 issues were resolved for release 1.12.1.
> > > Thanks for the efforts.
> > >
> > > We still have 3 unresolved release blockers at the moment.
> > >
> > >- [FLINK-20648] Unable to restore from savepoints with Kubernetes
> HA.
> > >Consensus has been reached on the solution. @Yang Wang is working
> on a
> > >PR.
> > >- [FLINK-20654] Unaligned checkpoint recovery may lead to corrupted
> > >data stream.
> > >@Roman Khachatryan is still investigating the problem.
> > >- [FLINK-20664] Support setting service account for TaskManager pod.
> > >Boris Lublinsky has opened a PR, which is already reviewed and close
> > >to mergeable.
> > >
> > > Since we are targeting a swift release, I'm not intended to further
> delay
> > > the release for other non-blocker issues, unless there's a good reason.
> > > If there's anything that you believe is absolutely necessary for
> release
> > > 1.12.1, please reach out to me.
> > > Otherwise, the voting process will be started as soon as the above
> > > blockers are addressed.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Mon, Dec 21, 2020 at 10:05 AM Xingbo Huang 
> > wrote:
> > >
> > >> Hi Xintong,
> > >>
> > >> Thanks a lot for driving this.
> > >>
> > >> I'd like to bring one more issue to your attention:
> > >> https://issues.apache.org/jira/browse/FLINK-20389.
> > >> This issue occurs quite frequently. Arvid and Kezhu have done some
> > >> investigations of this issue and it may indicate a bug of the new
> Source
> > >> API. It would be great to figure out the root cause of this issue.
> > >>
> > >> Best,
> > >> Xingbo
> > >>
> > >> Xintong Song  于2020年12月18日周五 下午7:49写道：
> > >>
> > >> > Thanks for the replies so far.
> > >> >
> > >> > I've been reaching out to the owners of the reported issues. It
> seems
> > >> most
> > >> > of the blockers are likely resolved in the next few days.
> > >> >
> > >> > Since some of the issues are quite critical, I'd like to aim for a
> > >> *feature
> > >> > freeze on Dec. 23rd*, and start the release voting process by the
> end
> > of
> > >> > this week.
> > >> >
> > >> > If there's anything you might need more time for, please reach out
> to
> > >> me.
> > >> >
> > >> > Thank you~
> > >> >
> > >> > Xintong Song
> > >> >
> > >> >
> > >> >
> > >> > On Fri, Dec 18, 2020 at 3:19 PM Tzu-Li (Gordon) Tai <
> > >> tzuli...@apache.org>
> > >> > wrote:
> > >> >
> > >> > > Thanks Xintong for driving this.
> > >> > >
> > >> > > I'd like to make two more issues related to the Kinesis connector
> > >> changes
> > >> > > in 1.12.0 a blocker for 1.12.1:
> > >> > > https://issues.apache.org/jira/browse/FLINK-20630
> > >> > > https://issues.apache.org/jira/browse/FLINK-20629
> > >> > >
> > >> > > There are already PRs for these issues from @Cranmer, Danny
> > >> > > , will try to merge these very soon.
> > >> > >
> > >> > > Cheers,
> > >> > > Gordon
> > >> > >
> > >> > > On Fri, Dec 18, 2020 at 1:19 PM Guowei Ma 
> > >> wrote:
> > >> > >
> > >> > >> Thanks for driving this release Xintong.
> > >> > >> I think https://issues.apache.org/jira/browse/FLINK-20652 should
> > be
> > >> > >> addressed.
> > >> > >>
> > >> > >> Best,
> > >> > >> Guowei
> > >> > >>
> > >> > >>
> > >> > >> On Fri, Dec 18, 2020 at 11:53 AM Jingsong Li <
> > jingsongl...@gmail.com
> > >> >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > Thanks for volunteering as our release manager Xintong. +1 for
> > >> > releasing
> > >> > >> > Flink 1.12.1 soon.
> > >> > >> >
> > >> > >> > I think https://issues.apache.org/jira/browse/FLINK-20665
> should
> > >> be
> > >> > >> > addressed, I marked it as a Blocker.
> > >> > >> >
> > >> > >> > Best,
> > >> > >> > Jingsong
> > >> > >> >
> > >> > >> > On Fri, Dec 18, 2020 at 11:16 AM Yang Wang <
> > danrtsey...@gmail.com>
> > >> > >> wrote:
> > >> > >> >
> > >> > >> > > Hi David,
> > >> > >> > >
> > >> > >> > > I will take a look this ticket FLINK-20648 and try to get it
> > >

[jira] [Created] (FLINK-20785) MapRelDataType for legacy planner has wrong digest

2020-12-28 Thread Danny Chen (Jira)

Danny Chen created FLINK-20785:
--

 Summary: MapRelDataType for legacy planner has wrong digest
 Key: FLINK-20785
 URL: https://issues.apache.org/jira/browse/FLINK-20785
 Project: Flink
  Issue Type: Task
  Components: Table SQL / Legacy Planner
Affects Versions: 1.12.0
Reporter: Danny Chen


In the legacy planner, we create the {{MapRelDataType}} ignoring the key and 
value nullability. If we implements the digest correctly, it would conflict 
with the logic.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Releasing Apache Flink 1.12.1

2020-12-28 Thread Piotr Nowojski

Hi,

Yes, as Xintong wrote above, I've wrote offline to him:

> I’m going to remove release blocker status from FLINK-20654. After all we
already have released it, at least in 1.12.0, and maybe even sooner. There
is no point from blocking a release (which has probably some important bug
fixes) in that case. It’s not a new bug.

By "it's not a new bug", I meant that it has already been released in
1.12.0. Also after ignoring the test for the time being, this bug should
not be causing build failures anymore.

Piotrek

pon., 28 gru 2020 o 10:29 Xintong Song  napisał(a):

> Hi Till,
>
> @Piotr and @Roman mentioned offline that FLINK-20648 is not a new bug and
> they don't think we should block a release on it.
>
>
> I guess we should have made the conversation public visible. Sorry for the
> confusion.
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Dec 28, 2020 at 4:59 PM Till Rohrmann 
> wrote:
>
> > Hi Xintong,
> >
> > quick question, what about FLINK-20654? Previously it was listed as a
> > release blocker but has not been fixed yet.
> >
> > Cheers,
> > Till
> >
> > On Fri, Dec 25, 2020 at 10:52 AM Xintong Song 
> > wrote:
> >
> > > Hi devs,
> > >
> > > I'm very glad to announce that all known blocker issues for
> > release-1.12.1
> > > have been resolved.
> > >
> > > I'm creating our first release candidate now and will start a separate
> > > voting thread as soon as RC1 is created.
> > >
> > > Thanks everyone, and Merry Christmas.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Wed, Dec 23, 2020 at 6:07 PM Xintong Song 
> > > wrote:
> > >
> > > > Hi devs,
> > > >
> > > > Updates on the progress of release.
> > > >
> > > > In the past week, more than 20 issues were resolved for release
> 1.12.1.
> > > > Thanks for the efforts.
> > > >
> > > > We still have 3 unresolved release blockers at the moment.
> > > >
> > > >- [FLINK-20648] Unable to restore from savepoints with Kubernetes
> > HA.
> > > >Consensus has been reached on the solution. @Yang Wang is working
> > on a
> > > >PR.
> > > >- [FLINK-20654] Unaligned checkpoint recovery may lead to
> corrupted
> > > >data stream.
> > > >@Roman Khachatryan is still investigating the problem.
> > > >- [FLINK-20664] Support setting service account for TaskManager
> pod.
> > > >Boris Lublinsky has opened a PR, which is already reviewed and
> close
> > > >to mergeable.
> > > >
> > > > Since we are targeting a swift release, I'm not intended to further
> > delay
> > > > the release for other non-blocker issues, unless there's a good
> reason.
> > > > If there's anything that you believe is absolutely necessary for
> > release
> > > > 1.12.1, please reach out to me.
> > > > Otherwise, the voting process will be started as soon as the above
> > > > blockers are addressed.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Mon, Dec 21, 2020 at 10:05 AM Xingbo Huang 
> > > wrote:
> > > >
> > > >> Hi Xintong,
> > > >>
> > > >> Thanks a lot for driving this.
> > > >>
> > > >> I'd like to bring one more issue to your attention:
> > > >> https://issues.apache.org/jira/browse/FLINK-20389.
> > > >> This issue occurs quite frequently. Arvid and Kezhu have done some
> > > >> investigations of this issue and it may indicate a bug of the new
> > Source
> > > >> API. It would be great to figure out the root cause of this issue.
> > > >>
> > > >> Best,
> > > >> Xingbo
> > > >>
> > > >> Xintong Song  于2020年12月18日周五 下午7:49写道：
> > > >>
> > > >> > Thanks for the replies so far.
> > > >> >
> > > >> > I've been reaching out to the owners of the reported issues. It
> > seems
> > > >> most
> > > >> > of the blockers are likely resolved in the next few days.
> > > >> >
> > > >> > Since some of the issues are quite critical, I'd like to aim for a
> > > >> *feature
> > > >> > freeze on Dec. 23rd*, and start the release voting process by the
> > end
> > > of
> > > >> > this week.
> > > >> >
> > > >> > If there's anything you might need more time for, please reach out
> > to
> > > >> me.
> > > >> >
> > > >> > Thank you~
> > > >> >
> > > >> > Xintong Song
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Fri, Dec 18, 2020 at 3:19 PM Tzu-Li (Gordon) Tai <
> > > >> tzuli...@apache.org>
> > > >> > wrote:
> > > >> >
> > > >> > > Thanks Xintong for driving this.
> > > >> > >
> > > >> > > I'd like to make two more issues related to the Kinesis
> connector
> > > >> changes
> > > >> > > in 1.12.0 a blocker for 1.12.1:
> > > >> > > https://issues.apache.org/jira/browse/FLINK-20630
> > > >> > > https://issues.apache.org/jira/browse/FLINK-20629
> > > >> > >
> > > >> > > There are already PRs for these issues from @Cranmer, Danny
> > > >> > > , will try to merge these very soon.
> > > >> > >
> > > >> > > Cheers,
> > > >> > > Gordon
> > > >> > >
> > > >> > > On Fri, Dec 18, 2020 at 1:19 PM Guowei Ma  >
> > > >> wrote:
> > > >> > >
> > > >> > >> Thanks for driving this release Xintong.
> > > >> > >> I think ht

[jira] [Created] (FLINK-20786) flink-playgrounds Git branch(release-1.12) does not exist

2020-12-28 Thread billyxie (Jira)

billyxie created FLINK-20786:


 Summary: flink-playgrounds Git branch(release-1.12) does not exist 
 Key: FLINK-20786
 URL: https://issues.apache.org/jira/browse/FLINK-20786
 Project: Flink
  Issue Type: Bug
  Components: Examples
Affects Versions: 1.12.1
Reporter: billyxie
 Fix For: 1.12.1
 Attachments: image-2020-12-28-17-55-42-117.png, 
image-2020-12-28-17-57-20-829.png

Based on the official document ,we can use docer-compose to show the test 
environment.

[Flink Operations Playground

|[https://ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/flink-operations-playground.html]]

!image-2020-12-28-17-57-20-829.png!

but we can't find release-1.12 branch . 

!image-2020-12-28-17-55-42-117.png!


{color:#FF}> git clone --branch release-1.12 
https://github.com/apache/flink-playgrounds.git{color}
{color:#FF}Cloning into 'flink-playgrounds'...{color}
{color:#FF}fatal: Remote branch release-1.12 not found in upstream 
origin{color}

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Releasing Apache Flink 1.12.1

Alright, thanks for the clarification. Should we issue a warning to not use
unaligned checkpoints for the time being because it can lead to corrupted
data streams on recovery? I can envision that some of our users might be
surprised about it. Maybe adding it to the 1.12.0 and 1.12.1 release notes?

Cheers,
Till

On Mon, Dec 28, 2020 at 10:50 AM Piotr Nowojski 
wrote:

> Hi,
>
> Yes, as Xintong wrote above, I've wrote offline to him:
>
> > I’m going to remove release blocker status from FLINK-20654. After all we
> already have released it, at least in 1.12.0, and maybe even sooner. There
> is no point from blocking a release (which has probably some important bug
> fixes) in that case. It’s not a new bug.
>
> By "it's not a new bug", I meant that it has already been released in
> 1.12.0. Also after ignoring the test for the time being, this bug should
> not be causing build failures anymore.
>
> Piotrek
>
> pon., 28 gru 2020 o 10:29 Xintong Song  napisał(a):
>
> > Hi Till,
> >
> > @Piotr and @Roman mentioned offline that FLINK-20648 is not a new bug and
> > they don't think we should block a release on it.
> >
> >
> > I guess we should have made the conversation public visible. Sorry for
> the
> > confusion.
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Mon, Dec 28, 2020 at 4:59 PM Till Rohrmann 
> > wrote:
> >
> > > Hi Xintong,
> > >
> > > quick question, what about FLINK-20654? Previously it was listed as a
> > > release blocker but has not been fixed yet.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Dec 25, 2020 at 10:52 AM Xintong Song 
> > > wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I'm very glad to announce that all known blocker issues for
> > > release-1.12.1
> > > > have been resolved.
> > > >
> > > > I'm creating our first release candidate now and will start a
> separate
> > > > voting thread as soon as RC1 is created.
> > > >
> > > > Thanks everyone, and Merry Christmas.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Wed, Dec 23, 2020 at 6:07 PM Xintong Song 
> > > > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > Updates on the progress of release.
> > > > >
> > > > > In the past week, more than 20 issues were resolved for release
> > 1.12.1.
> > > > > Thanks for the efforts.
> > > > >
> > > > > We still have 3 unresolved release blockers at the moment.
> > > > >
> > > > >- [FLINK-20648] Unable to restore from savepoints with
> Kubernetes
> > > HA.
> > > > >Consensus has been reached on the solution. @Yang Wang is
> working
> > > on a
> > > > >PR.
> > > > >- [FLINK-20654] Unaligned checkpoint recovery may lead to
> > corrupted
> > > > >data stream.
> > > > >@Roman Khachatryan is still investigating the problem.
> > > > >- [FLINK-20664] Support setting service account for TaskManager
> > pod.
> > > > >Boris Lublinsky has opened a PR, which is already reviewed and
> > close
> > > > >to mergeable.
> > > > >
> > > > > Since we are targeting a swift release, I'm not intended to further
> > > delay
> > > > > the release for other non-blocker issues, unless there's a good
> > reason.
> > > > > If there's anything that you believe is absolutely necessary for
> > > release
> > > > > 1.12.1, please reach out to me.
> > > > > Otherwise, the voting process will be started as soon as the above
> > > > > blockers are addressed.
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Dec 21, 2020 at 10:05 AM Xingbo Huang 
> > > > wrote:
> > > > >
> > > > >> Hi Xintong,
> > > > >>
> > > > >> Thanks a lot for driving this.
> > > > >>
> > > > >> I'd like to bring one more issue to your attention:
> > > > >> https://issues.apache.org/jira/browse/FLINK-20389.
> > > > >> This issue occurs quite frequently. Arvid and Kezhu have done some
> > > > >> investigations of this issue and it may indicate a bug of the new
> > > Source
> > > > >> API. It would be great to figure out the root cause of this issue.
> > > > >>
> > > > >> Best,
> > > > >> Xingbo
> > > > >>
> > > > >> Xintong Song  于2020年12月18日周五 下午7:49写道：
> > > > >>
> > > > >> > Thanks for the replies so far.
> > > > >> >
> > > > >> > I've been reaching out to the owners of the reported issues. It
> > > seems
> > > > >> most
> > > > >> > of the blockers are likely resolved in the next few days.
> > > > >> >
> > > > >> > Since some of the issues are quite critical, I'd like to aim
> for a
> > > > >> *feature
> > > > >> > freeze on Dec. 23rd*, and start the release voting process by
> the
> > > end
> > > > of
> > > > >> > this week.
> > > > >> >
> > > > >> > If there's anything you might need more time for, please reach
> out
> > > to
> > > > >> me.
> > > > >> >
> > > > >> > Thank you~
> > > > >> >
> > > > >> > Xintong Song
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Dec 18, 2020 at 3:19 PM Tzu-Li (Gordon) Tai <
> > > > >> tzuli...@apache.org>
> > > > >> > wrote:
> > > > >> >
> > > >

Re: Changing application configuration when restoring from checkpoint/savepoint

2020-12-28 Thread Aljoscha Krettek

On Wed Dec 16, 2020 at 6:41 PM CET, vishalovercome wrote:
> 1. Is there any way to restore from a checkpoint as well as recreate
> client
> using newer configuration?

I think that would only work if you somehow read the configuration from an 
external system

> 2. If we take a savepoint (drain and save) and then resume the job, then
> will the configuration changes happen?

This should work, but you won't even need a drain. Just a stop with savepoint 
should work: 
https://ci.apache.org/projects/flink/flink-docs-stable/deployment/cli.html#stopping-a-job-gracefully-creating-a-final-savepoint

> For functions that aren't AsyncFunction, is leveraging BroadcastState
> the
> only way to dynamically update configuration?

I think this is true, yes.

Best,
Aljoscha

[jira] [Created] (FLINK-20787) Improve the Table API to make it usable

Dian Fu created FLINK-20787:
---

 Summary: Improve the Table API to make it usable
 Key: FLINK-20787
 URL: https://issues.apache.org/jira/browse/FLINK-20787
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Reporter: Dian Fu
 Fix For: 1.13.0


Currently, there are quite a few bugs in the Table API which makes it difficult 
to use. Users will encounter all kinds of problems when using Table API and 
have to fall back to SQL from time to time. This is an umbrella JIRA for all 
the issues specific in the Table API.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Releasing Apache Flink 1.12.1

Adding it as a warning of known issues to 1.12.1 release notes makes sense
to me. (If it doesn't get fixed in this release. I'm canceling 1.12.1-rc1
for another blocker.)

I'm not entirely sure about adding a warning to 1.12.0 release notes. Is it
how we usually do, adding warnings to release notes for bugs found after
the release?
Putting it another way, should we modify the release notes silently after
they are posted/published?

@Piotr,
Do we understand in which cases the recovery of unaligned checkpoints can
lead to a corrupted data stream? Or shall we suggest the users to never use
unaligned checkpoints for this version? Maybe you or Roman is the better
person to draft this warning?

Thank you~

Xintong Song



On Mon, Dec 28, 2020 at 6:10 PM Till Rohrmann  wrote:

> Alright, thanks for the clarification. Should we issue a warning to not use
> unaligned checkpoints for the time being because it can lead to corrupted
> data streams on recovery? I can envision that some of our users might be
> surprised about it. Maybe adding it to the 1.12.0 and 1.12.1 release notes?
>
> Cheers,
> Till
>
> On Mon, Dec 28, 2020 at 10:50 AM Piotr Nowojski 
> wrote:
>
> > Hi,
> >
> > Yes, as Xintong wrote above, I've wrote offline to him:
> >
> > > I’m going to remove release blocker status from FLINK-20654. After all
> we
> > already have released it, at least in 1.12.0, and maybe even sooner.
> There
> > is no point from blocking a release (which has probably some important
> bug
> > fixes) in that case. It’s not a new bug.
> >
> > By "it's not a new bug", I meant that it has already been released in
> > 1.12.0. Also after ignoring the test for the time being, this bug should
> > not be causing build failures anymore.
> >
> > Piotrek
> >
> > pon., 28 gru 2020 o 10:29 Xintong Song 
> napisał(a):
> >
> > > Hi Till,
> > >
> > > @Piotr and @Roman mentioned offline that FLINK-20648 is not a new bug
> and
> > > they don't think we should block a release on it.
> > >
> > >
> > > I guess we should have made the conversation public visible. Sorry for
> > the
> > > confusion.
> > >
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Mon, Dec 28, 2020 at 4:59 PM Till Rohrmann 
> > > wrote:
> > >
> > > > Hi Xintong,
> > > >
> > > > quick question, what about FLINK-20654? Previously it was listed as a
> > > > release blocker but has not been fixed yet.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Fri, Dec 25, 2020 at 10:52 AM Xintong Song  >
> > > > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I'm very glad to announce that all known blocker issues for
> > > > release-1.12.1
> > > > > have been resolved.
> > > > >
> > > > > I'm creating our first release candidate now and will start a
> > separate
> > > > > voting thread as soon as RC1 is created.
> > > > >
> > > > > Thanks everyone, and Merry Christmas.
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Dec 23, 2020 at 6:07 PM Xintong Song <
> tonysong...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi devs,
> > > > > >
> > > > > > Updates on the progress of release.
> > > > > >
> > > > > > In the past week, more than 20 issues were resolved for release
> > > 1.12.1.
> > > > > > Thanks for the efforts.
> > > > > >
> > > > > > We still have 3 unresolved release blockers at the moment.
> > > > > >
> > > > > >- [FLINK-20648] Unable to restore from savepoints with
> > Kubernetes
> > > > HA.
> > > > > >Consensus has been reached on the solution. @Yang Wang is
> > working
> > > > on a
> > > > > >PR.
> > > > > >- [FLINK-20654] Unaligned checkpoint recovery may lead to
> > > corrupted
> > > > > >data stream.
> > > > > >@Roman Khachatryan is still investigating the problem.
> > > > > >- [FLINK-20664] Support setting service account for
> TaskManager
> > > pod.
> > > > > >Boris Lublinsky has opened a PR, which is already reviewed and
> > > close
> > > > > >to mergeable.
> > > > > >
> > > > > > Since we are targeting a swift release, I'm not intended to
> further
> > > > delay
> > > > > > the release for other non-blocker issues, unless there's a good
> > > reason.
> > > > > > If there's anything that you believe is absolutely necessary for
> > > > release
> > > > > > 1.12.1, please reach out to me.
> > > > > > Otherwise, the voting process will be started as soon as the
> above
> > > > > > blockers are addressed.
> > > > > >
> > > > > > Thank you~
> > > > > >
> > > > > > Xintong Song
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Dec 21, 2020 at 10:05 AM Xingbo Huang <
> hxbks...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Hi Xintong,
> > > > > >>
> > > > > >> Thanks a lot for driving this.
> > > > > >>
> > > > > >> I'd like to bring one more issue to your attention:
> > > > > >> https://issues.apache.org/jira/browse/FLINK-20389.
> > > > > >> This issue occurs quite frequently. Arvid and Kezhu have done
> some
> > > > > >> investig

Re: [VOTE] Release 1.12.1, release candidate #1

@Becket has confirmed on the jira ticket that FLINK-20781 is indeed a bug.

I‘m hereby officially canceling this vote.

At the same time, it is appreciated to continue verifying the release
candidate and reporting issues, so they can be fixed by the next release
candidate.

The RC#2 will be created as soon as the reported blockers are fixed.

Thank you~

Xintong Song



On Mon, Dec 28, 2020 at 3:37 PM Xintong Song  wrote:

> Thanks for reporting the issue, @Matthias.
>
> @Becket, could you check if this should be a release blocker?
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Dec 28, 2020 at 3:24 PM Matthias Pohl 
> wrote:
>
>> -1 (non-binding)
>>
>> Could someone have a look into FLINK-20781 [1] to see whether this is a
>> problem for 1.12.1? It seems that it's introduced by FLINK-20492 [2].
>>
>> Best,
>> Matthias
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-20781
>> [2] https://issues.apache.org/jira/browse/FLINK-20492
>>
>> On Sat, Dec 26, 2020 at 1:53 AM Xintong Song  wrote:
>>
>> > Hi everyone,
>> >
>> > Please review and vote on the release candidate #1 for the version
>> 1.12.1,
>> > as follows:
>> >
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific comments)
>> >
>> > The complete staging area is available for your review, which includes:
>> > * JIRA release notes [1],
>> > * the official Apache source release and binary convenience releases to
>> be
>> > deployed to dist.apache.org [2], which are signed with the key with
>> > fingerprint F8E419AA0B60C28879E876859DFF40967ABFC5A4 [3],
>> > * all artifacts to be deployed to the Maven Central Repository [4],
>> > * source code tag "release-1.12.1-rc1" [5],
>> > * website pull request listing the new release and adding announcement
>> blog
>> > post [6].
>> >
>> > The vote will be open for at least 72 hours. It is adopted by majority
>> > approval, with at least 3 PMC affirmative votes.
>> >
>> > Thanks,
>> > Xintong Song
>> >
>> > [1]
>> >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12349459
>> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.1-rc1
>> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> > 
>> > [4]
>> https://repository.apache.org/content/repositories/orgapacheflink-1410
>> > [5] https://github.com/apache/flink/releases/tag/release-1.12.1-rc1
>> > [6] https://github.com/apache/flink-web/pull/405
>> > 
>>
>

[jira] [Created] (FLINK-20788) It doesn't support to use cube/rollup/grouping sets in the Table API

Dian Fu created FLINK-20788:
---

 Summary: It doesn't support to use cube/rollup/grouping sets in 
the Table API
 Key: FLINK-20788
 URL: https://issues.apache.org/jira/browse/FLINK-20788
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Dian Fu
 Fix For: 1.13.0


Currently, it doesn't support to use cube/rollup/grouping sets in the Table 
API. For the following job:
{code}
table.groupBy("cube(a, b)")
{code}

It will throw the following exception:
{code}
org.apache.flink.table.api.ValidationException: Undefined function: cube
at 
org.apache.flink.table.expressions.resolver.LookupCallResolver.lambda$visit$0(LookupCallResolver.java:49)
at java.util.Optional.orElseThrow(Optional.java:290)
at 
org.apache.flink.table.expressions.resolver.LookupCallResolver.visit(LookupCallResolver.java:49)
at 
org.apache.flink.table.expressions.resolver.LookupCallResolver.visit(LookupCallResolver.java:38)
at 
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)
at 
org.apache.flink.table.expressions.LookupCallExpression.accept(LookupCallExpression.java:65)
at 
org.apache.flink.table.expressions.resolver.rules.LookupCallByNameRule.lambda$apply$0(LookupCallByNameRule.java:38)
at 
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at 
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at 
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at 
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at 
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at 
org.apache.flink.table.expressions.resolver.rules.LookupCallByNameRule.apply(LookupCallByNameRule.java:38)
at 
org.apache.flink.table.expressions.resolver.ExpressionResolver.lambda$null$1(ExpressionResolver.java:211)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at 
org.apache.flink.table.expressions.resolver.ExpressionResolver.resolve(ExpressionResolver.java:178)
at 
org.apache.flink.table.operations.utils.OperationTreeBuilder.aggregate(OperationTreeBuilder.java:236)
at 
org.apache.flink.table.api.internal.TableImpl$GroupedTableImpl.select(TableImpl.java:632)
at 
org.apache.flink.table.api.internal.TableImpl$GroupedTableImpl.select(TableImpl.java:615)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.12.1, release candidate #1

2020-12-28 Thread Becket Qin

Hi Xintong,

Thanks for driving the release. I just replied to FLINK-20781. Personally I
don't think that is a blocker for release-1.12.1. The reasons are:

1. The root cause of the issue is thread unsafe shutdown sequence of the
mailbox task. It is an issue for a while, SourceOperator just exposes this
issue.
2. The issue only occurs occasionally when a task exits, and the
consequence is that the job will just failover.

I am happy to put a band-aid on the SourceOperator so the NPE won't be
thrown. But a correct fix would be fixing the mailbox shutdown sequence,
which does not seem a blocker for release-1.12.1.

Thanks,

Jiangjie (Becket) Qin

On Mon, Dec 28, 2020 at 7:37 PM Xintong Song  wrote:

> @Becket has confirmed on the jira ticket that FLINK-20781 is indeed a bug.
>
> I‘m hereby officially canceling this vote.
>
> At the same time, it is appreciated to continue verifying the release
> candidate and reporting issues, so they can be fixed by the next release
> candidate.
>
> The RC#2 will be created as soon as the reported blockers are fixed.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Dec 28, 2020 at 3:37 PM Xintong Song 
> wrote:
>
> > Thanks for reporting the issue, @Matthias.
> >
> > @Becket, could you check if this should be a release blocker?
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Mon, Dec 28, 2020 at 3:24 PM Matthias Pohl 
> > wrote:
> >
> >> -1 (non-binding)
> >>
> >> Could someone have a look into FLINK-20781 [1] to see whether this is a
> >> problem for 1.12.1? It seems that it's introduced by FLINK-20492 [2].
> >>
> >> Best,
> >> Matthias
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-20781
> >> [2] https://issues.apache.org/jira/browse/FLINK-20492
> >>
> >> On Sat, Dec 26, 2020 at 1:53 AM Xintong Song  wrote:
> >>
> >> > Hi everyone,
> >> >
> >> > Please review and vote on the release candidate #1 for the version
> >> 1.12.1,
> >> > as follows:
> >> >
> >> > [ ] +1, Approve the release
> >> > [ ] -1, Do not approve the release (please provide specific comments)
> >> >
> >> > The complete staging area is available for your review, which
> includes:
> >> > * JIRA release notes [1],
> >> > * the official Apache source release and binary convenience releases
> to
> >> be
> >> > deployed to dist.apache.org [2], which are signed with the key with
> >> > fingerprint F8E419AA0B60C28879E876859DFF40967ABFC5A4 [3],
> >> > * all artifacts to be deployed to the Maven Central Repository [4],
> >> > * source code tag "release-1.12.1-rc1" [5],
> >> > * website pull request listing the new release and adding announcement
> >> blog
> >> > post [6].
> >> >
> >> > The vote will be open for at least 72 hours. It is adopted by majority
> >> > approval, with at least 3 PMC affirmative votes.
> >> >
> >> > Thanks,
> >> > Xintong Song
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12349459
> >> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.1-rc1
> >> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> > 
> >> > [4]
> >> https://repository.apache.org/content/repositories/orgapacheflink-1410
> >> > [5] https://github.com/apache/flink/releases/tag/release-1.12.1-rc1
> >> > [6] https://github.com/apache/flink-web/pull/405
> >> > 
> >>
> >
>

Re: [VOTE] Release 1.12.1, release candidate #1

@Becket,

Thanks for the clarification. Sorry, I misunderstood what you meant on the
jira.

Seems it's not necessary to cancel this vote. WDYT, @Matthias?

Thank you~

Xintong Song



On Mon, Dec 28, 2020 at 8:42 PM Becket Qin  wrote:

> Hi Xintong,
>
> Thanks for driving the release. I just replied to FLINK-20781. Personally I
> don't think that is a blocker for release-1.12.1. The reasons are:
>
> 1. The root cause of the issue is thread unsafe shutdown sequence of the
> mailbox task. It is an issue for a while, SourceOperator just exposes this
> issue.
> 2. The issue only occurs occasionally when a task exits, and the
> consequence is that the job will just failover.
>
> I am happy to put a band-aid on the SourceOperator so the NPE won't be
> thrown. But a correct fix would be fixing the mailbox shutdown sequence,
> which does not seem a blocker for release-1.12.1.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Mon, Dec 28, 2020 at 7:37 PM Xintong Song 
> wrote:
>
> > @Becket has confirmed on the jira ticket that FLINK-20781 is indeed a
> bug.
> >
> > I‘m hereby officially canceling this vote.
> >
> > At the same time, it is appreciated to continue verifying the release
> > candidate and reporting issues, so they can be fixed by the next release
> > candidate.
> >
> > The RC#2 will be created as soon as the reported blockers are fixed.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Mon, Dec 28, 2020 at 3:37 PM Xintong Song 
> > wrote:
> >
> > > Thanks for reporting the issue, @Matthias.
> > >
> > > @Becket, could you check if this should be a release blocker?
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Mon, Dec 28, 2020 at 3:24 PM Matthias Pohl 
> > > wrote:
> > >
> > >> -1 (non-binding)
> > >>
> > >> Could someone have a look into FLINK-20781 [1] to see whether this is
> a
> > >> problem for 1.12.1? It seems that it's introduced by FLINK-20492 [2].
> > >>
> > >> Best,
> > >> Matthias
> > >>
> > >> [1] https://issues.apache.org/jira/browse/FLINK-20781
> > >> [2] https://issues.apache.org/jira/browse/FLINK-20492
> > >>
> > >> On Sat, Dec 26, 2020 at 1:53 AM Xintong Song 
> wrote:
> > >>
> > >> > Hi everyone,
> > >> >
> > >> > Please review and vote on the release candidate #1 for the version
> > >> 1.12.1,
> > >> > as follows:
> > >> >
> > >> > [ ] +1, Approve the release
> > >> > [ ] -1, Do not approve the release (please provide specific
> comments)
> > >> >
> > >> > The complete staging area is available for your review, which
> > includes:
> > >> > * JIRA release notes [1],
> > >> > * the official Apache source release and binary convenience releases
> > to
> > >> be
> > >> > deployed to dist.apache.org [2], which are signed with the key with
> > >> > fingerprint F8E419AA0B60C28879E876859DFF40967ABFC5A4 [3],
> > >> > * all artifacts to be deployed to the Maven Central Repository [4],
> > >> > * source code tag "release-1.12.1-rc1" [5],
> > >> > * website pull request listing the new release and adding
> announcement
> > >> blog
> > >> > post [6].
> > >> >
> > >> > The vote will be open for at least 72 hours. It is adopted by
> majority
> > >> > approval, with at least 3 PMC affirmative votes.
> > >> >
> > >> > Thanks,
> > >> > Xintong Song
> > >> >
> > >> > [1]
> > >> >
> > >> >
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12349459
> > >> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.1-rc1
> > >> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > >> > 
> > >> > [4]
> > >>
> https://repository.apache.org/content/repositories/orgapacheflink-1410
> > >> > [5] https://github.com/apache/flink/releases/tag/release-1.12.1-rc1
> > >> > [6] https://github.com/apache/flink-web/pull/405
> > >> > 
> > >>
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.12.1

Given that there might still be users who want to upgrade and, thus, are
reading the release notes for the 1.12.0 release, I would also update the
1.12.0 release notes. Moreover, I would consider the risk of losing state
because unaligned checkpoints might break recovery as quite a serious
problem. Imagine a user is running a production job which does not need to
recover for some time because of some lucky coincidence and then all of a
sudden Flink fails fatally with the first job failure. I would even argue
that such a problem would warrant a documentation update where we add a
warning box to [1] which states the current limitations. Of course, this
only holds true under the assumption that this is indeed a real problem and
not a test instability.

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html#unaligned-checkpoints

Cheers,
Till

On Mon, Dec 28, 2020 at 12:29 PM Xintong Song  wrote:

> Adding it as a warning of known issues to 1.12.1 release notes makes sense
> to me. (If it doesn't get fixed in this release. I'm canceling 1.12.1-rc1
> for another blocker.)
>
> I'm not entirely sure about adding a warning to 1.12.0 release notes. Is it
> how we usually do, adding warnings to release notes for bugs found after
> the release?
> Putting it another way, should we modify the release notes silently after
> they are posted/published?
>
> @Piotr,
> Do we understand in which cases the recovery of unaligned checkpoints can
> lead to a corrupted data stream? Or shall we suggest the users to never use
> unaligned checkpoints for this version? Maybe you or Roman is the better
> person to draft this warning?
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Dec 28, 2020 at 6:10 PM Till Rohrmann 
> wrote:
>
> > Alright, thanks for the clarification. Should we issue a warning to not
> use
> > unaligned checkpoints for the time being because it can lead to corrupted
> > data streams on recovery? I can envision that some of our users might be
> > surprised about it. Maybe adding it to the 1.12.0 and 1.12.1 release
> notes?
> >
> > Cheers,
> > Till
> >
> > On Mon, Dec 28, 2020 at 10:50 AM Piotr Nowojski 
> > wrote:
> >
> > > Hi,
> > >
> > > Yes, as Xintong wrote above, I've wrote offline to him:
> > >
> > > > I’m going to remove release blocker status from FLINK-20654. After
> all
> > we
> > > already have released it, at least in 1.12.0, and maybe even sooner.
> > There
> > > is no point from blocking a release (which has probably some important
> > bug
> > > fixes) in that case. It’s not a new bug.
> > >
> > > By "it's not a new bug", I meant that it has already been released in
> > > 1.12.0. Also after ignoring the test for the time being, this bug
> should
> > > not be causing build failures anymore.
> > >
> > > Piotrek
> > >
> > > pon., 28 gru 2020 o 10:29 Xintong Song 
> > napisał(a):
> > >
> > > > Hi Till,
> > > >
> > > > @Piotr and @Roman mentioned offline that FLINK-20648 is not a new bug
> > and
> > > > they don't think we should block a release on it.
> > > >
> > > >
> > > > I guess we should have made the conversation public visible. Sorry
> for
> > > the
> > > > confusion.
> > > >
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Mon, Dec 28, 2020 at 4:59 PM Till Rohrmann 
> > > > wrote:
> > > >
> > > > > Hi Xintong,
> > > > >
> > > > > quick question, what about FLINK-20654? Previously it was listed
> as a
> > > > > release blocker but has not been fixed yet.
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Fri, Dec 25, 2020 at 10:52 AM Xintong Song <
> tonysong...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi devs,
> > > > > >
> > > > > > I'm very glad to announce that all known blocker issues for
> > > > > release-1.12.1
> > > > > > have been resolved.
> > > > > >
> > > > > > I'm creating our first release candidate now and will start a
> > > separate
> > > > > > voting thread as soon as RC1 is created.
> > > > > >
> > > > > > Thanks everyone, and Merry Christmas.
> > > > > >
> > > > > > Thank you~
> > > > > >
> > > > > > Xintong Song
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 23, 2020 at 6:07 PM Xintong Song <
> > tonysong...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi devs,
> > > > > > >
> > > > > > > Updates on the progress of release.
> > > > > > >
> > > > > > > In the past week, more than 20 issues were resolved for release
> > > > 1.12.1.
> > > > > > > Thanks for the efforts.
> > > > > > >
> > > > > > > We still have 3 unresolved release blockers at the moment.
> > > > > > >
> > > > > > >- [FLINK-20648] Unable to restore from savepoints with
> > > Kubernetes
> > > > > HA.
> > > > > > >Consensus has been reached on the solution. @Yang Wang is
> > > working
> > > > > on a
> > > > > > >PR.
> > > > > > >- [FLINK-20654] Unaligned checkpoint recovery may lead to
> > > > corrupted
> > > > > > >data stream.
> > > > > > >@Roman Khachatryan is still i

Re: [DISCUSS][FLINK-20726] Introduce Pulsar connector

Hi Jianyun,

Thanks a lot for reviving this discussion. I think it would be great to
have a well working Pulsar connector for Flink. Before diving into the
detailed plan on how to do it technically, I think we should discuss where
exactly the connector should live. From the Flink community's perspective
every additional connector adds a considerable maintenance burden and,
thus, needs to be carefully considered.

1) Who from the Flink community will mentor this effort and could take
responsibility for it?
2) How can Pulsar be tested together with Flink (is there a Pulsar mini
cluster for IT tests)?
2.1) Do we need additional e2e tests?
3) Which Pulsar versions will/can be supported?

If the Pulsar community is willing to help with some of these tasks, then
this should be much easier but we need buy in from someone in the Flink
community to help with this effort.

The Flink community might be a bit unresponsive in the next couple of days
because of Christmas and New Year. Beginning of January the discussion
should see a couple of additional comments.

Cheers,
Till

On Wed, Dec 23, 2020 at 10:37 AM 赵 建云  wrote:

> We are now done with the Flink Pulsar connector. It is ready to be merged.
>
> Now our plan is
>
>   1.  merge Source and Sink based on Function, because it has been
> verified by users and is reliable.
>   2.  Next is to merge the DynamicTable feature. table implementation
> depends on Step 1.
>   3.  merge Pulsar Catalog
>   4.  merge the PulsarSource based on FLIP-27: Refactor Source Interface<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface>
> of PulsarSource
>   5.  merge PulsarSink based on FLIP-143: Unified Sink API<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> >
>   6.  merge upsert-pulsar
>
> About wiki, I need to update the FLIP-72: Introduce Pulsar Connector<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector>,
> but I don't have access to it.
>
> StreamNative/Pulsar Flink Connector:
> https://github.com/streamnative/pulsar-flink
>
>
> Looking forward to your feedback~
>
> Jianyun Zhao
>
>

[jira] [Created] (FLINK-20789) Add a metric named `deserializeFaildCount` for kafka connectors

2020-12-28 Thread xiaozilong (Jira)

xiaozilong created FLINK-20789:
--

 Summary: Add a metric named `deserializeFaildCount` for kafka 
connectors
 Key: FLINK-20789
 URL: https://issues.apache.org/jira/browse/FLINK-20789
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Metrics
Affects Versions: 1.12.0
Reporter: xiaozilong


Counts the number of deserialize failed when the option `ignore-parse-errors` 
is enabled for kafka connectors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20790) Generated classes should not be put under src/ directory

2020-12-28 Thread Chesnay Schepler (Jira)

Chesnay Schepler created FLINK-20790:


 Summary: Generated classes should not be put under src/ directory
 Key: FLINK-20790
 URL: https://issues.apache.org/jira/browse/FLINK-20790
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.13.0
Reporter: Chesnay Schepler


The flink-parquet and flink-confluent-schema-registry modules generate classes 
and put them into the src/ directory.

Standard convention is to put generated sources under target/generated-sources, 
which has the main advantage that they are deleted when doing a {{mvn clean}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20791) Support local aggregate push down for Blink batch planner

2020-12-28 Thread Sebastian Liu (Jira)

Sebastian Liu created FLINK-20791:
-

 Summary: Support local aggregate push down for Blink batch planner
 Key: FLINK-20791
 URL: https://issues.apache.org/jira/browse/FLINK-20791
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API, Table SQL / Planner
Reporter: Sebastian Liu


Aggregate operator of Flink SQL is currently fully done at Flink layer. With 
the developing of storage, many downstream storage of Flink SQL has the ability 
to deal with Aggregation operator.
Pushing down Aggregate to data source layer will improve performance from the 
perspective of the network IO and computation overhead.
 
I have drafted a design doc for this new feature. 
[https://docs.google.com/document/d/1kGwC_h4qBNxF2eMEz6T6arByOB8yilrPLqDN0QBQXW4/edit?usp=sharing]
 
Any comment or discussion is welcome.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Insufficient number of network buffers after restarting

2020-12-28 Thread Piotr Nowojski

Hi Yufei,

My prime suspect would be changes to the memory configuration introduced in
1.11 [1]

Piotrek

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html#memory-management

pon., 28 gru 2020 o 09:52 Till Rohrmann  napisał(a):

> Hi Yufei,
>
> I cannot remember exactly the changes in this area between Flink 1.10.0 and
> Flink 1.12.0. It sounds a bit as if we were not releasing memory segments
> fast enough or had a memory leak. One thing to try out is to increase the
> restart delay to see whether it is the first problem. Alternatively, you
> can also try to bisect the commits in between these versions. If you have a
> test failing reliably, this shouldn't take too long. Maybe Piotr knows
> about a fix which could have solved this problem.
>
> Cheers,
> Till
>
> On Fri, Dec 25, 2020 at 3:05 AM Yangze Guo  wrote:
>
> > Hi, Yufei.
> >
> > Can you reproduce this issue in 1.10.0? The deterministic slot sharing
> > introduced in 1.12.0 is one possible reason. Before 1.12.0, the
> > distribution of tasks in slots is not determined. Even if the network
> > buffers are enough from the perspective of the cluster. Bad
> > distribution of tasks can lead to the "insufficient network buffer" as
> > well.
> >
> > Best,
> > Yangze Guo
> >
> > On Fri, Dec 25, 2020 at 12:54 AM Yufei Liu 
> wrote:
> > >
> > > Hey,
> > > I’ve found that job will throw “java.io.IOException: Insufficient
> number
> > of network buffers: required 51, but only 1 available” after job
> retstart,
> > and I’ve observed TM use much more network buffers than before.
> > > My internal branch is under 1.10.0 can easily  reproduce, but I use
> > 1.12.0 doesn’t have this issue. I Think maybe was already fixed after
> some
> > PR, I'm curious about what can lead to this problem?
> > >
> > > Best.
> > > YuFei.
> >
>

Support local aggregate push down for Blink batch planner

2020-12-28 Thread Sebastian Liu

Hi all,

I'd like to discuss a new feature for the Blink Planner.
Aggregate operator of Flink SQL is currently fully done at Flink layer.
With the developing of storage, many downstream storage of Flink SQL has
the ability to deal with Aggregation operator.
Pushing down Aggregate to data source layer will improve performance from
the perspective of the network IO and computation overhead.

I have drafted a design doc for this new feature.
https://docs.google.com/document/d/1kGwC_h4qBNxF2eMEz6T6arByOB8yilrPLqDN0QBQXW4/edit?usp=sharing

Any comment or discussion is welcome.

-- 

*With kind regards

Sebastian Liu 刘洋
Institute of Computing Technology, Chinese Academy of Science
Mobile\WeChat: +86—15201613655
E-mail: liuyang0...@gmail.com 
QQ: 3239559*

[Discussion][FLINK-20416] Need a cached catalog for batch SQL job

2020-12-28 Thread Sebastian Liu

Hi all,

I‘d like to discuss a feature which supports the Flink OLAP scenario.

For OLAP scenarios, There are usually some analytical queries which running
time is relatively short. These queries are also sensitive to latency. In
the current Blink sql processing, parse/validate/optimize stages are all
need meta data from the catalog API. But each request to the catalog
requires re-run of the underlying meta query.

We may need a cached catalog which can cache the table schema and statistic
info to avoid unnecessary repeated meta requests. The most straightforward
scenario is to use Flink Batch SQL to query Hive data. If there is a Cached
Hive Catalog, we will save lots of interaction latency with HMS.

I have draft a design doc about this:
https://docs.google.com/document/d/1oL8HUpv2WaF6OkFvbH5iefXkOJB__Dal_bYsIZJA_Gk/edit?usp=sharing

Jira issue: https://issues.apache.org/jira/browse/FLINK-20416

IMO, this feature can further improve the stability and execution speed of
analyze query for Flink SQL.

Looking forward to your feedback, and any discussion or comments are
welcome.


-- 

*With kind regards

Sebastian Liu 刘洋
Institute of Computing Technology, Chinese Academy of Science
Mobile\WeChat: +86—15201613655
E-mail: liuyang0...@gmail.com 
QQ: 3239559*

[ANNOUNCE] New formatting rules are now in effect

2020-12-28 Thread Chesnay Schepler


Hello everyone,

I have just merged the commits for FLINK-20651 
 to master, 
release-1.12 and release-11, which enforces new formatting rules using 
the spotless plugin, following the google-java-format.


This change touched every single java file in the repository, 
predominantly due to switching from tabs to spaces. This implies that 
every PR and WIP branch will require a rebase.



Most of the changes were done by a single commit, which you can exclude 
from git blame by configuring git as follows (note that this requires 
git 2.23+, and also works for IntelliJ):


git config blame.ignoreRevsFile .git-blame-ignore-revs


You can setup the IDE to follow the new code style as follows:

1. Install the google-java-format plugin 
 and 
enable it for the project

2. In the plugin settings, change the code style to "AOSP" (4-space indents)
3. Install the Save Actions plugin 


4. Enable the plugin, along with "Optimize imports" and "Reformat file"

To manually apply the formatting, run:

mvn com.diffplug.spotless:spotless-maven-plugin:apply


Please be on the lookout for any suspicious formatting, outdated 
instructions or other inconveniences.



Finally, a big thank you to Aljoscha for pushing this topic and finally 
bringing it to an end.

[jira] [Created] (FLINK-20792) Allow shorthand invocation of spotless

2020-12-28 Thread Chesnay Schepler (Jira)

Chesnay Schepler created FLINK-20792:


 Summary: Allow shorthand invocation of spotless
 Key: FLINK-20792
 URL: https://issues.apache.org/jira/browse/FLINK-20792
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Affects Versions: 1.13.0, 1.11.4, 1.12.1
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.13.0, 1.11.4, 1.12.1


It is currently not possible to just run {{mvn spotless:check}}, but instead 
one needs to use the fully qualified name of the plugin, i.e., {{mvn 
com.diffplug.spotless:spotless-maven-plugin:check}}.

This is due to {{force-shading}}; this module is declared as a child of the 
flink-parent pom (i.e., the root pom of the project), but is not actually 
declaring that very module as it's parent.
This seems to throw of maven, which fails to retrieve the plugin information 
from the parent.

(Essentially, flink-parent tells maven "run this plugin for each of my 
children", maven asks child "so what should we actually do", child says "duh 
ask my parent duh", but that parent is just  some random dude that doesn't know 
anything and maven doesn't know what to do.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.12.1, release candidate #1

2020-12-28 Thread Chesnay Schepler

Should we maybe wait with this release until we have the docker issue
sorted out?

On 12/28/2020 1:51 PM, Xintong Song wrote:

@Becket,

Thanks for the clarification. Sorry, I misunderstood what you meant on the
jira.

Seems it's not necessary to cancel this vote. WDYT, @Matthias?

Thank you~

Xintong Song

On Mon, Dec 28, 2020 at 8:42 PM Becket Qin wrote:

Hi Xintong,

Thanks for driving the release. I just replied to FLINK-20781. Personally I
don't think that is a blocker for release-1.12.1. The reasons are:

1. The root cause of the issue is thread unsafe shutdown sequence of the
mailbox task. It is an issue for a while, SourceOperator just exposes this
issue.
2. The issue only occurs occasionally when a task exits, and the
consequence is that the job will just failover.

I am happy to put a band-aid on the SourceOperator so the NPE won't be
thrown. But a correct fix would be fixing the mailbox shutdown sequence,
which does not seem a blocker for release-1.12.1.

Thanks,

Jiangjie (Becket) Qin

On Mon, Dec 28, 2020 at 7:37 PM Xintong Song
wrote:

@Becket has confirmed on the jira ticket that FLINK-20781 is indeed a

bug.

I‘m hereby officially canceling this vote.

At the same time, it is appreciated to continue verifying the release
candidate and reporting issues, so they can be fixed by the next release
candidate.

The RC#2 will be created as soon as the reported blockers are fixed.

Thank you~

Xintong Song

On Mon, Dec 28, 2020 at 3:37 PM Xintong Song
wrote:

Thanks for reporting the issue, @Matthias.

@Becket, could you check if this should be a release blocker?

Thank you~

Xintong Song

On Mon, Dec 28, 2020 at 3:24 PM Matthias Pohl
wrote:

-1 (non-binding)

Could someone have a look into FLINK-20781 [1] to see whether this is

problem for 1.12.1? It seems that it's introduced by FLINK-20492 [2].

Best,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-20781
[2] https://issues.apache.org/jira/browse/FLINK-20492

On Sat, Dec 26, 2020 at 1:53 AM Xintong Song

wrote:

Hi everyone,

Please review and vote on the release candidate #1 for the version

1.12.1,

as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific

comments)

The complete staging area is available for your review, which

includes:

* JIRA release notes [1],
* the official Apache source release and binary convenience releases

deployed to dist.apache.org [2], which are signed with the key with
fingerprint F8E419AA0B60C28879E876859DFF40967ABFC5A4 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.12.1-rc1" [5],
* website pull request listing the new release and adding

announcement

blog

post [6].

The vote will be open for at least 72 hours. It is adopted by

majority

approval, with at least 3 PMC affirmative votes.

Thanks,
Xintong Song

[1]

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12349459

[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.1-rc1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS

[4]

https://repository.apache.org/content/repositories/orgapacheflink-1410

[5] https://github.com/apache/flink/releases/tag/release-1.12.1-rc1
[6] https://github.com/apache/flink-web/pull/405

Re: [VOTE] Release 1.12.1, release candidate #1

I think we may hold announcing this release until the 1.12.1 official image
is available.

For the other approaches discussed in FLINK-20632 (hosting Docker images
ourselves, moving most of entrypoint scripts into the Flink repository,
etc.), maybe we can address them in future releases.

Thank you~

Xintong Song


[1] https://issues.apache.org/jira/browse/FLINK-20632



On Tue, Dec 29, 2020 at 3:08 AM Chesnay Schepler  wrote:

> Should we maybe wait with this release until we have the docker issue
> sorted out?
>
> On 12/28/2020 1:51 PM, Xintong Song wrote:
> > @Becket,
> >
> > Thanks for the clarification. Sorry, I misunderstood what you meant on
> the
> > jira.
> >
> > Seems it's not necessary to cancel this vote. WDYT, @Matthias?
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Mon, Dec 28, 2020 at 8:42 PM Becket Qin  wrote:
> >
> >> Hi Xintong,
> >>
> >> Thanks for driving the release. I just replied to FLINK-20781.
> Personally I
> >> don't think that is a blocker for release-1.12.1. The reasons are:
> >>
> >> 1. The root cause of the issue is thread unsafe shutdown sequence of the
> >> mailbox task. It is an issue for a while, SourceOperator just exposes
> this
> >> issue.
> >> 2. The issue only occurs occasionally when a task exits, and the
> >> consequence is that the job will just failover.
> >>
> >> I am happy to put a band-aid on the SourceOperator so the NPE won't be
> >> thrown. But a correct fix would be fixing the mailbox shutdown sequence,
> >> which does not seem a blocker for release-1.12.1.
> >>
> >> Thanks,
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On Mon, Dec 28, 2020 at 7:37 PM Xintong Song 
> >> wrote:
> >>
> >>> @Becket has confirmed on the jira ticket that FLINK-20781 is indeed a
> >> bug.
> >>> I‘m hereby officially canceling this vote.
> >>>
> >>> At the same time, it is appreciated to continue verifying the release
> >>> candidate and reporting issues, so they can be fixed by the next
> release
> >>> candidate.
> >>>
> >>> The RC#2 will be created as soon as the reported blockers are fixed.
> >>>
> >>> Thank you~
> >>>
> >>> Xintong Song
> >>>
> >>>
> >>>
> >>> On Mon, Dec 28, 2020 at 3:37 PM Xintong Song 
> >>> wrote:
> >>>
>  Thanks for reporting the issue, @Matthias.
> 
>  @Becket, could you check if this should be a release blocker?
> 
>  Thank you~
> 
>  Xintong Song
> 
> 
> 
>  On Mon, Dec 28, 2020 at 3:24 PM Matthias Pohl  >
>  wrote:
> 
> > -1 (non-binding)
> >
> > Could someone have a look into FLINK-20781 [1] to see whether this is
> >> a
> > problem for 1.12.1? It seems that it's introduced by FLINK-20492 [2].
> >
> > Best,
> > Matthias
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-20781
> > [2] https://issues.apache.org/jira/browse/FLINK-20492
> >
> > On Sat, Dec 26, 2020 at 1:53 AM Xintong Song 
> >> wrote:
> >> Hi everyone,
> >>
> >> Please review and vote on the release candidate #1 for the version
> > 1.12.1,
> >> as follows:
> >>
> >> [ ] +1, Approve the release
> >> [ ] -1, Do not approve the release (please provide specific
> >> comments)
> >> The complete staging area is available for your review, which
> >>> includes:
> >> * JIRA release notes [1],
> >> * the official Apache source release and binary convenience releases
> >>> to
> > be
> >> deployed to dist.apache.org [2], which are signed with the key with
> >> fingerprint F8E419AA0B60C28879E876859DFF40967ABFC5A4 [3],
> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >> * source code tag "release-1.12.1-rc1" [5],
> >> * website pull request listing the new release and adding
> >> announcement
> > blog
> >> post [6].
> >>
> >> The vote will be open for at least 72 hours. It is adopted by
> >> majority
> >> approval, with at least 3 PMC affirmative votes.
> >>
> >> Thanks,
> >> Xintong Song
> >>
> >> [1]
> >>
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12349459
> >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.1-rc1
> >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> 
> >> [4]
> >> https://repository.apache.org/content/repositories/orgapacheflink-1410
> >> [5] https://github.com/apache/flink/releases/tag/release-1.12.1-rc1
> >> [6] https://github.com/apache/flink-web/pull/405
> >> 
>
>
>

[jira] [Created] (FLINK-20793) Fix NamesTest due to code style refactor

2020-12-28 Thread Huang Xingbo (Jira)

Huang Xingbo created FLINK-20793:


 Summary: Fix NamesTest due to code style refactor
 Key: FLINK-20793
 URL: https://issues.apache.org/jira/browse/FLINK-20793
 Project: Flink
  Issue Type: Bug
  Components: API / Core
Affects Versions: 1.12.0, 1.11.0, 1.13.0
Reporter: Huang Xingbo


Due to the [FLINK-20651|https://issues.apache.org/jira/browse/FLINK-20651], the 
NameTest failed

[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11403&view=results]

I will fix it asap



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20794) Support to select distinct columns in the Table API

Dian Fu created FLINK-20794:
---

 Summary: Support to select distinct columns in the Table API
 Key: FLINK-20794
 URL: https://issues.apache.org/jira/browse/FLINK-20794
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Dian Fu
 Fix For: 1.13.0


Currently, there is no corresponding functionality in Table API for the 
following SQL:
{code:java}
SELECT DISTINCT users FROM Orders
{code}
For example, for the following job:
{code:java}
table.select("distinct a")
{code}
It will thrown the following exception:
{code:java}
org.apache.flink.table.api.ExpressionParserException: Could not parse 
expression at column 10: ',' expected but 'a' 
foundorg.apache.flink.table.api.ExpressionParserException: Could not parse 
expression at column 10: ',' expected but 'a' founddistinct a         ^
 at 
org.apache.flink.table.expressions.PlannerExpressionParserImpl$.throwError(PlannerExpressionParserImpl.scala:726)
 at 
org.apache.flink.table.expressions.PlannerExpressionParserImpl$.parseExpressionList(PlannerExpressionParserImpl.scala:710)
 at 
org.apache.flink.table.expressions.PlannerExpressionParserImpl.parseExpressionList(PlannerExpressionParserImpl.scala:47)
 at 
org.apache.flink.table.expressions.ExpressionParser.parseExpressionList(ExpressionParser.java:40)
 at 
org.apache.flink.table.api.internal.TableImpl.select(TableImpl.java:121){code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New formatting rules are now in effect

Great! Thanks Aljoscha and Chesnay for driving this.

Thank you~

Xintong Song



On Tue, Dec 29, 2020 at 1:28 AM Chesnay Schepler  wrote:

> Hello everyone,
>
> I have just merged the commits for FLINK-20651
>  to master,
> release-1.12 and release-11, which enforces new formatting rules using
> the spotless plugin, following the google-java-format.
>
> This change touched every single java file in the repository,
> predominantly due to switching from tabs to spaces. This implies that
> every PR and WIP branch will require a rebase.
>
>
> Most of the changes were done by a single commit, which you can exclude
> from git blame by configuring git as follows (note that this requires
> git 2.23+, and also works for IntelliJ):
>
> git config blame.ignoreRevsFile .git-blame-ignore-revs
>
>
> You can setup the IDE to follow the new code style as follows:
>
> 1. Install the google-java-format plugin
>  and
> enable it for the project
> 2. In the plugin settings, change the code style to "AOSP" (4-space
> indents)
> 3. Install the Save Actions plugin
> 
> 4. Enable the plugin, along with "Optimize imports" and "Reformat file"
>
> To manually apply the formatting, run:
>
> mvn com.diffplug.spotless:spotless-maven-plugin:apply
>
>
> Please be on the lookout for any suspicious formatting, outdated
> instructions or other inconveniences.
>
>
> Finally, a big thank you to Aljoscha for pushing this topic and finally
> bringing it to an end.
>
>

[jira] [Created] (FLINK-20795) add a parameter to decide whether or not print dirty record when `ignore-parse-errors` is true

2020-12-28 Thread zoucao (Jira)

zoucao created FLINK-20795:
--

 Summary: add a parameter to decide whether or not print dirty 
record when `ignore-parse-errors` is true
 Key: FLINK-20795
 URL: https://issues.apache.org/jira/browse/FLINK-20795
 Project: Flink
  Issue Type: Improvement
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.13.0
Reporter: zoucao


add a parameter to decide whether or not to print dirty data when 
`ignore-parse-errors`=true, some users want to make his task stability and know 
the dirty record to fix the upstream, too.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New formatting rules are now in effect

2020-12-28 Thread Jark Wu

Thanks Aljoscha and Chesnay for the great work!

Best,
Jark

On Tue, 29 Dec 2020 at 11:11, Xintong Song  wrote:

> Great! Thanks Aljoscha and Chesnay for driving this.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Tue, Dec 29, 2020 at 1:28 AM Chesnay Schepler 
> wrote:
>
> > Hello everyone,
> >
> > I have just merged the commits for FLINK-20651
> >  to master,
> > release-1.12 and release-11, which enforces new formatting rules using
> > the spotless plugin, following the google-java-format.
> >
> > This change touched every single java file in the repository,
> > predominantly due to switching from tabs to spaces. This implies that
> > every PR and WIP branch will require a rebase.
> >
> >
> > Most of the changes were done by a single commit, which you can exclude
> > from git blame by configuring git as follows (note that this requires
> > git 2.23+, and also works for IntelliJ):
> >
> > git config blame.ignoreRevsFile .git-blame-ignore-revs
> >
> >
> > You can setup the IDE to follow the new code style as follows:
> >
> > 1. Install the google-java-format plugin
> >  and
> > enable it for the project
> > 2. In the plugin settings, change the code style to "AOSP" (4-space
> > indents)
> > 3. Install the Save Actions plugin
> > 
> > 4. Enable the plugin, along with "Optimize imports" and "Reformat file"
> >
> > To manually apply the formatting, run:
> >
> > mvn com.diffplug.spotless:spotless-maven-plugin:apply
> >
> >
> > Please be on the lookout for any suspicious formatting, outdated
> > instructions or other inconveniences.
> >
> >
> > Finally, a big thank you to Aljoscha for pushing this topic and finally
> > bringing it to an end.
> >
> >
>

[jira] [Created] (FLINK-20796) Support unnest in the Table API

Dian Fu created FLINK-20796:
---

 Summary: Support unnest in the Table API
 Key: FLINK-20796
 URL: https://issues.apache.org/jira/browse/FLINK-20796
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Dian Fu
 Fix For: 1.13.0


Currently, there is no corresponding functionality in Table API for the 
following SQL:
{code:java}
SELECT a, s FROM T, UNNEST(T.b) as A(s) {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20797) can flink on k8s use pv using NFS and pvc as the hight avalibility storagedir

2020-12-28 Thread hayden zhou (Jira)

hayden zhou created FLINK-20797:
---

 Summary: can flink on k8s use pv using NFS and pvc as the hight 
avalibility storagedir
 Key: FLINK-20797
 URL: https://issues.apache.org/jira/browse/FLINK-20797
 Project: Flink
  Issue Type: New Feature
  Components: Client / Job Submission
 Environment: FLINK 1.12.0

 
Reporter: hayden zhou


I want to deploy Flink on k8s with HA mode, and I don't want to deploy the HDFS 
cluster, and I have an NFS so that I am created a PV that use NFS as the 
backend storage, and I created a PVC for deployment mount.

this is my FLINK configMap

```

kubernetes.cluster-id: mta-flink
 high-availability: org.apache.flink.kubernetes.highavailability. 
KubernetesHaServicesFactory
 high-availability.storageDir: file:///opt/flink/nfs/ha

```

and this is my jobmanager yaml file:

```

volumeMounts:
 - name: flink-config-volume
 mountPath: /opt/flink/conf
 - name: flink-nfs
 mountPath: /opt/flink/nfs
 securityContext:
 runAsUser:  # refers to user _flink_ from official flink image, change if 
necessary
 #fsGroup: 
 volumes:
 - name: flink-config-volume
 configMap:
 name: mta-flink-config
 items:
 - key: flink-conf.yaml
 path: flink-conf.yaml
 - key: log4j-console.properties
 path: log4j-console.properties
 - name: flink-nfs
 persistentVolumeClaim:
 claimName: mta-flink-nfs-pvc

```

It can be deployed successfully, but if I browser the jobmanager:8081 website, 
I get the result below:

```

{"errors": ["Service temporarily unavailable due to an ongoing leader election. 
Please refresh."]}

```

 

is the PVC can be used as `high-availability.storageDir`?  if it's can be used, 
how can I fix this error?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New formatting rules are now in effect

2020-12-28 Thread Matthias Pohl

Yes, thanks for driving this, Aljoscha. ...and Chesnay as well for helping
to finalize it.
Good job.

Matthias

On Tue, Dec 29, 2020 at 5:23 AM Jark Wu  wrote:

> Thanks Aljoscha and Chesnay for the great work!
>
> Best,
> Jark
>
> On Tue, 29 Dec 2020 at 11:11, Xintong Song  wrote:
>
> > Great! Thanks Aljoscha and Chesnay for driving this.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Tue, Dec 29, 2020 at 1:28 AM Chesnay Schepler 
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I have just merged the commits for FLINK-20651
> > >  to master,
> > > release-1.12 and release-11, which enforces new formatting rules using
> > > the spotless plugin, following the google-java-format.
> > >
> > > This change touched every single java file in the repository,
> > > predominantly due to switching from tabs to spaces. This implies that
> > > every PR and WIP branch will require a rebase.
> > >
> > >
> > > Most of the changes were done by a single commit, which you can exclude
> > > from git blame by configuring git as follows (note that this requires
> > > git 2.23+, and also works for IntelliJ):
> > >
> > > git config blame.ignoreRevsFile .git-blame-ignore-revs
> > >
> > >
> > > You can setup the IDE to follow the new code style as follows:
> > >
> > > 1. Install the google-java-format plugin
> > >  and
> > > enable it for the project
> > > 2. In the plugin settings, change the code style to "AOSP" (4-space
> > > indents)
> > > 3. Install the Save Actions plugin
> > > 
> > > 4. Enable the plugin, along with "Optimize imports" and "Reformat file"
> > >
> > > To manually apply the formatting, run:
> > >
> > > mvn com.diffplug.spotless:spotless-maven-plugin:apply
> > >
> > >
> > > Please be on the lookout for any suspicious formatting, outdated
> > > instructions or other inconveniences.
> > >
> > >
> > > Finally, a big thank you to Aljoscha for pushing this topic and finally
> > > bringing it to an end.
> > >
> > >
> >

Re: [VOTE] Release 1.12.1, release candidate #1

2020-12-28 Thread Matthias Pohl

I think Becket's proposal in [1] sounds reasonable having the quickfix in
place for 1.12.1.

Thanks,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-20781

On Mon, Dec 28, 2020 at 1:51 PM Xintong Song  wrote:

> @Becket,
>
> Thanks for the clarification. Sorry, I misunderstood what you meant on the
> jira.
>
> Seems it's not necessary to cancel this vote. WDYT, @Matthias?
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Dec 28, 2020 at 8:42 PM Becket Qin  wrote:
>
> > Hi Xintong,
> >
> > Thanks for driving the release. I just replied to FLINK-20781.
> Personally I
> > don't think that is a blocker for release-1.12.1. The reasons are:
> >
> > 1. The root cause of the issue is thread unsafe shutdown sequence of the
> > mailbox task. It is an issue for a while, SourceOperator just exposes
> this
> > issue.
> > 2. The issue only occurs occasionally when a task exits, and the
> > consequence is that the job will just failover.
> >
> > I am happy to put a band-aid on the SourceOperator so the NPE won't be
> > thrown. But a correct fix would be fixing the mailbox shutdown sequence,
> > which does not seem a blocker for release-1.12.1.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Mon, Dec 28, 2020 at 7:37 PM Xintong Song 
> > wrote:
> >
> > > @Becket has confirmed on the jira ticket that FLINK-20781 is indeed a
> > bug.
> > >
> > > I‘m hereby officially canceling this vote.
> > >
> > > At the same time, it is appreciated to continue verifying the release
> > > candidate and reporting issues, so they can be fixed by the next
> release
> > > candidate.
> > >
> > > The RC#2 will be created as soon as the reported blockers are fixed.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Mon, Dec 28, 2020 at 3:37 PM Xintong Song 
> > > wrote:
> > >
> > > > Thanks for reporting the issue, @Matthias.
> > > >
> > > > @Becket, could you check if this should be a release blocker?
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Mon, Dec 28, 2020 at 3:24 PM Matthias Pohl <
> matth...@ververica.com>
> > > > wrote:
> > > >
> > > >> -1 (non-binding)
> > > >>
> > > >> Could someone have a look into FLINK-20781 [1] to see whether this
> is
> > a
> > > >> problem for 1.12.1? It seems that it's introduced by FLINK-20492
> [2].
> > > >>
> > > >> Best,
> > > >> Matthias
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/FLINK-20781
> > > >> [2] https://issues.apache.org/jira/browse/FLINK-20492
> > > >>
> > > >> On Sat, Dec 26, 2020 at 1:53 AM Xintong Song 
> > wrote:
> > > >>
> > > >> > Hi everyone,
> > > >> >
> > > >> > Please review and vote on the release candidate #1 for the version
> > > >> 1.12.1,
> > > >> > as follows:
> > > >> >
> > > >> > [ ] +1, Approve the release
> > > >> > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > >> >
> > > >> > The complete staging area is available for your review, which
> > > includes:
> > > >> > * JIRA release notes [1],
> > > >> > * the official Apache source release and binary convenience
> releases
> > > to
> > > >> be
> > > >> > deployed to dist.apache.org [2], which are signed with the key
> with
> > > >> > fingerprint F8E419AA0B60C28879E876859DFF40967ABFC5A4 [3],
> > > >> > * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > >> > * source code tag "release-1.12.1-rc1" [5],
> > > >> > * website pull request listing the new release and adding
> > announcement
> > > >> blog
> > > >> > post [6].
> > > >> >
> > > >> > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > >> > approval, with at least 3 PMC affirmative votes.
> > > >> >
> > > >> > Thanks,
> > > >> > Xintong Song
> > > >> >
> > > >> > [1]
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12349459
> > > >> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.1-rc1
> > > >> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > >> > 
> > > >> > [4]
> > > >>
> > https://repository.apache.org/content/repositories/orgapacheflink-1410
> > > >> > [5]
> https://github.com/apache/flink/releases/tag/release-1.12.1-rc1
> > > >> > [6] https://github.com/apache/flink-web/pull/405
> > > >> > 
> > > >>
> > > >
> > >
> >

Re: Support local aggregate push down for Blink batch planner

2020-12-28 Thread Jark Wu

Hi Sebastian,

Thanks for the proposal. I think this is a great improvement for Flink SQL.
I went through the design doc and have following thoughts:

1) Flink has deprecated the legacy TableSource in 1.11 and proposed a new
 set of DynamicTableSource interfaces. Could you update your proposal to
use the new interfaces?
 Follow the existing ability interfaces, e.g.
SupportsFilterPushDown, SupportsProjectionPushDown.

2) Personally, I think CallExpression would be a better representation than
separate `FunctionDefinition` and args. Because, it would be easier to know
what's the index and type of the arguments.

3) It would be better to list which connectors will be supported in the
plan?

Best,
Jark

On Tue, 29 Dec 2020 at 00:49, Sebastian Liu  wrote:

> Hi all,
>
> I'd like to discuss a new feature for the Blink Planner.
> Aggregate operator of Flink SQL is currently fully done at Flink layer.
> With the developing of storage, many downstream storage of Flink SQL has
> the ability to deal with Aggregation operator.
> Pushing down Aggregate to data source layer will improve performance from
> the perspective of the network IO and computation overhead.
>
> I have drafted a design doc for this new feature.
>
> https://docs.google.com/document/d/1kGwC_h4qBNxF2eMEz6T6arByOB8yilrPLqDN0QBQXW4/edit?usp=sharing
>
> Any comment or discussion is welcome.
>
> --
>
> *With kind regards
> 
> Sebastian Liu 刘洋
> Institute of Computing Technology, Chinese Academy of Science
> Mobile\WeChat: +86—15201613655
> E-mail: liuyang0...@gmail.com 
> QQ: 3239559*
>

[jira] [Created] (FLINK-20798) Service temporarily unavailable due to an ongoing leader election. Please refresh.

2020-12-28 Thread hayden zhou (Jira)

hayden zhou created FLINK-20798:
---

 Summary: Service temporarily unavailable due to an ongoing leader 
election. Please refresh.
 Key: FLINK-20798
 URL: https://issues.apache.org/jira/browse/FLINK-20798
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes
Affects Versions: 1.12.0
 Environment: FLINK 1.12.0
Reporter: hayden zhou


我这边 部署 flink 到 k8s 使用 PVC 作为 high avalibility storagedir ， 我看jobmanager 
的日志，选举成功了。但是 web 一直显示选举进行中。

 

下面是 jobmanager 的日志

```

2020-12-29T06:45:54.177850394Z 2020-12-29 14:45:54,177 DEBUG 
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Leader 
election started
2020-12-29T06:45:54.177855303Z 2020-12-29 14:45:54,177 DEBUG 
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - 
Attempting to acquire leader lease 'ConfigMapLock: default - 
mta-flink-resourcemanager-leader (6f6479c6-86cc-4d62-84f9-37ff968bd0e5)'...
2020-12-29T06:45:54.178668055Z 2020-12-29 14:45:54,178 DEBUG 
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - WebSocket 
successfully opened
2020-12-29T06:45:54.178895963Z 2020-12-29 14:45:54,178 INFO 
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - 
Starting DefaultLeaderRetrievalService with 
KubernetesLeaderRetrievalDriver\{configMapName='mta-flink-resourcemanager-leader'}.
2020-12-29T06:45:54.179327491Z 2020-12-29 14:45:54,179 DEBUG 
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - 
Connecting websocket ... 
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6d303498
2020-12-29T06:45:54.230081993Z 2020-12-29 14:45:54,229 DEBUG 
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - WebSocket 
successfully opened
2020-12-29T06:45:54.230202329Z 2020-12-29 14:45:54,230 INFO 
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - 
Starting DefaultLeaderRetrievalService with 
KubernetesLeaderRetrievalDriver\{configMapName='mta-flink-dispatcher-leader'}.
2020-12-29T06:45:54.230219281Z 2020-12-29 14:45:54,229 DEBUG 
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - WebSocket 
successfully opened
2020-12-29T06:45:54.230353912Z 2020-12-29 14:45:54,230 INFO 
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - 
Starting DefaultLeaderElectionService with 
KubernetesLeaderElectionDriver\{configMapName='mta-flink-resourcemanager-leader'}.
2020-12-29T06:45:54.237004177Z 2020-12-29 14:45:54,236 DEBUG 
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Leader 
changed from null to 6f6479c6-86cc-4d62-84f9-37ff968bd0e5
2020-12-29T06:45:54.237024655Z 2020-12-29 14:45:54,236 INFO 
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - 
New leader elected 6f6479c6-86cc-4d62-84f9-37ff968bd0e5 for 
mta-flink-restserver-leader.
2020-12-29T06:45:54.237027811Z 2020-12-29 14:45:54,236 DEBUG 
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - 
Successfully Acquired leader lease 'ConfigMapLock: default - 
mta-flink-restserver-leader (6f6479c6-86cc-4d62-84f9-37ff968bd0e5)'
2020-12-29T06:45:54.237297376Z 2020-12-29 14:45:54,237 DEBUG 
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Grant 
leadership to contender http://mta-flink-jobmanager:8081 with session ID 
9587e13f-322f-4cd5-9fff-b4941462be0f.
2020-12-29T06:45:54.237353551Z 2020-12-29 14:45:54,237 INFO 
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - 
http://mta-flink-jobmanager:8081 was granted leadership with 
leaderSessionID=9587e13f-322f-4cd5-9fff-b4941462be0f
2020-12-29T06:45:54.237440354Z 2020-12-29 14:45:54,237 DEBUG 
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - 
Confirm leader session ID 9587e13f-322f-4cd5-9fff-b4941462be0f for leader 
http://mta-flink-jobmanager:8081.
2020-12-29T06:45:54.254555127Z 2020-12-29 14:45:54,254 DEBUG 
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Leader 
changed from null to 6f6479c6-86cc-4d62-84f9-37ff968bd0e5
2020-12-29T06:45:54.254588299Z 2020-12-29 14:45:54,254 INFO 
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - 
New leader elected 6f6479c6-86cc-4d62-84f9-37ff968bd0e5 for 
mta-flink-resourcemanager-leader.
2020-12-29T06:45:54.254628053Z 2020-12-29 14:45:54,254 DEBUG 
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - 
Successfully Acquired leader lease 'ConfigMapLock: default - 
mta-flink-resourcemanager-leader (6f6479c6-86cc-4d62-84f9-37ff968bd0e5)'
2020-12-29T06:45:54.254871569Z 2020-12-29 14:45:54,254 DEBUG 
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Grant 
leadership to contender LeaderContender: StandaloneResourceManager with session 
ID b1730dc6-0f94-49f4-b519-56917f3027b7.
2020-12-29T06:45:54.256608291Z 202

Re: [VOTE] Release 1.12.1, release candidate #1