Re: MODERATE for dev@flink.apache.org

2020-06-02 Thread Henry Saputra
Hi,

Looks like you have not subscribe to Apache Flink dev@ mailing list.

Please subscribe by following instruction here to continue with the
discussions and receive followups:
https://flink.apache.org/community.html#mailing-lists


Thanks,

Henry Saputra
On behalf of Apache Flink PMC


>
>
> -- Forwarded message --
> From: Teng Fei Liao 
> To: dev@flink.apache.org
> Cc:
> Bcc:
> Date: Tue, 2 Jun 2020 03:04:38 -0400
> Subject: Common HA setups
>
> Hi Flink devs!
>
> After reading through documentation and reading posts others have made
> online for their setups, it seems like there’s similarities in achieving HA
> with just a single job manager. For example, the yarn setup
> 
> specifies only a single job manager is necessary and will be restarted on
> failures. This Kubernetes post
> 
> has a similar single job manager setup. It has a fill-in for zookeeper but
> I think abstractly, the two have these same features in common:
>
> 1. Persistent storage (the high-availability.storageDir config value)
>
> 2. Low latency job manager restart times.
>
> For our setup, we're actually experimenting with a variation of the
> kubernetes set up that removes zookeeper altogether by implementing a file
> based HighAvailabilityServices and trivial leader election services. Given
> the relative simplicity of the setup and code, I was wondering how
> recommended and supported this variant is. Potentially, this could be made
> available by default to help other users simplify their setups. Curious
> what your thoughts are.
>
> Thanks,
>
> Teng.
>


[jira] [Created] (FLINK-18060) Add notice for heap-based state backend of state access reference problem

2020-06-02 Thread Yun Tang (Jira)
Yun Tang created FLINK-18060:


 Summary: Add notice for heap-based state backend of state access 
reference problem
 Key: FLINK-18060
 URL: https://issues.apache.org/jira/browse/FLINK-18060
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: Yun Tang


Currently, when talking about the limitation of {{MemoryStateBackend}} and 
{{FsStateBackend}} in 
[documentation|https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html#the-memorystatebackend],
 Flink only focus on the memory checkpoint size limit. However, current 
documentation forgets to talk about the reference problem, which added in 
{{CopyOnWriteStateMap}}:

{noformat}
IMPORTANT: the contracts for this class rely on the user not holding any 
references to objects returned by this map beyond the life cycle of per-element 
operations. Or phrased differently, all get-update-put operations on a mapping 
should be within one call of processElement. Otherwise, the user must take care 
of taking deep copies, e.g. for caching purposes.
{noformat}

This actually also applies to {{NestedStateMap}}.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[Discussion] flink elasticsearch connector supports

2020-06-02 Thread Jacky Lau
Hi all!

We have started some preliminary work on the flink elasticsearch integration
(es connector for es version7) at hikvision research institute. 
It seems that the integration should think throughly. And we want to
contribute our code for the conmunity. 
So I think I should open a discussion thread with my initial ideas to get
some early feedback.

*Minimal background*
Some companies truly have the scenario of reading elasticsearch. You can see
our initial discussion here.
https://issues.apache.org/jira/browse/FLINK-16713 

*Design docs*
Here is our design docs for adapt diffirent es version such as 5,6,7
Becase we can access google docs, so we just use yuque project.
https://www.yuque.com/jackylau-sc7w6/bve18l/14a2ad5b7f86998433de83dd0f8ec067

We are very looking forwary your response.

Cheers,
Jacky Lau



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/


[jira] [Created] (FLINK-18061) TableResult#collect should return closeable iterator to avoid resource leak

2020-06-02 Thread godfrey he (Jira)
godfrey he created FLINK-18061:
--

 Summary: TableResult#collect should return closeable iterator to 
avoid resource leak
 Key: FLINK-18061
 URL: https://issues.apache.org/jira/browse/FLINK-18061
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Reporter: godfrey he
 Fix For: 1.11.0


as discussed in ML: 
http://mail-archives.apache.org/mod_mbox/flink-dev/202005.mbox/%3CCADQYLGuk%2BnnrPv-PR6Gi7D_rZqp_DhjfA%3DVtkRB-aGPxYxOQPw%40mail.gmail.com%3E,
 we should return a closeable iterator for TableResult#collect method *to avoid 
resource leak*. The suggested change is:

public interface TableResult {

  CloseableRowIterator collect();

}

public interface CloseableRowIterator extends Iterator, AutoCloseable {

} 

This change does not break current api.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18062) prometheus metrics reporter support custom labels

2020-06-02 Thread Luke Sun (Jira)
Luke Sun created FLINK-18062:


 Summary: prometheus metrics reporter support custom labels
 Key: FLINK-18062
 URL: https://issues.apache.org/jira/browse/FLINK-18062
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Metrics
Affects Versions: 1.10.1
Reporter: Luke Sun
 Fix For: 1.12.0


Currently prometheus pushgateway metrics reporter supports custom labels 
through 
"[groupingkey|[https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#groupingkey]]";
 for global labels of all metrics.

I propose also supporting custom global labels for prometheus metrics reporter. 
It is a very common scenario to pass labels in a production environment, e.g. 
project name.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18063) Fix the race condition for aborting current checkpoint in CheckpointBarrierUnaligner#processEndOfPartition

2020-06-02 Thread Zhijiang (Jira)
Zhijiang created FLINK-18063:


 Summary: Fix the race condition for aborting current checkpoint in 
CheckpointBarrierUnaligner#processEndOfPartition
 Key: FLINK-18063
 URL: https://issues.apache.org/jira/browse/FLINK-18063
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.11.0
Reporter: Zhijiang
Assignee: Zhijiang
 Fix For: 1.11.0, 1.12.0


In the handle of CheckpointBarrierUnaligner#processEndOfPartition, it only 
aborts the current checkpoint by judging the condition of pending checkpoint 
from task thread processing, so it will miss one scenario that checkpoint 
triggered by notifyBarrierReceived from netty thread.

The proper fix should also judge the pending checkpoint inside 
ThreadSafeUnaligner in order to abort it and reset internal variables in case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18064) Update documentation taking unaligned checkpoints into the account

2020-06-02 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-18064:
--

 Summary: Update documentation taking unaligned checkpoints into 
the account
 Key: FLINK-18064
 URL: https://issues.apache.org/jira/browse/FLINK-18064
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation, Runtime / Checkpointing
Reporter: Piotr Nowojski
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: [Discussion] flink elasticsearch connector supports

2020-06-02 Thread Robert Metzger
Hi Jacky,
thanks a lot for starting the discussion. I have no objections to adding
support for reading data from ElasticSearch as well, as long as we clearly
state the performance and correctness implications / guarantees in the
Flink documentation.

On Tue, Jun 2, 2020 at 9:52 AM Jacky Lau  wrote:

> Hi all!
>
> We have started some preliminary work on the flink elasticsearch
> integration
> (es connector for es version7) at hikvision research institute.
> It seems that the integration should think throughly. And we want to
> contribute our code for the conmunity.
> So I think I should open a discussion thread with my initial ideas to get
> some early feedback.
>
> *Minimal background*
> Some companies truly have the scenario of reading elasticsearch. You can
> see
> our initial discussion here.
> https://issues.apache.org/jira/browse/FLINK-16713
>
> *Design docs*
> Here is our design docs for adapt diffirent es version such as 5,6,7
> Becase we can access google docs, so we just use yuque project.
>
> https://www.yuque.com/jackylau-sc7w6/bve18l/14a2ad5b7f86998433de83dd0f8ec067
>
> We are very looking forwary your response.
>
> Cheers,
> Jacky Lau
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>


[jira] [Created] (FLINK-18065) Add documentation for new scalar/table functions

2020-06-02 Thread Timo Walther (Jira)
Timo Walther created FLINK-18065:


 Summary: Add documentation for new scalar/table functions
 Key: FLINK-18065
 URL: https://issues.apache.org/jira/browse/FLINK-18065
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Timo Walther
Assignee: Timo Walther


Write documentation for scalar/table functions of FLIP-65.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18066) Add documentation for how to develop a new table source/sink

2020-06-02 Thread Timo Walther (Jira)
Timo Walther created FLINK-18066:


 Summary: Add documentation for how to develop a new table 
source/sink
 Key: FLINK-18066
 URL: https://issues.apache.org/jira/browse/FLINK-18066
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Timo Walther
Assignee: Timo Walther


Covers how to write a custom source/sink and format using FLIP-95 interfaces.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Questions around MemoryStateBackend size limits

2020-06-02 Thread vishalovercome
The documentation says: 

*The size of each individual state is by default limited to 5 MB. This value
can be increased in the constructor of the MemoryStateBackend.*

1. I want to know what would happen if a job ended up adding elements to a
state variable causing its size to exceed 5MB. There are other questions I
have around this topic.
2. The doc mentions that akka frame size is the upper limit on state size.
What would happen if we were to exceed that as well? Would it cause the
program to fail or would it only affect checkpointing (as communication
between job manager and task manager would breakdown)
3. If the size is within 5MB but the size of the checkpoint (additions,
removals, updates) were to be greater than 5MB (or the akka frame size),
then would the checkpoint fail?

It will also help if you could provide this information within the
documentation itself. 



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/


[jira] [Created] (FLINK-18067) Invalid default value for yarnMinAllocationMB in YarnClusterDescriptor

2020-06-02 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-18067:
--

 Summary: Invalid default value for yarnMinAllocationMB in 
YarnClusterDescriptor
 Key: FLINK-18067
 URL: https://issues.apache.org/jira/browse/FLINK-18067
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / YARN
Affects Versions: 1.10.1
Reporter: Jiayi Liao


Currently Flink sets {{yarnMinAllocationMB}} default value to 0, which will 
crash the job in normalizing the allocation memory. There should be two minor 
changes after discussion with [~fly_in_gis]:

* Make the default value to 
YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB, not 0
* Add pre-check for yarnMinAllocationMB, it should be greater than 0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: [DISCUSS] Stability guarantees for @PublicEvolving classes

2020-06-02 Thread Till Rohrmann
FYI: The API and binary compatibility check has been activated [1].

[1] https://issues.apache.org/jira/browse/FLINK-17844

Cheers,
Till

On Mon, May 18, 2020 at 1:11 PM Chesnay Schepler  wrote:

> yes
>
> On 18/05/2020 13:05, Piotr Nowojski wrote:
> > How would we enforce this change? That we would enable different, more
> strict compatibility checks on the `release-1.x` branches?
> >
> > Piotrek
> >
> >> On 16 May 2020, at 08:33, Congxian Qiu  wrote:
> >>
> >> Sorry for the late jump in.
> >>
> >> +1 to keep the compatibility of @PublicEvolving between minor
> >> releases(x.y.a -> x.y.b), as a user I always think this as a bug-fix
> >> release, break the compatibility between minor releases may give users a
> >> surprise.
> >>
> >> As the previous emails said, how and when will a @PublicEvolving
> >> become @Public, and I'm not sure if we can have a technical solution to
> >> keep such a rule. (In my opinion, check such things -- change
> >> @PublicEvolving to @Public -- manually may not so easy)
> >>
> >> Best
> >> Congxian
> >>
> >>
> >> Till Rohrmann  于2020年5月15日周五 下午9:18写道:
> >>
> >>> The vote thread can be found here
> >>>
> >>>
> https://lists.apache.org/thread.html/rc58099fb0e31d0eac951a7bbf7f8bda8b7b65c9ed0c04622f5333745%40%3Cdev.flink.apache.org%3E
> >>> .
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Fri, May 15, 2020 at 3:03 PM Till Rohrmann 
> >>> wrote:
> >>>
>  I completely agree that there are many other aspect of our guarantees
> and
>  processes around the @Public and @PublicEvolving classes which need
> to be
>  discussed and properly defined. For the sake of keeping this
> discussion
>  thread narrowly scoped, I would suggest to start a separate discussion
>  about the following points (not exhaustive):
> 
>  - What should be annotated with @Public and @PublicEvolving?
>  - Process for transforming @PublicEvolving into @Public; How to ensure
>  that @PublicEvolving will eventually be promoted to @Public?
>  - Process of retiring a @Public/@PublicEvolving API
> 
>  I will start a vote thread about the change I proposed here which is
> to
>  ensure API and binary compatibility for @PublicEvolving classes
> between
>  bugfix releases (x.y.z and x.y.u).
> 
>  Cheers,
>  Till
> 
>  On Fri, May 15, 2020 at 6:33 AM Zhu Zhu  wrote:
> 
> > +1 for "API + binary compatibility for @PublicEvolving classes for
> all
> >>> bug
> > fix
> > releases in a minor release (x.y.z is compatible to x.y.u)"
> >
> > This @PublicEnvolving would then be a hard limit to changes.
> > So it's important to rethink the policy towards using it, as Stephan
> > proposed.
> >
> > I think any Flink interfaces that are visible to users should be
> > explicitly
> > marked as @Public or @PublicEnvolving.
> > Any other interfaces should not be marked as
> @Public/@PublicEnvolving.
> > This would be essential for us to check whether we are breaking any
> user
> > faced interfaces unexpectedly.
> > The only exception would be the case that we had to expose a
> >>> method/class
> > due to implementation limitations, it should be explicitly marked it
> > as @Internal.
> >
> > Thanks,
> > Zhu Zhu
> >
> > Yun Tang  于2020年5月15日周五 上午11:41写道:
> >
> >> +1 for this idea, and I also like Xintong's suggestion to make it
> >> explicitly when the @PublicEvolving API could upgrade to @Public
> API.
> >> If we have the rule to upgrade API stable level but not define the
> >>> clear
> >> timeline, I'm afraid not everyone have the enthusiasm to upgrade
> this.
> >>
> >> The minor suggestion is that I think two major release (which is
> x.y.0
> > as
> >> Chesnay clarified) might be a bit quick. From the release history
> [1],
> >> Flink bump major version every 3 ~ 6 months and two major release
> gap
> >> could only be at least half a year.
> >> I think half a year might be a bit too frequent for users to collect
> >> enough feedbacks, and upgrading API stable level every 3 major
> >>> versions
> >> should be better.
> >>
> >> [1] https://flink.apache.org/downloads.html#flink
> >>
> >> Best
> >> Yun Tang
> >>
> >>
> >> 
> >> From: Xintong Song 
> >> Sent: Friday, May 15, 2020 11:04
> >> To: dev 
> >> Subject: Re: [DISCUSS] Stability guarantees for @PublicEvolving
> >>> classes
> >> ### Documentation on API compatibility policies
> >>
> >> Do we have any formal documentation about the API compatibility
> > policies?
> >> The only things I found are:
> >>
> >>- In the release announcement (take 1.10.0 as an example) [1]:
> >>"This version is API-compatible with previous 1.x releases for
> APIs
> >>annotated with the @Public annotation."
> >>- JavaDoc for Public [2] and PublicEvolving [3].
> >>>

Re: [DISCUSS] (Document) Backwards Compatibility of Savepoints

2020-06-02 Thread Yun Tang
Since Flink lacks of such kind of experiments to ensure the backwards 
compatibility of savepoints before, especially those built-in operators with 
their own operator state.
I am afraid we need huge energy to cover all cases to give the most correct 
result.

I prefer to just point out this in documentation to say explicitly Flink does 
not guarantee such kind of backwards compatibility.

Best
Yun Tang

From: Ufuk Celebi 
Sent: Wednesday, May 27, 2020 16:42
To: dev@flink.apache.org 
Subject: Re: [DISCUSS] (Document) Backwards Compatibility of Savepoints

I agree with Konstantin and Steven that it makes sense to point this out
explicitly.

I think that the following would be helpful:

1/ Mention breaking compatibility in release notes

2/ Update the linked table to reflect compatibilities while pointing out
what the community commits to maintain going forward (e.g. "happens to
work" vs. "guaranteed to work")

In general, the table is quite large. Would it make sense to order the
releases in reverse order (assuming that the table is more relevant for
recent releases)?

– Ufuk

On Tue, May 26, 2020 at 8:36 PM Steven Wu  wrote:

> > A use case for this might be when you want to rollback a framework
> upgrade (after some time) due to e.g. a performance
> or stability issue.
>
> Downgrade (that Konstantin called out) is an important and realistic
> scenario. It will be great to support backward compatibility for savepoint
> or at least document any breaking change.
>
> On Tue, May 26, 2020 at 4:39 AM Piotr Nowojski 
> wrote:
>
> > Hi,
> >
> > It might have been implicit choice, but so far we were not supporting the
> > scenario that you are asking for. It has never been tested and we have
> > lot’s of state migration code sprinkled among our code base (for example
> > upgrading state fields of the operators like [1]), that only supports
> > upgrades, not downgrades.
> >
> > Also we do not have testing infrastructure for checking the downgrades.
> We
> > would need to check if save points taken from master branch, are readable
> > by previous releases (not release branch!).
> >
> > So all in all, I don’t think it can be easily done. It would require some
> > effort to start maintaining backward compatibility.
> >
> > Piotrek
> >
> > [1]
> >
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011#migrateNextTransactionalIdHindState
> >
> > > On 26 May 2020, at 13:18, Konstantin Knauf  wrote:
> > >
> > > Hi everyone,
> > >
> > > I recently stumbled across the fact that Savepoints created with Flink
> > 1.11
> > > can not be read by Flink 1.10. A use case for this might be when you
> want
> > > to rollback a framework upgrade (after some time) due to e.g. a
> > performance
> > > or stability issue.
> > >
> > > From the documentation [1] it seems as if the Savepoint format is
> > generally
> > > only forward-compatible although in many cases it is actually also
> > > backwards compatible (e.g. Savepoint taken in Flink 1.10, restored with
> > > Flink 1.9).
> > >
> > > Was it a deliberate choice not to document any backwards compatibility?
> > If
> > > not, should we add the missing entries in the compatibility table?
> > >
> > > Thanks,
> > >
> > > Konstantin
> > >
> > > [1]
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#compatibility-table
> > >
> > > --
> > >
> > > Konstantin Knauf
> > >
> > > https://twitter.com/snntrable
> > >
> > > https://github.com/knaufk
> >
> >
>


[jira] [Created] (FLINK-18068) Job stuck after IllegalStateException in scheduling

2020-06-02 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-18068:
--

 Summary: Job stuck after IllegalStateException in scheduling
 Key: FLINK-18068
 URL: https://issues.apache.org/jira/browse/FLINK-18068
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / YARN
Affects Versions: 1.10.1
Reporter: Jiayi Liao


The job will stop but still be alive with doing nothing forever if any non 
fatal exception is thrown from interacting with YARN. Here is the example :


{code:java}

java.lang.IllegalStateException: The RMClient's and YarnResourceManagers 
internal state about the number of pending container requests for resource 
 has diverged. Number client's pending container 
requests 40 != Number RM's pending container requests 0.
at 
org.apache.flink.util.Preconditions.checkState(Preconditions.java:217) 
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.yarn.YarnResourceManager.getPendingRequestsAndCheckConsistency(YarnResourceManager.java:518)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.yarn.YarnResourceManager.onContainersOfResourceAllocated(YarnResourceManager.java:431)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:395)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) 
[flink-ljy-1.0.jar:?]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) 
[flink-ljy-1.0.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) 
[flink-ljy-1.0.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) 
[flink-ljy-1.0.jar:?]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at akka.actor.ActorCell.invoke(ActorCell.scala:561) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at akka.dispatch.Mailbox.run(Mailbox.scala:225) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: [NOTICE] Release guide updated for updating japicmp configuration

2020-06-02 Thread Till Rohrmann
Thanks for activating the checks and providing the tooling Chesnay.

Cheers,
Till

On Fri, May 29, 2020 at 6:51 PM Yu Li  wrote:

> Thanks Chesnay for the efforts!
>
> Best Regards,
> Yu
>
>
> On Fri, 29 May 2020 at 18:03, Piotr Nowojski  wrote:
>
> > Thanks Chesney for adding those scripts and configuring checks!
> >
> > Piotrek
> >
> > > On 29 May 2020, at 10:04, Chesnay Schepler  wrote:
> > >
> > > Hello everyone,
> > >
> > > We recently decided to enforce compatibility for @PublicEvolving APIs
> > for minor releases.
> > >
> > > This requires modifications to the japicmp-maven-plugin execution on
> the
> > corresponding release-X.Y branch after X.Y.Z was released.
> > >
> > > In FLINK-17844 new tooling was added to take care of this
> > (tools/releasing/updated_japicmp_configuration.sh), but it must be run
> > manually by the release manager, after the release has concluded.
> > >
> > > Note that this is also run automatically when an RC is created, as a
> > final safeguard in case the manual step is missed.
> > >
> > > I have amended the release guide accordingly:
> > >
> > > Update japicmp configuration
> > >
> > > Update the japicmp reference version and enable API compatibility
> checks
> > for @PublicEvolving  APIs on the corresponding SNAPSHOT branch.
> > >
> > > For a new major release (x.y.0), run the same command also on the
> master
> > branch for updating the japicmp reference version.
> > >
> > > tools $ NEW_VERSION=$RELEASE_VERSION
> > releasing/update_japicmp_configuration.sh
> > > tools $ cd ..
> > > $ git add *
> > > $ git commit -m "Update japicmp configuration for $RELEASE_VERSION"
> >
> >
>


Re: [ANNOUNCE] Apache Flink-shaded 11.0 released

2020-06-02 Thread Till Rohrmann
Thanks for creating the latest flink-shaded release Chesnay!

Cheers,
Till

On Sun, May 31, 2020 at 1:25 PM Chesnay Schepler  wrote:

> The Apache Flink community is very happy to announce the release of
> Apache Flink-shaded 11.0.
>
> The flink-shaded project contains a number of shaded dependencies for
> Apache Flink.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data
> streaming applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347784
>
> We would like to thank all contributors of the Apache Flink community
> who made this release possible!
>
> Regards,
> Chesnay
>
>


Re: [ANNOUNCE] Apache Flink-shaded 11.0 released

2020-06-02 Thread Yu Li
Thanks Chesnay!

Best Regards,
Yu


On Tue, 2 Jun 2020 at 18:21, Till Rohrmann  wrote:

> Thanks for creating the latest flink-shaded release Chesnay!
>
> Cheers,
> Till
>
> On Sun, May 31, 2020 at 1:25 PM Chesnay Schepler 
> wrote:
>
> > The Apache Flink community is very happy to announce the release of
> > Apache Flink-shaded 11.0.
> >
> > The flink-shaded project contains a number of shaded dependencies for
> > Apache Flink.
> >
> > Apache Flink® is an open-source stream processing framework for
> > distributed, high-performing, always-available, and accurate data
> > streaming applications.
> >
> > The release is available for download at:
> > https://flink.apache.org/downloads.html
> >
> > The full release notes are available in Jira:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347784
> >
> > We would like to thank all contributors of the Apache Flink community
> > who made this release possible!
> >
> > Regards,
> > Chesnay
> >
> >
>


[jira] [Created] (FLINK-18069) Scaladocs not building since inner Java interfaces cannot be recognized

2020-06-02 Thread Tzu-Li (Gordon) Tai (Jira)
Tzu-Li (Gordon) Tai created FLINK-18069:
---

 Summary: Scaladocs not building since inner Java interfaces cannot 
be recognized
 Key: FLINK-18069
 URL: https://issues.apache.org/jira/browse/FLINK-18069
 Project: Flink
  Issue Type: Bug
  Components: API / Scala, Documentation
Affects Versions: 1.11.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai


Error:

{code}
/home/buildslave/slave/flink-docs-master/build/flink-scala/src/main/java/org/apache/flink/api/scala/typeutils/Tuple2CaseClassSerializerSnapshot.java:98:
 error: not found: type OuterSchemaCompatibility
protected OuterSchemaCompatibility 
resolveOuterSchemaCompatibility(ScalaCaseClassSerializer> 
newSerializer) {
  ^
/home/buildslave/slave/flink-docs-master/build/flink-scala/src/main/java/org/apache/flink/api/scala/typeutils/TraversableSerializerSnapshot.java:101:
 error: not found: type OuterSchemaCompatibility
protected OuterSchemaCompatibility 
resolveOuterSchemaCompatibility(TraversableSerializer newSerializer) {
  ^
/home/buildslave/slave/flink-docs-master/build/flink-scala/src/main/java/org/apache/flink/api/scala/typeutils/ScalaCaseClassSerializerSnapshot.java:106:
 error: not found: type OuterSchemaCompatibility
protected OuterSchemaCompatibility 
resolveOuterSchemaCompatibility(ScalaCaseClassSerializer newSerializer) {
  ^
{code}

This is a similar issue as reported here: 
https://github.com/scala/bug/issues/10509.

This seems to be a problem with Scala 2.12.x. The only workaround is to
redundantly add the full-length qualifiers for such interfaces.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18070) Time attribute been materialized after sub graph optimize

2020-06-02 Thread YufeiLiu (Jira)
YufeiLiu created FLINK-18070:


 Summary: Time attribute been materialized after sub graph optimize
 Key: FLINK-18070
 URL: https://issues.apache.org/jira/browse/FLINK-18070
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.10.0
Reporter: YufeiLiu


Hi, I want to use window aggregate after create temporary, and has multiple 
sinks. But throw exception:

{code:java}
java.lang.AssertionError: type mismatch:
ref:
TIME ATTRIBUTE(PROCTIME) NOT NULL
input:
TIMESTAMP(3) NOT NULL
{code}

I look into the optimizer logic, there is comment at 
{{CommonSubGraphBasedOptimizer}}:
"1. In general, for multi-sinks users tend to use VIEW which is a natural 
common sub-graph."

After sub graph optimize, time attribute from source have been convert to basic 
TIMESTAMP type according to {{FlinkRelTimeIndicatorProgram}}. But my create 
view sql is simple query, I think didn't need to materialized time attribute in 
theory.

Here is my code:

{code:java}
// connector.type COLLECTION is for debug use
tableEnv.sqlUpdate("CREATE TABLE source (\n" +
"`ts` AS PROCTIME(),\n" +
"`order_type` INT\n" +
") WITH (\n" +
"'connector.type' = 'COLLECTION',\n" +
"'format.type' = 'json'\n" +
")\n");
tableEnv.createTemporaryView("source_view", tableEnv.sqlQuery("SELECT * FROM 
source"));
tableEnv.sqlUpdate("CREATE TABLE sink (\n" +
"`result` BIGINT\n" +
") WITH (\n" +
"'connector.type' = 'COLLECTION',\n" +
"'format.type' = 'json'\n" +
")\n");
tableEnv.sqlUpdate("INSERT INTO sink \n" +
"SELECT\n" +
"COUNT(1)\n" +
"FROM\n" +
"`source_view`\n" +
"WHERE\n" +
" `order_type` = 33\n" +
"GROUP BY\n" +
"TUMBLE(`ts`, INTERVAL '5' SECOND)\n");
tableEnv.sqlUpdate("INSERT INTO sink \n" +
"SELECT\n" +
"COUNT(1)\n" +
"FROM\n" +
"`source_view`\n" +
"WHERE\n" +
" `order_type` = 34\n" +
"GROUP BY\n" +
"TUMBLE(`ts`, INTERVAL '5' SECOND)\n");
{code}






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18071) CoordinatorEventsExactlyOnceITCase.checkListContainsSequence fails on CI

2020-06-02 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-18071:
--

 Summary: 
CoordinatorEventsExactlyOnceITCase.checkListContainsSequence fails on CI
 Key: FLINK-18071
 URL: https://issues.apache.org/jira/browse/FLINK-18071
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination, Tests
Affects Versions: 1.12.0
Reporter: Robert Metzger
 Fix For: 1.12.0


CI: 
https://dev.azure.com/georgeryan1322/Flink/_build/results?buildId=330&view=logs&j=6e58d712-c5cc-52fb-0895-6ff7bd56c46b&t=f30a8e80-b2cf-535c-9952-7f521a4ae374

{code}
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.795 s 
<<< FAILURE! - in 
org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase
[ERROR] 
test(org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase)
  Time elapsed: 4.647 s  <<< FAILURE!
java.lang.AssertionError: List did not contain expected sequence of 200 
elements, but was: [152, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 
36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 
56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 
76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 
96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 
112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 
128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 
144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 
160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 
176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 
192, 193, 194, 195, 196, 197, 198, 199]
at org.junit.Assert.fail(Assert.java:88)
at 
org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.failList(CoordinatorEventsExactlyOnceITCase.java:160)
at 
org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.checkListContainsSequence(CoordinatorEventsExactlyOnceITCase.java:148)
at 
org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.test(CoordinatorEventsExactlyOnceITCase.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18072) HBaseLookupFunction can not work with new internal data structure RowData

2020-06-02 Thread Leonard Xu (Jira)
Leonard Xu created FLINK-18072:
--

 Summary: HBaseLookupFunction can not work with new internal data 
structure RowData
 Key: FLINK-18072
 URL: https://issues.apache.org/jira/browse/FLINK-18072
 Project: Flink
  Issue Type: Bug
  Components: Connectors / HBase
Affects Versions: 1.11.0
Reporter: Leonard Xu
 Fix For: 1.11.0, 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18073) AvroRowDataSerializationSchema is not always serializable

2020-06-02 Thread Seth Wiesman (Jira)
Seth Wiesman created FLINK-18073:


 Summary: AvroRowDataSerializationSchema is not always serializable
 Key: FLINK-18073
 URL: https://issues.apache.org/jira/browse/FLINK-18073
 Project: Flink
  Issue Type: Improvement
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: Seth Wiesman
 Fix For: 1.11.0, 1.12.0


AvroRowDeserializationSchema member runtimeConverter may hold a reference to 
non serializable avro schema fields. This value should be transient and 
generated inside of open. 


{code:java}
Caused by: java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema
at 
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
~[?:1.8.0_252]
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378) 
~[?:1.8.0_252]
at 
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) 
~[?:1.8.0_252]
at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
~[?:1.8.0_252]
at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
~[?:1.8.0_252]
at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
~[?:1.8.0_252]
at 
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
~[?:1.8.0_252]
at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
~[?:1.8.0_252]
at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
~[?:1.8.0_252]
at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
~[?:1.8.0_252]
at 
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
~[?:1.8.0_252]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
~[?:1.8.0_252]
at 
org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:586)
 ~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
at 
org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:133) 
~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
at 
org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:126) 
~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
at 
org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:71) 
~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.(FlinkKafkaProducer.java:632)
 ~[flink-sql-connector-kafka_2.12-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.(FlinkKafkaProducer.java:522)
 ~[flink-sql-connector-kafka_2.12-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.(FlinkKafkaProducer.java:483)
 ~[flink-sql-connector-kafka_2.12-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSink.createKafkaProducer(KafkaDynamicSink.java:60)
 ~[flink-sql-connector-kafka_2.12-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSinkBase.getSinkRuntimeProvider(KafkaDynamicSinkBase.java:84)
 ~[flink-sql-connector-kafka_2.12-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
at 
org.apache.flink.table.planner.plan.nodes.common.CommonPhysicalSink.createSinkTransformation(CommonPhysicalSink.scala:69)
 ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
at 
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:108)
 ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
at 
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:43)
 ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
at 
org.apache.flink.table.planner.plan.nodes.exec.ExecNode.translateToPlan(ExecNode.scala:58)
 ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
at 
org.apache.flink.table.planner.plan.nodes.exec.ExecNode.translateToPlan$(ExecNode.scala:56)
 ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
at 
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlan(StreamExecSink.scala:43)
 ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
at 
org.apache.flink.table.planner.delegation.StreamPlanner.$anonfun$translateToPlan$1(StreamPlanner.scala:67)
 ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
at 
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233) 
~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
at scala.collection.Iterator.foreach(Iterator.scala:937) 
~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
at scala.collection.Iterator.foreach$(Iterator.scala:937) 
~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
   

[jira] [Created] (FLINK-18074) Confirm checkpoint completed on task side would not fail the task if exception thrown out

2020-06-02 Thread Yun Tang (Jira)
Yun Tang created FLINK-18074:


 Summary: Confirm checkpoint completed on task side would not fail 
the task if exception thrown out
 Key: FLINK-18074
 URL: https://issues.apache.org/jira/browse/FLINK-18074
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing, Runtime / Task
Affects Versions: 1.11.0
Reporter: Yun Tang
 Fix For: 1.11.0


FLINK-17350 let the task fail immediately once sync phase of checkpoint failed. 
However, the included commit ['Simplify checkpoint exception 
handling'|https://github.com/apache/flink/pull/12101/commits/a2cd3daceca16ae841119d94a24328b4af37dcd8]
 actually would not fail the task if the runnable of {{() -> 
notifyCheckpointComplete}} throwing exception out.
In a nutshell, this actually changes previous checkpoint exception handling.

Moreover, that part of code also affect the implemented code of 
{{notifyCheckpointAbortAsync}} when I introduce {{notifyCheckpointAborted}} on 
task side. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18075) Kafka connector does not call open method of (de)serialization schema

2020-06-02 Thread Seth Wiesman (Jira)
Seth Wiesman created FLINK-18075:


 Summary: Kafka connector does not call open method of 
(de)serialization schema
 Key: FLINK-18075
 URL: https://issues.apache.org/jira/browse/FLINK-18075
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka, Formats (JSON, Avro, Parquet, ORC, 
SequenceFile)
Affects Versions: 1.11.0, 1.12.0
Reporter: Seth Wiesman


The Kafka consumer and producer do not  call the open methods of plain 
(De)SerializationSchema interfaces. Only the Keyed and Kafka specific 
interfaces. The updated SQL implementations such as 
AvroRowDataSeriailzationSchema use these methods and so SQL queries using avro 
and kafka will fail in a null pointer exception. 

cc [~aljoscha]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18076) Sql client uses wrong class loader when parsing queries

2020-06-02 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-18076:


 Summary: Sql client uses wrong class loader when parsing queries
 Key: FLINK-18076
 URL: https://issues.apache.org/jira/browse/FLINK-18076
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.11.0, 1.12.0
Reporter: Dawid Wysakowicz
 Fix For: 1.11.0


Sql-client when parsing queries does not use the user class loader from 
{{ExecutionContext}}. This makes it impossible to query any sources if the 
dependencies are added with {{-j}} flag.

In order to reproduce it try querying e.g. KafkaDynamicSource with
{code}
CREATE TABLE MyUserTable (
   f0 BIGINT
) WITH (
  'connector' = 'kafka',   

  'topic' = 'topic_name', -- required: topic name from which the table is read

 -- required: specify the Kafka server connection string
  'properties.bootstrap.servers' = 'localhost:9092',
  -- required for Kafka source, optional for Kafka sink, specify consumer group
  'properties.group.id' = 'testGroup',
  -- optional: valid modes are "earliest-offset", "latest-offset", 
"group-offsets", "specific-offsets" or "timestamp"
'scan.startup.mode' = 'earliest-offset',

   'format' = 'avro'
);


SELECT * FROM MyUserTable;
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: [DISCUSS] (Document) Backwards Compatibility of Savepoints

2020-06-02 Thread Steven Wu
Current Flink documentation is actually pretty clear about no guarantees
for backward compatibility.
https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#compatibility-table

On Tue, Jun 2, 2020 at 3:20 AM Yun Tang  wrote:

> Since Flink lacks of such kind of experiments to ensure the backwards
> compatibility of savepoints before, especially those built-in operators
> with their own operator state.
> I am afraid we need huge energy to cover all cases to give the most
> correct result.
>
> I prefer to just point out this in documentation to say explicitly Flink
> does not guarantee such kind of backwards compatibility.
>
> Best
> Yun Tang
> 
> From: Ufuk Celebi 
> Sent: Wednesday, May 27, 2020 16:42
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS] (Document) Backwards Compatibility of Savepoints
>
> I agree with Konstantin and Steven that it makes sense to point this out
> explicitly.
>
> I think that the following would be helpful:
>
> 1/ Mention breaking compatibility in release notes
>
> 2/ Update the linked table to reflect compatibilities while pointing out
> what the community commits to maintain going forward (e.g. "happens to
> work" vs. "guaranteed to work")
>
> In general, the table is quite large. Would it make sense to order the
> releases in reverse order (assuming that the table is more relevant for
> recent releases)?
>
> – Ufuk
>
> On Tue, May 26, 2020 at 8:36 PM Steven Wu  wrote:
>
> > > A use case for this might be when you want to rollback a framework
> > upgrade (after some time) due to e.g. a performance
> > or stability issue.
> >
> > Downgrade (that Konstantin called out) is an important and realistic
> > scenario. It will be great to support backward compatibility for
> savepoint
> > or at least document any breaking change.
> >
> > On Tue, May 26, 2020 at 4:39 AM Piotr Nowojski 
> > wrote:
> >
> > > Hi,
> > >
> > > It might have been implicit choice, but so far we were not supporting
> the
> > > scenario that you are asking for. It has never been tested and we have
> > > lot’s of state migration code sprinkled among our code base (for
> example
> > > upgrading state fields of the operators like [1]), that only supports
> > > upgrades, not downgrades.
> > >
> > > Also we do not have testing infrastructure for checking the downgrades.
> > We
> > > would need to check if save points taken from master branch, are
> readable
> > > by previous releases (not release branch!).
> > >
> > > So all in all, I don’t think it can be easily done. It would require
> some
> > > effort to start maintaining backward compatibility.
> > >
> > > Piotrek
> > >
> > > [1]
> > >
> >
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011#migrateNextTransactionalIdHindState
> > >
> > > > On 26 May 2020, at 13:18, Konstantin Knauf 
> wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I recently stumbled across the fact that Savepoints created with
> Flink
> > > 1.11
> > > > can not be read by Flink 1.10. A use case for this might be when you
> > want
> > > > to rollback a framework upgrade (after some time) due to e.g. a
> > > performance
> > > > or stability issue.
> > > >
> > > > From the documentation [1] it seems as if the Savepoint format is
> > > generally
> > > > only forward-compatible although in many cases it is actually also
> > > > backwards compatible (e.g. Savepoint taken in Flink 1.10, restored
> with
> > > > Flink 1.9).
> > > >
> > > > Was it a deliberate choice not to document any backwards
> compatibility?
> > > If
> > > > not, should we add the missing entries in the compatibility table?
> > > >
> > > > Thanks,
> > > >
> > > > Konstantin
> > > >
> > > > [1]
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#compatibility-table
> > > >
> > > > --
> > > >
> > > > Konstantin Knauf
> > > >
> > > > https://twitter.com/snntrable
> > > >
> > > > https://github.com/knaufk
> > >
> > >
> >
>


Re: [Discussion] flink elasticsearch connector supports

2020-06-02 Thread Jacky Lau
Hi Robert Metzger:
Thanks for your response. could you please read this docs.
https://www.yuque.com/jackylau-sc7w6/bve18l/14a2ad5b7f86998433de83dd0f8ec067
. Any Is it any problem here? we are worried about
we do not think  throughly. thanks.



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/


[jira] [Created] (FLINK-18077) E2E tests manually for Hive streaming source

2020-06-02 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-18077:


 Summary: E2E tests manually for Hive streaming source
 Key: FLINK-18077
 URL: https://issues.apache.org/jira/browse/FLINK-18077
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Reporter: Jingsong Lee
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18078) E2E tests manually for Hive streaming dim join

2020-06-02 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-18078:


 Summary: E2E tests manually for Hive streaming dim join
 Key: FLINK-18078
 URL: https://issues.apache.org/jira/browse/FLINK-18078
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Reporter: Jingsong Lee
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18079) KafkaShuffle Manual Tests

2020-06-02 Thread Yuan Mei (Jira)
Yuan Mei created FLINK-18079:


 Summary: KafkaShuffle Manual Tests
 Key: FLINK-18079
 URL: https://issues.apache.org/jira/browse/FLINK-18079
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream, Runtime / Checkpointing
Affects Versions: 1.11.0
Reporter: Yuan Mei
 Fix For: 1.11.0


Manual Tests and Results to demonstrate KafkaShuffle is working as expected.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18080) Translate the "Kerberos Authentication Setup and Configuration" page into Chinese

2020-06-02 Thread Yangze Guo (Jira)
Yangze Guo created FLINK-18080:
--

 Summary: Translate the "Kerberos Authentication Setup and 
Configuration" page into Chinese
 Key: FLINK-18080
 URL: https://issues.apache.org/jira/browse/FLINK-18080
 Project: Flink
  Issue Type: Sub-task
Reporter: Yangze Guo






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18081) Fix broken links in "Kerberos Authentication Setup and Configuration" doc

2020-06-02 Thread Yangze Guo (Jira)
Yangze Guo created FLINK-18081:
--

 Summary: Fix broken links in "Kerberos Authentication Setup and 
Configuration" doc
 Key: FLINK-18081
 URL: https://issues.apache.org/jira/browse/FLINK-18081
 Project: Flink
  Issue Type: Bug
Reporter: Yangze Guo


The {{config.html#kerberos-based-security}} is not valid now.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-18082) UnsignedTypeConversionITCase stalls in ch.vorburger.mariadb4j.DB.stop

2020-06-02 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-18082:
--

 Summary: UnsignedTypeConversionITCase stalls in 
ch.vorburger.mariadb4j.DB.stop
 Key: FLINK-18082
 URL: https://issues.apache.org/jira/browse/FLINK-18082
 Project: Flink
  Issue Type: Bug
  Components: Connectors / JDBC
Affects Versions: 1.12.0
Reporter: Robert Metzger


CI: 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2582&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=03dca39c-73e8-5aaf-601d-328ae5c35f20

{code}
2020-06-02T19:01:31.8486456Z 
==
2020-06-02T19:01:31.8487052Z Printing stack trace of Java process 8653
2020-06-02T19:01:31.8487424Z 
==
2020-06-02T19:01:31.8541169Z Picked up JAVA_TOOL_OPTIONS: 
-XX:+HeapDumpOnOutOfMemoryError
2020-06-02T19:01:32.1665740Z 2020-06-02 19:01:32
2020-06-02T19:01:32.1666470Z Full thread dump OpenJDK 64-Bit Server VM 
(25.242-b08 mixed mode):
2020-06-02T19:01:32.1666735Z 
2020-06-02T19:01:32.1667614Z "Attach Listener" #537 daemon prio=9 os_prio=0 
tid=0x7f61f8001000 nid=0x3b9f waiting on condition [0x]
2020-06-02T19:01:32.1668130Zjava.lang.Thread.State: RUNNABLE
2020-06-02T19:01:32.1668311Z 
2020-06-02T19:01:32.1668958Z "flink-akka.actor.default-dispatcher-193" #535 
prio=5 os_prio=0 tid=0x7f6034001000 nid=0x3af7 waiting on condition 
[0x7f61a25b8000]
2020-06-02T19:01:32.1669418Zjava.lang.Thread.State: TIMED_WAITING (parking)
2020-06-02T19:01:32.1669730Zat sun.misc.Unsafe.park(Native Method)
2020-06-02T19:01:32.1670301Z- parking to wait for  <0x80c51528> (a 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
2020-06-02T19:01:32.1670791Zat 
akka.dispatch.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
2020-06-02T19:01:32.1671329Zat 
akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
2020-06-02T19:01:32.1671763Zat 
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
2020-06-02T19:01:32.1672211Zat 
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2020-06-02T19:01:32.1672491Z 
2020-06-02T19:01:32.1673104Z "flink-akka.actor.default-dispatcher-191" #533 
prio=5 os_prio=0 tid=0x7f619801e000 nid=0x3ae1 waiting on condition 
[0x7f60770f1000]
2020-06-02T19:01:32.1673564Zjava.lang.Thread.State: WAITING (parking)
2020-06-02T19:01:32.1673839Zat sun.misc.Unsafe.park(Native Method)
2020-06-02T19:01:32.1674422Z- parking to wait for  <0x80c51528> (a 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
2020-06-02T19:01:32.1674865Zat 
akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
2020-06-02T19:01:32.1675305Zat 
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
2020-06-02T19:01:32.1675751Zat 
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2020-06-02T19:01:32.1676046Z 
2020-06-02T19:01:32.1676669Z "jobmanager-future-thread-2" #466 daemon prio=5 
os_prio=0 tid=0x7f6124001000 nid=0x3795 waiting on condition 
[0x7f61a23b6000]
2020-06-02T19:01:32.1677316Zjava.lang.Thread.State: WAITING (parking)
2020-06-02T19:01:32.1677617Zat sun.misc.Unsafe.park(Native Method)
2020-06-02T19:01:32.1678220Z- parking to wait for  <0x816a4c90> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
2020-06-02T19:01:32.1678702Zat 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
2020-06-02T19:01:32.1679209Zat 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
2020-06-02T19:01:32.1679822Zat 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
2020-06-02T19:01:32.1680422Zat 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
2020-06-02T19:01:32.1680962Zat 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
2020-06-02T19:01:32.1681424Zat 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
2020-06-02T19:01:32.1682062Zat 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2020-06-02T19:01:32.1682445Zat java.lang.Thread.run(Thread.java:748)
2020-06-02T19:01:32.1682656Z 
2020-06-02T19:01:32.1683271Z "Flink-DispatcherRestEndpoint-thread-4" #349 
daemon prio=5 os_prio=0 tid=0x7f618c00a000 nid=0x29a4 waiting on condition 
[0x7f61a029f000]
2020-06-02T19:01:32.1683750Zjava.lang.Thread.State: TIMED_WAITING (parking)
2020-06-02T19:01:32.1684057Zat sun.misc.Unsafe.park(Native Method)
2020-06-02T19:01:32.1684648Z- parking to wait for  <0x8