Re: [DISCUSS] Move Redis and Flume connectors to Apache Bahir and redirect contributions there

[Aljoscha Krettek]

+1

On Thu, 18 Aug 2016 at 03:54 Jark Wu  wrote:

> Good start point. +1
>
>
>
> - Jark Wu
>
> > 在 2016年8月17日，下午11:24，Robert Metzger  写道：
> >
> > Hi,
> >
> > following the discussions in [1], the community decided to offer
> > contributors to submit streaming connectors the Flink community can not
> > maintain to Apache Bahir (there is still the option that we are moving
> > popular connectors to the main Flink codebase)
> >
> > The ActiveMQ connector pull request [2], the Sink for RethinkDB [3], and
> > the streaming sink for HBase [4] are in my opinion examples of connectors
> > we should first give to Bahir.
> >
> > Also, I'm proposing to move the Redis sink [5] and the Flume sink [6]
> from
> > Apache Flink to Bahir. I can not recall any users asking about those
> Sinks
> > on the mailing list.
> >
> > Please object to this proposal within 72 hours, otherwise, I'll start
> > moving the code out of Flink to Bahir (+1 confirmations of the proposal
> are
> > appreciated)
> >
> >
> > Regards,
> > Robert
> >
> >
> >
> > [1]
> >
> http://mail-archives.apache.org/mod_mbox/flink-dev/201608.mbox/%3CCAGr9p8CAN8KQTM6%2B3%2B%3DNv8M3ggYEE9gSqdKaKLQiWsWsKzZ21Q%40mail.gmail.com%3E
> > [2] https://github.com/apache/flink/pull/2314
> > [3] https://github.com/apache/flink/pull/2031
> > [4] https://github.com/apache/flink/pull/2332
> > [5]
> >
> https://github.com/apache/flink/tree/master/flink-streaming-connectors/flink-connector-redis
> > [6]
> >
> https://github.com/apache/flink/tree/master/flink-streaming-connectors/flink-connector-flume
>
>

Re: Reducing the JIRA PR message verbosity

[Maximilian Michels]

Actually that is a good suggestion. I know from other Apache projects
that they only mirror the initial description of the pull request but
not the discussion. I agree with you that it's very hard to have
meaningful discussion in JIRA if it is overlapped with GitHub
comments.

Cheers,
Max

On Wed, Aug 17, 2016 at 10:52 PM, Neelesh Salian  wrote:
> Hello,
>
> I have noticed there is a high verbosity coming from the PR into the JIRA
> which makes it hard to focus on the message or the content of the PR
> through the JIRA itself.
> Has there been a discussion over minimizing the details displayed back on
> the PR's JIRA?
>
> Maybe just add the PR link and the description of the changes rather than
> the entire template?
>
> Any thoughts?
>
> Regards,
>
> --
> Neelesh Srinivas Salian

Re: Reducing the JIRA PR message verbosity

[Stephan Ewen]

@max - can you contact infra about that?

On Thu, Aug 18, 2016 at 10:25 AM, Maximilian Michels  wrote:

> Actually that is a good suggestion. I know from other Apache projects
> that they only mirror the initial description of the pull request but
> not the discussion. I agree with you that it's very hard to have
> meaningful discussion in JIRA if it is overlapped with GitHub
> comments.
>
> Cheers,
> Max
>
> On Wed, Aug 17, 2016 at 10:52 PM, Neelesh Salian 
> wrote:
> > Hello,
> >
> > I have noticed there is a high verbosity coming from the PR into the JIRA
> > which makes it hard to focus on the message or the content of the PR
> > through the JIRA itself.
> > Has there been a discussion over minimizing the details displayed back on
> > the PR's JIRA?
> >
> > Maybe just add the PR link and the description of the changes rather than
> > the entire template?
> >
> > Any thoughts?
> >
> > Regards,
> >
> > --
> > Neelesh Srinivas Salian
>

Re: [DISCUSS] API breaking change in DataStream Windows

[Robert Metzger]

I forgot to start the VOTE after 24 hours.
However, I checked again Apache's voting rules [1] and code changes require
consensus. So a -1 vote by a PMC member effectively is a veto.
I don't have the impression from the discussion that we have a clear
majority for one approach, so a VOTE thread would quickly turn into another
big discussion thread.

Let me summarize the options at hand:
#1 Break the DataStream API, by changing the return type of the apply()
method
#2 Keep the API and ask users to manually cast the return type of apply()
to change the parallelism
#3 Introduce a deprecated with() method that returns the correct type


I prefer #2, because I think the should not break the API. We had some
other API breaking discussions and we always found a solution.


[1] http://www.apache.org/foundation/voting.html#votes-on-code-modification

On Mon, Aug 15, 2016 at 12:21 PM, Robert Metzger 
wrote:

> I would like have a decision on this so that we can merge the pull request.
>
> If we can not come up with a solution everybody agrees, and nobody rejects
> the VOTE, I'll start a VOTE thread in 24 hours.
>
>
> On Tue, Aug 9, 2016 at 3:57 PM, Till Rohrmann 
> wrote:
>
>> That is a tough call but I'm personally leaning slightly towards not
>> breaking the API and adding a note for the casting workaround.
>>
>> My main concern is where do we set the limit for future API breaking
>> issues? How critical does an issue has to be to be allowed to break the
>> API? Currently, we have 10 API breaking issues for Flink 2.0 [1]. Why not
>> including one of them as well?
>>
>> I think that backwards compatibility (source as well as binary) is really
>> important for many users and we have the duty to live up to our promises.
>> Imho, if things are API breaking, then we should indeed bump the major
>> version, as Greg suggested.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-3957
>>
>> Cheers,
>> Till
>>
>> On Tue, Aug 9, 2016 at 2:20 PM, Greg Hogan  wrote:
>>
>> > I agree that expecting users to cast is undesirable. Upon changing the
>> API,
>> > why would we not mark the next release as 2.0?
>> >
>> > The same issue arose with Gabor's addition of hash-combine in the Scala
>> > DataSet API where DataSet was returned rather than a specialized
>> Operator.
>> > The solution was to add an overloaded method.
>> >
>> > https://github.com/apache/flink/pull/1517/files#diff-
>> > b1fea8d5283d978e9ccbc202dad36b5fR322
>> >
>> > On Mon, Aug 8, 2016 at 10:11 AM, Stephan Ewen  wrote:
>> >
>> > > Hi all!
>> > >
>> > > We have a problem in the *DataStream API* around Windows for *CoGroup*
>> > and
>> > > *Join*.
>> > > These operations currently do not allow to set a parallelism, which
>> is a
>> > > pretty heavy problem.
>> > >
>> > > To fix it properly, we need to change the return types of the
>> coGroup()
>> > and
>> > > join() operations, which *breaks the binary compatibility* - it*
>> retains
>> > > source compatibility*, though.
>> > >
>> > > The pull request with the change is:
>> > > https://github.com/apache/flink/pull/2305
>> > >
>> > > There are very clumsy ways to work around this (custom casts in the
>> user
>> > > code or making the join() / coGroup() behave differently than the
>> other
>> > > operators) which we did not really think of as viable, because they
>> would
>> > > need to be changed again in the future once we pull the API straight
>> > > (breaking even source compatibility then).
>> > >
>> > > *I would suggest to actually break the API* at that point (binary, not
>> > > source) for *Flink 1.2* and add a big note in the release docs. An
>> > > uncomfortable step, but the alternatives are quite bad, too.
>> > >
>> > > Have a look at what has been suggested in the pull request discussion
>> and
>> > > please let us know what you think about that so we can proceed.
>> > >
>> > > Greetings,
>> > > Stephan
>> > >
>> >
>>
>
>

Re: [DISCUSS] API breaking change in DataStream Windows

[Stephan Ewen]

I think no vote is needed. The pull request is actually converging towards
a decision.

On Thu, Aug 18, 2016 at 12:07 PM, Robert Metzger 
wrote:

> I forgot to start the VOTE after 24 hours.
> However, I checked again Apache's voting rules [1] and code changes require
> consensus. So a -1 vote by a PMC member effectively is a veto.
> I don't have the impression from the discussion that we have a clear
> majority for one approach, so a VOTE thread would quickly turn into another
> big discussion thread.
>
> Let me summarize the options at hand:
> #1 Break the DataStream API, by changing the return type of the apply()
> method
> #2 Keep the API and ask users to manually cast the return type of apply()
> to change the parallelism
> #3 Introduce a deprecated with() method that returns the correct type
>
>
> I prefer #2, because I think the should not break the API. We had some
> other API breaking discussions and we always found a solution.
>
>
> [1] http://www.apache.org/foundation/voting.html#votes-
> on-code-modification
>
> On Mon, Aug 15, 2016 at 12:21 PM, Robert Metzger 
> wrote:
>
> > I would like have a decision on this so that we can merge the pull
> request.
> >
> > If we can not come up with a solution everybody agrees, and nobody
> rejects
> > the VOTE, I'll start a VOTE thread in 24 hours.
> >
> >
> > On Tue, Aug 9, 2016 at 3:57 PM, Till Rohrmann 
> > wrote:
> >
> >> That is a tough call but I'm personally leaning slightly towards not
> >> breaking the API and adding a note for the casting workaround.
> >>
> >> My main concern is where do we set the limit for future API breaking
> >> issues? How critical does an issue has to be to be allowed to break the
> >> API? Currently, we have 10 API breaking issues for Flink 2.0 [1]. Why
> not
> >> including one of them as well?
> >>
> >> I think that backwards compatibility (source as well as binary) is
> really
> >> important for many users and we have the duty to live up to our
> promises.
> >> Imho, if things are API breaking, then we should indeed bump the major
> >> version, as Greg suggested.
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-3957
> >>
> >> Cheers,
> >> Till
> >>
> >> On Tue, Aug 9, 2016 at 2:20 PM, Greg Hogan  wrote:
> >>
> >> > I agree that expecting users to cast is undesirable. Upon changing the
> >> API,
> >> > why would we not mark the next release as 2.0?
> >> >
> >> > The same issue arose with Gabor's addition of hash-combine in the
> Scala
> >> > DataSet API where DataSet was returned rather than a specialized
> >> Operator.
> >> > The solution was to add an overloaded method.
> >> >
> >> > https://github.com/apache/flink/pull/1517/files#diff-
> >> > b1fea8d5283d978e9ccbc202dad36b5fR322
> >> >
> >> > On Mon, Aug 8, 2016 at 10:11 AM, Stephan Ewen 
> wrote:
> >> >
> >> > > Hi all!
> >> > >
> >> > > We have a problem in the *DataStream API* around Windows for
> *CoGroup*
> >> > and
> >> > > *Join*.
> >> > > These operations currently do not allow to set a parallelism, which
> >> is a
> >> > > pretty heavy problem.
> >> > >
> >> > > To fix it properly, we need to change the return types of the
> >> coGroup()
> >> > and
> >> > > join() operations, which *breaks the binary compatibility* - it*
> >> retains
> >> > > source compatibility*, though.
> >> > >
> >> > > The pull request with the change is:
> >> > > https://github.com/apache/flink/pull/2305
> >> > >
> >> > > There are very clumsy ways to work around this (custom casts in the
> >> user
> >> > > code or making the join() / coGroup() behave differently than the
> >> other
> >> > > operators) which we did not really think of as viable, because they
> >> would
> >> > > need to be changed again in the future once we pull the API straight
> >> > > (breaking even source compatibility then).
> >> > >
> >> > > *I would suggest to actually break the API* at that point (binary,
> not
> >> > > source) for *Flink 1.2* and add a big note in the release docs. An
> >> > > uncomfortable step, but the alternatives are quite bad, too.
> >> > >
> >> > > Have a look at what has been suggested in the pull request
> discussion
> >> and
> >> > > please let us know what you think about that so we can proceed.
> >> > >
> >> > > Greetings,
> >> > > Stephan
> >> > >
> >> >
> >>
> >
> >
>

Re: [DISCUSS] Move Redis and Flume connectors to Apache Bahir and redirect contributions there

[Till Rohrmann]

+1

On Thu, Aug 18, 2016 at 10:21 AM, Aljoscha Krettek 
wrote:

> +1
>
> On Thu, 18 Aug 2016 at 03:54 Jark Wu  wrote:
>
> > Good start point. +1
> >
> >
> >
> > - Jark Wu
> >
> > > 在 2016年8月17日，下午11:24，Robert Metzger  写道：
> > >
> > > Hi,
> > >
> > > following the discussions in [1], the community decided to offer
> > > contributors to submit streaming connectors the Flink community can not
> > > maintain to Apache Bahir (there is still the option that we are moving
> > > popular connectors to the main Flink codebase)
> > >
> > > The ActiveMQ connector pull request [2], the Sink for RethinkDB [3],
> and
> > > the streaming sink for HBase [4] are in my opinion examples of
> connectors
> > > we should first give to Bahir.
> > >
> > > Also, I'm proposing to move the Redis sink [5] and the Flume sink [6]
> > from
> > > Apache Flink to Bahir. I can not recall any users asking about those
> > Sinks
> > > on the mailing list.
> > >
> > > Please object to this proposal within 72 hours, otherwise, I'll start
> > > moving the code out of Flink to Bahir (+1 confirmations of the proposal
> > are
> > > appreciated)
> > >
> > >
> > > Regards,
> > > Robert
> > >
> > >
> > >
> > > [1]
> > >
> > http://mail-archives.apache.org/mod_mbox/flink-dev/201608.
> mbox/%3CCAGr9p8CAN8KQTM6%2B3%2B%3DNv8M3ggYEE9gSqdKaKLQiWsWsKzZ
> 21Q%40mail.gmail.com%3E
> > > [2] https://github.com/apache/flink/pull/2314
> > > [3] https://github.com/apache/flink/pull/2031
> > > [4] https://github.com/apache/flink/pull/2332
> > > [5]
> > >
> > https://github.com/apache/flink/tree/master/flink-
> streaming-connectors/flink-connector-redis
> > > [6]
> > >
> > https://github.com/apache/flink/tree/master/flink-
> streaming-connectors/flink-connector-flume
> >
> >
>

Re: [DISCUSS] Move Redis and Flume connectors to Apache Bahir and redirect contributions there

[戴資力]

+1

On Thu, Aug 18, 2016 at 6:24 PM, Till Rohrmann  wrote:

> +1
>
> On Thu, Aug 18, 2016 at 10:21 AM, Aljoscha Krettek 
> wrote:
>
> > +1
> >
> > On Thu, 18 Aug 2016 at 03:54 Jark Wu  wrote:
> >
> > > Good start point. +1
> > >
> > >
> > >
> > > - Jark Wu
> > >
> > > > 在 2016年8月17日，下午11:24，Robert Metzger  写道：
> > > >
> > > > Hi,
> > > >
> > > > following the discussions in [1], the community decided to offer
> > > > contributors to submit streaming connectors the Flink community can
> not
> > > > maintain to Apache Bahir (there is still the option that we are
> moving
> > > > popular connectors to the main Flink codebase)
> > > >
> > > > The ActiveMQ connector pull request [2], the Sink for RethinkDB [3],
> > and
> > > > the streaming sink for HBase [4] are in my opinion examples of
> > connectors
> > > > we should first give to Bahir.
> > > >
> > > > Also, I'm proposing to move the Redis sink [5] and the Flume sink [6]
> > > from
> > > > Apache Flink to Bahir. I can not recall any users asking about those
> > > Sinks
> > > > on the mailing list.
> > > >
> > > > Please object to this proposal within 72 hours, otherwise, I'll start
> > > > moving the code out of Flink to Bahir (+1 confirmations of the
> proposal
> > > are
> > > > appreciated)
> > > >
> > > >
> > > > Regards,
> > > > Robert
> > > >
> > > >
> > > >
> > > > [1]
> > > >
> > > http://mail-archives.apache.org/mod_mbox/flink-dev/201608.
> > mbox/%3CCAGr9p8CAN8KQTM6%2B3%2B%3DNv8M3ggYEE9gSqdKaKLQiWsWsKzZ
> > 21Q%40mail.gmail.com%3E
> > > > [2] https://github.com/apache/flink/pull/2314
> > > > [3] https://github.com/apache/flink/pull/2031
> > > > [4] https://github.com/apache/flink/pull/2332
> > > > [5]
> > > >
> > > https://github.com/apache/flink/tree/master/flink-
> > streaming-connectors/flink-connector-redis
> > > > [6]
> > > >
> > > https://github.com/apache/flink/tree/master/flink-
> > streaming-connectors/flink-connector-flume
> > >
> > >
> >
>



-- 
Tzu-Li (Gordon) Tai

[jira] [Created] (FLINK-4421) Make clocks and time measurements monotonous

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-4421:
---

 Summary: Make clocks and time measurements monotonous
 Key: FLINK-4421
 URL: https://issues.apache.org/jira/browse/FLINK-4421
 Project: Flink
  Issue Type: Improvement
  Components: Core
Reporter: Stephan Ewen
Priority: Minor


Currently, many places use {{System.currentTimeMillis()}} to acquire timestamps 
or measure time intervals.

Since this relies on the system clock, and the system clock is not necessarily 
monotonous (in the presence of clock updates), this can lead to negative 
duration and decreasing timestamps where increasing timestamps are expected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4422) Convert all time interval measurements to System.nanoTime()

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-4422:
---

 Summary: Convert all time interval measurements to 
System.nanoTime()
 Key: FLINK-4422
 URL: https://issues.apache.org/jira/browse/FLINK-4422
 Project: Flink
  Issue Type: Sub-task
Reporter: Stephan Ewen
Priority: Minor


In contrast to {{System.currentTimeMillis()}}, {{System.nanoTime()}} is 
monotonous. To measure delays and time intervals, {{System.nanoTime()}} is 
hence reliable, while {{System.currentTimeMillis()}} is not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4423) Introduce a Clock utility for monotonous system timestamps

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-4423:
---

 Summary: Introduce a Clock utility for monotonous system timestamps
 Key: FLINK-4423
 URL: https://issues.apache.org/jira/browse/FLINK-4423
 Project: Flink
  Issue Type: Sub-task
Reporter: Stephan Ewen


I suggest to introduce a {{Clock}} class that provides a 
{{currentTimeMillis()}} function that calls {{System.currentTimeMillis()}} but 
also remembers the max returned timestamp so far. That way it would never 
return decreasing timestamps.

In the presence of clock backwards adjustments, the appearance would be that 
time stands still for a while, until the clock has caught up with the previous 
timestamp.

Since we don't rely on this for measuring timeouts, but only for logging / 
visualization / etc (see [FLINK-4422])  it should not mess up any distributed 
system behavior.

We would use this in places like the {{ExecutionGraph}}, where we record 
timestamps for state transitions. That way, the utilities that derive charts 
and times from the status timestamps would not be thrown off if timestamps were 
decreasing when expected increasing.

The same holds for ingestion time timestamps and for processing time triggers.

NOTE: I would like some other opinions on that - it is a somewhat delicate 
matter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4424) Make network environment start-up/shutdown independent of JobManager association

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-4424:


 Summary: Make network environment start-up/shutdown independent of 
JobManager association
 Key: FLINK-4424
 URL: https://issues.apache.org/jira/browse/FLINK-4424
 Project: Flink
  Issue Type: Sub-task
  Components: Network, TaskManager
Reporter: Till Rohrmann


Currently, the {{TaskManager}} starts the netty network server only after it 
has registered with a {{JobManager}}. Upon loss of connection to the 
{{JobManager}} the {{NetworkEnvironment}} is closed.

The start-up and shutdown of the network server should be independent of the 
{{JobManager}} connection, especially if we assume that a TM can be associated 
with multiple JobManagers in the future (FLIP-6).

Starting the network server only once when the {{TaskManager}} is started has 
the advantage that we don't have to preconfigure the {{TaskManager's}} data 
port. Furthermore we don't risk to get stuck when disassociating from a 
{{JobManager}} because the start-up and shutdown of a {{NetworkEnvironment}} 
can cause problems (because it has to reserve/free resources).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4425) "Out Of Memory" during savepoint deserialization

[Sergii Koshel (JIRA)]

Sergii Koshel created FLINK-4425:


 Summary: "Out Of Memory" during savepoint deserialization
 Key: FLINK-4425
 URL: https://issues.apache.org/jira/browse/FLINK-4425
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sergii Koshel


I've created savepoint and trying to start job using it (via -s param) and 
getting exception like below:

{code:title=Exception|borderStyle=solid}
java.lang.OutOfMemoryError: Java heap space
at 
org.apache.flink.runtime.checkpoint.savepoint.SavepointV1Serializer.deserialize(SavepointV1Serializer.java:167)
at 
org.apache.flink.runtime.checkpoint.savepoint.SavepointV1Serializer.deserialize(SavepointV1Serializer.java:42)
at 
org.apache.flink.runtime.checkpoint.savepoint.FsSavepointStore.loadSavepoint(FsSavepointStore.java:133)
at 
org.apache.flink.runtime.checkpoint.savepoint.SavepointCoordinator.restoreSavepoint(SavepointCoordinator.java:201)
at 
org.apache.flink.runtime.executiongraph.ExecutionGraph.restoreSavepoint(ExecutionGraph.java:983)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1302)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1291)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}

jobmanager.heap.mb: 1280
taskmanager.heap.mb: 1024

java 1.8

savepoint + checkpoint size < 1 Mb in total



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

[Aljoscha Krettek]

Hi,
I personally would like it a lot if the SQL queries for batch and stream
programs looked the same. With the decision to move the Table API on top of
Calcite and also use the Calcite SQL parser Flink is somewhat tied to
Calcite so I don't know whether we can add our own window constructs and
teach the parser to properly read them.

Maybe Fabian and Timo have more insights here since they worked on the move
to Calcite.

Cheers,
Aljoscha

+Timo looping him in directly

On Tue, 16 Aug 2016 at 09:29 Jark Wu  wrote:

> Hi,
>
> Currently, Flink use Calcite for SQL parsing. So we use the StreamSQL
> grammer proposed by Calcite[1] which we have to use the `STREAM` keyword in
> SQL. For example, `SELECT *
> FROM Orders` is a regular standard SQL and will be translated to a batch
> job. If you want to statement a stream job, you have add the `STREAM`
> keyword, `SELECT STREAM *
> FROM Orders`.
>
> I'm thinking of why do we distinguish between StreamSQL and BatchSQL
> grammer? We already have separate high-level API for batch(DataSet) and
> stream(DataStream). And we have a unified Table API for batch and stream
> (that's great!). Why do we have to separate them again in SQL?
>
> I hope we can manipulate stream data like a table. Such as `SELECT *
> FROM Orders`, if Orders is a table (or run in batch execution env), then
> it's a batch job. If Orders is a stream (or run in stream execution env),
> then it's a stream job. The grammer of StreamSQL and BatchSQL is totally
> the same. And that is what we did in Blink SQL.
>
> The benefits if we unify the grammar :
>
> 1. Easy to use StreamSQL for anyone who knows regular SQL. There is no
> difference between StreamSQL and regular SQL.
> 2. Not blocked by Calcite. Currently, Calcite StreamSQL is not fullly
> supported. Not support stream-to-stream JOIN, not support window aggregate,
> not support aggregate without window, etc. We may need to wait for calcite
> to support them before we start work. As they are supported by regular SQL
> besides window. We can implement window via user-defined-function. So if we
> can use regular SQL instead of StreamSQL, we can start to work it right now
> and not wait for Calcite.
> 3. Blink SQL can merge back to community to accelerate Flink SQL evolving.
> Blink SQL has done most work of it. We implement UDF/UDTF/UDAF, aggregate
> with/without window, and stream-to-stream JOIN, and so on.
> 4. Window also can work in batch job.
>
> Just my thoughts :)
>
> What do you think about this ?
>
> [1] https://calcite.apache.org/docs/stream.html
>
> - Jark Wu
>
>

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

[Timo Walther]

Hi Jark,

sorry that I didn't wrote back earlier. I wanted to talk to Fabian first 
about this. In general, according to Calcite's plans, even SQL queries 
containing the "STREAM" keyword are regular standard SQL. In theory we 
could omit the "STREAM" keyword as long as it is guaranteed that the 
generated logical plans look the same. So I'm not against having the 
same grammar for both batch and streaming queries. However, I think we 
should contribute code to Calcite if the logical representation is not 
there already for operators we need. We need to research how far the 
Calcite development is. We can implement windows via 
user-defined-function as it also done in Calcite streaming design document.


It would be very interesting for the upcoming design phase if you could 
show us how you implemented your Blink SQL. For instance, how do you 
define windows there?


Regards,
Timo


Am 18/08/16 um 16:34 schrieb Aljoscha Krettek:

Hi,
I personally would like it a lot if the SQL queries for batch and 
stream programs looked the same. With the decision to move the Table 
API on top of Calcite and also use the Calcite SQL parser Flink is 
somewhat tied to Calcite so I don't know whether we can add our own 
window constructs and teach the parser to properly read them.


Maybe Fabian and Timo have more insights here since they worked on the 
move to Calcite.


Cheers,
Aljoscha

+Timo looping him in directly

On Tue, 16 Aug 2016 at 09:29 Jark Wu > wrote:


Hi,

Currently, Flink use Calcite for SQL parsing. So we use the
StreamSQL grammer proposed by Calcite[1] which we have to use the
`STREAM` keyword in SQL. For example, `SELECT *
FROM Orders` is a regular standard SQL and will be translated to a
batch job. If you want to statement a stream job, you have add the
`STREAM` keyword, `SELECT STREAM *
FROM Orders`.

I'm thinking of why do we distinguish between StreamSQL and
BatchSQL grammer? We already have separate high-level API for
batch(DataSet) and stream(DataStream). And we have a unified Table
API for batch and stream (that's great!). Why do we have to
separate them again in SQL?

I hope we can manipulate stream data like a table. Such as `SELECT *
FROM Orders`, if Orders is a table (or run in batch execution
env), then it's a batch job. If Orders is a stream (or run in
stream execution env), then it's a stream job. The grammer of
StreamSQL and BatchSQL is totally the same. And that is what we
did in Blink SQL.

The benefits if we unify the grammar :

1. Easy to use StreamSQL for anyone who knows regular SQL. There
is no difference between StreamSQL and regular SQL.
2. Not blocked by Calcite. Currently, Calcite StreamSQL is not
fullly supported. Not support stream-to-stream JOIN, not support
window aggregate, not support aggregate without window, etc. We
may need to wait for calcite to support them before we start work.
As they are supported by regular SQL besides window. We can
implement window via user-defined-function. So if we can use
regular SQL instead of StreamSQL, we can start to work it right
now and not wait for Calcite.
3. Blink SQL can merge back to community to accelerate Flink SQL
evolving. Blink SQL has done most work of it. We implement
UDF/UDTF/UDAF, aggregate with/without window, and stream-to-stream
JOIN, and so on.
4. Window also can work in batch job.

Just my thoughts :)

What do you think about this ?

[1] https://calcite.apache.org/docs/stream.html

- Jark Wu




--
Freundliche Grüße / Kind Regards

Timo Walther

Follow me: @twalthr
https://www.linkedin.com/in/twalthr

Re: [DISCUSS] FLIP-3 - Organization of Documentation

[Ufuk Celebi]

Initial PR for the layout: https://github.com/apache/flink/pull/2387

On Tue, Aug 2, 2016 at 5:18 PM, Aljoscha Krettek  wrote:
> +1
>
> On Tue, 2 Aug 2016 at 03:15 Till Rohrmann  wrote:
>
>> +1 :-)
>>
>> On Tue, Aug 2, 2016 at 6:09 PM, Stephan Ewen  wrote:
>>
>> > +1, thanks :-)
>> >
>> > On Tue, Aug 2, 2016 at 11:39 AM, Ufuk Celebi  wrote:
>> >
>> > > If there are no objections, I would like to work on this in the next
>> > > days. I would like to only do the restructuring and don't add any new
>> > > content (e.g. we would have a few empty pages in the beginning).
>> > >
>> > > On Wed, Jul 20, 2016 at 9:57 PM, Stephan Ewen 
>> wrote:
>> > > > I added to the "Application Development" Docs the Section "Types,
>> > > > TypeInformation, Serialization".
>> > > > I think that is an important enough aspect to warrant separate docs.
>> > > >
>> > > > On Mon, Jul 18, 2016 at 3:36 PM, Till Rohrmann > >
>> > > wrote:
>> > > >
>> > > >> +1 for the FLIP and making streaming the common case. Very good
>> > proposal
>> > > >> :-)
>> > > >>
>> > > >> On Mon, Jul 18, 2016 at 11:48 AM, Aljoscha Krettek <
>> > aljos...@apache.org
>> > > >
>> > > >> wrote:
>> > > >>
>> > > >> > +1 I like it a lot!
>> > > >> >
>> > > >> > On Fri, 15 Jul 2016 at 18:43 Stephan Ewen 
>> wrote:
>> > > >> >
>> > > >> > > My take would be to take streaming as the common case and make
>> > > special
>> > > >> > > sections for batch.
>> > > >> > >
>> > > >> > > We can still have a few streaming-only sections (end to end
>> > exactly
>> > > >> once)
>> > > >> > > and a few batch-only sections (optimizer).
>> > > >> > >
>> > > >> > > On Fri, Jul 15, 2016 at 6:03 PM, Ufuk Celebi 
>> > > wrote:
>> > > >> > >
>> > > >> > > > I very much like this proposal. This is long overdue. Our
>> > > >> > > > documentation never "broke up" with the old batch focus.
>> That's
>> > > where
>> > > >> > > > the current structure comes from and why people often don't
>> find
>> > > what
>> > > >> > > > they are looking for. We were trying to treat streaming and
>> > batch
>> > > as
>> > > >> > > > equals. We never were "brave" enough to move streaming-only
>> > > concepts
>> > > >> > > > to the top-level. I really like that you are proposing this
>> now
>> > > (for
>> > > >> > > > example for Event time, State Backends etc.). I would love to
>> > have
>> > > >> > > > this go hand in hand with the 1.2 release.
>> > > >> > > >
>> > > >> > > > What is your opinion about pages affecting both streaming and
>> > > batch
>> > > >> > > > like "Connectors" or "Failure model"? We could have the
>> landing
>> > > page
>> > > >> > > > cover the general material (e.g. restart strategies) and then
>> > have
>> > > >> > > > sub-pages for streaming- and batch-specific stuff. Or we treat
>> > > >> > > > streaming as the common case and have a sub-section for batch.
>> > We
>> > > >> > > > probably have to decide this case-by-case, but to me it feels
>> > like
>> > > >> > > > this was the main problem with the old documentation structure
>> > > >> > > > (content is a different story of course ;)).
>> > > >> > > >
>> > > >> > > > On Fri, Jul 15, 2016 at 4:09 PM, Stephan Ewen <
>> se...@apache.org
>> > >
>> > > >> > wrote:
>> > > >> > > > > Hi all!
>> > > >> > > > >
>> > > >> > > > > I posted another FLIP - this time about a suggestion to make
>> > the
>> > > >> > > > > documentation more accessible.
>> > > >> > > > >
>> > > >> > > > > FLIP-3 - Organization of Documentation
>> > > >> > > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-3+-+Organization+of+Documentation
>> > > >> > > > >
>> > > >> > > > > The issue of accessibility of information came up repeatedly
>> > > from
>> > > >> > > users I
>> > > >> > > > > talked to, so this is a suggestion how to improve this.
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > > Greetings,
>> > > >> > > > > Stephan
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>>

[FLINK-305] Code test coverage - how FLINK using it?

[Pavel Fadeev]

Dear team,

I`m just looking around into the project - complete novice at Flink :)
Sorry if Qs below already have answers!

At first glance I`ve discovered that code coverage feature has been
introduced with FLINK-305 
and then removed for some reason on March 2015.

Are you aware if it is not required anymore? Worried a bit about this after
local coverage run for flink-code as well. Do we have some integration like
codecov  here and do you feel if it is required?

Also, do you know if there are some statistics (or team knowledge) for
regression bugs from uncovered code?

[jira] [Created] (FLINK-4426) Unable to create proxy to the ResourceManager

[Harpreet Sawhney (JIRA)]

Harpreet Sawhney created FLINK-4426:
---

 Summary: Unable to create proxy to the ResourceManager
 Key: FLINK-4426
 URL: https://issues.apache.org/jira/browse/FLINK-4426
 Project: Flink
  Issue Type: Bug
  Components: ResourceManager
Affects Versions: 1.0.3
 Environment: Flink 1.0.3 built with MapR  (2.7.0-mapr-1602)
Reporter: Harpreet Sawhney


We have a Mapr cluster on which I am trying to run a single flink job (from 
examples) on YARN

Running the example (./bin/flink run -m yarn-cluster -yn 4  
./examples/batch/WordCount.jar) fails with an "Unable to create proxy to the 
ResourceManager null" error:

More detailed logs from the flink run below (server addresses removed):

=
2016-08-18 23:02:32,249 DEBUG 
org.apache.hadoop.metrics2.lib.MutableMetricsFactory  - field 
org.apache.hadoop.metrics2.lib.MutableRate 
org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with 
annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of 
successful kerberos logins and latency (milliseconds)], valueName=Time, about=, 
type=DEFAULT, always=false, sampleName=Ops)
2016-08-18 23:02:32,261 DEBUG 
org.apache.hadoop.metrics2.lib.MutableMetricsFactory  - field 
org.apache.hadoop.metrics2.lib.MutableRate 
org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with 
annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of failed 
kerberos logins and latency (milliseconds)], valueName=Time, about=, 
type=DEFAULT, always=false, sampleName=Ops)
2016-08-18 23:02:32,261 DEBUG 
org.apache.hadoop.metrics2.lib.MutableMetricsFactory  - field 
org.apache.hadoop.metrics2.lib.MutableRate 
org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with 
annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[GetGroups], 
valueName=Time, about=, type=DEFAULT, always=false, sampleName=Ops)
2016-08-18 23:02:32,263 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl 
- UgiMetrics, User and group related metrics
2016-08-18 23:02:33,777 DEBUG com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils   
- init
2016-08-18 23:02:33,793 DEBUG com.mapr.baseutils.JVMProperties  
- Setting JVM property zookeeper.saslprovider to 
com.mapr.security.simplesasl.SimpleSaslProvider
2016-08-18 23:02:33,794 DEBUG com.mapr.baseutils.JVMProperties  
- Setting JVM property zookeeper.sasl.clientconfig to Client_simple
2016-08-18 23:02:33,794 DEBUG com.mapr.baseutils.JVMProperties  
- Setting JVM property java.security.auth.login.config to 
/opt/mapr/conf/mapr.login.conf
2016-08-18 23:02:33,797 DEBUG org.apache.hadoop.conf.Configuration  
- Loaded org.apache.hadoop.conf.CoreDefaultProperties
2016-08-18 23:02:33,805 DEBUG org.apache.hadoop.security.UserGroupInformation   
- HADOOP_SECURITY_AUTHENTICATION is set to: SIMPLE
2016-08-18 23:02:33,805 DEBUG org.apache.hadoop.security.UserGroupInformation   
- Login configuration entry is hadoop_simple
2016-08-18 23:02:33,806 DEBUG org.apache.hadoop.security.UserGroupInformation   
- authenticationMethod from JAAS configuration:SIMPLE
2016-08-18 23:02:33,867 DEBUG org.apache.hadoop.conf.Configuration  
- Loaded org.apache.hadoop.conf.CoreDefaultProperties
2016-08-18 23:02:33,875 DEBUG org.apache.hadoop.security.Groups 
-  Creating new Groups object
2016-08-18 23:02:33,878 DEBUG org.apache.hadoop.util.PerformanceAdvisory
- Falling back to shell based
2016-08-18 23:02:33,879 DEBUG 
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback  - Group 
mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2016-08-18 23:02:33,934 DEBUG org.apache.hadoop.conf.Configuration  
- Loaded org.apache.hadoop.conf.CoreDefaultProperties
2016-08-18 23:02:34,002 DEBUG org.apache.hadoop.conf.Configuration  
- Loaded org.apache.hadoop.yarn.conf.YarnDefaultProperties
2016-08-18 23:02:34,021 DEBUG org.apache.hadoop.util.Shell  
- setsid exited with exit code 0
2016-08-18 23:02:34,047 DEBUG org.apache.hadoop.security.Groups 
- Group mapping 
impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; 
cacheTimeout=30; warningDeltaMs=5000
2016-08-18 23:02:34,058 DEBUG 
org.apache.hadoop.security.login.HadoopLoginModule- Priority 
principal search list is [class javax.security.auth.kerberos.KerberosPrincipal]
2016-08-18 23:02:34,058 DEBUG 
org.apache.hadoop.security.login.HadoopLoginModule- Additional 
principal search list is [class com.sun.security.auth.UnixPri

[jira] [Created] (FLINK-4427) Add slot / container releasing logic to SlotManager (Standalone / Yarn / Mesos)

[Kurt Young (JIRA)]

Kurt Young created FLINK-4427:
-

 Summary: Add slot / container releasing logic to SlotManager 
(Standalone / Yarn / Mesos)
 Key: FLINK-4427
 URL: https://issues.apache.org/jira/browse/FLINK-4427
 Project: Flink
  Issue Type: Sub-task
Reporter: Kurt Young


Currently we only have allocation logic for SlotManager / ResourceManager, for 
some batch job, slots that already finished be released, and should trigger 
container release in different cluster modes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4428) Method map/flatMapWithState may need a eviction policy

[Renkai Ge (JIRA)]

Renkai Ge created FLINK-4428:


 Summary: Method map/flatMapWithState may need a eviction policy
 Key: FLINK-4428
 URL: https://issues.apache.org/jira/browse/FLINK-4428
 Project: Flink
  Issue Type: New Feature
  Components: DataStream API
Affects Versions: 1.1.2
Reporter: Renkai Ge


I want to count the number of unique visitors of a website every day.
 If the number changes, I want get the newest number in 1 second, and
 it should keep silence if the number doesn't change.I implemented this 
 by time window of 1 day,trigger of 1 second and flatMapWithState to 
 filter duplicated results. 
{code}
 //case class Visit(uuid: String, time: Long, platform: Int)
 
 //case class WindowUv(platform: Int, uv: Long, windowStart: Long, 
WindowEnd: Long)
 
 //  val consumer: FlinkKafkaConsumer08[Visit]
 val stream =
 env.addSource(consumer)
   .keyBy(_.platform)
   .timeWindow(Time.days(1))
   .trigger(ContinuousProcessingTimeTrigger.of(Time.seconds(1)))
   .applyWith((0, Set.empty[Int], 0l, 0l))(
 foldFunction = {
   case ((_, set, _, 0), visit) =>
 (visit.platform, set + visit.uuid.hashCode, 0, 0)
 },
 windowFunction = {
   case (key, window, results) =>
 results.map {
   case (platform, set, _, _) =>
 (platform, set, window.getStart, window.getEnd)
 }
 }
   )
   .mapWith {
 case (key, set, windowStart, windowEnd) =>
   WindowUv(key, set.size, windowStart, windowEnd)
   }
   .keyBy(uv => (uv.platform, uv.windowStart))
   .flatMapWithState[WindowUv, Int] {
   case ((key, num, begin, end), curr) =>
 curr match {
   case Some(numCurr) if numCurr == num =>
 (Seq.empty, Some(num))
   case _ =>
 (Seq(WindowUv(key, num, begin, end)), Some(num))
 }
 }
 stream.print()
 env.execute("Boom")
{code}

There is a problem that I used flatMapWithState,the state of one day will
be never updated and never used after the day passed, but it will stay
in the memory forever, there is no way to evict it. So I think the status
in map may need some eviction policy related with time or global conditions
rather than only with the last message of the key(It's hard to tell whether 
a message is the last when the last is coming).  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)