Why checkpointing is synchronized with data processing?

Hi, flink dev team: I'm reading flink's source code and it seems that the process of saving checkpointing data to external storage system will block data processing, and I wonder why this has to be designed like this? Isn't is a potential risk that may increase latency of processing? Is the follo

Re: [DISCUSSION] Timely function interface and timer params

Hi Gyula, if you look at the internal API you'll notice that it is pretty much like your second proposal. Just for reference, the interface is roughly this: public interface InternalTimerService { long currentProcessingTime(); long currentWatermark(); void registerProcessingTimeTimer(N names

RE: trouble with type casting

I agree with you, Fabian. I will add a small hint in exception and cast bigDecimal manually in test cases -Original Message- From: Fabian Hueske [mailto:fhue...@gmail.com] Sent: Friday, October 28, 2016 7:17 PM To: dev@flink.apache.org Subject: Re: trouble with type casting Hi Anton,

[jira] [Created] (FLINK-4963) Tabulate edge direction for directed VertexMetrics

Greg Hogan created FLINK-4963: - Summary: Tabulate edge direction for directed VertexMetrics Key: FLINK-4963 URL: https://issues.apache.org/jira/browse/FLINK-4963 Project: Flink Issue Type: Improv

Re: [DISCUSS] Make FieldAccessor logic consistent with remaining API

IMO, FieldAccessors are certainly valuable and should exist. I only want to make sure that there is a common syntax for field references. Maybe we can refactor the parsing code for field references in ExpressionKeys and expose it in static public methods that the FieldAccessors can call. Best, Fa

Re: [DISCUSS] Schedule and Scope for Flink 1.2

Hi Jark, FLINK-4469: I agree, UDTFs would be a great feature. I don't think we should make it a blocker but I will help to get it the master before the feature freeze. FLINK-4704: Regarding the renaming of the namespace, we have two positive votes in JIRA, but we should also start a discussion on

Re: [DISCUSS] Releasing Flink 1.1.4

Benefit of a backport, as I see it, is increased stability. The danger is potentially breaking some code that was casting FileSystems to subtypes like LocalFileSytem. I don’t know how common that would be in user code. > Am 28.10.2016 um 14:27 schrieb Ufuk Celebi : > > Thanks for all your feed

Re: trouble with type casting

Hi Anton, it seems that the Table API validation is more strict than Calcite's SQL validator (maybe because it is not aware of the actual implementation). In principle, it is correct to prevent the auto-casting from BigDecimal to double. I think it is fine to request an explicit cast from users (

Re: [DISCUSS] FLIP-11: Table API Stream Aggregations

Thanks for bringing up this point Stephan. I think it is a good decision to have a standard compliant SQL interface that SQL developers and tools can use. The SQL proposal [1] I posted a few days ago follows this approach. However, SQL was not designed with stream processing in mind and many impo

[jira] [Created] (FLINK-4962) Improve exception information about operand type inference failed in UDF SQL validation

Jark Wu created FLINK-4962: -- Summary: Improve exception information about operand type inference failed in UDF SQL validation Key: FLINK-4962 URL: https://issues.apache.org/jira/browse/FLINK-4962 Project: Fl

[DISCUSSION] Timely function interface and timer params

Hello, I was thinking about the methods provided by the timely functions and the timerservice and I am wondering if it makes sense to change them a little so they can cover a wider set of use case. Maybe I missed something completely obvious so please shoot me down in that case :) Currently the u

[jira] [Created] (FLINK-4961) SGD for Matrix Factorization

Gábor Hermann created FLINK-4961: Summary: SGD for Matrix Factorization Key: FLINK-4961 URL: https://issues.apache.org/jira/browse/FLINK-4961 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-4960) Enable testing of different re-scaling scenarios in the AbstractStreamOperatorTestHarness to

Kostas Kloudas created FLINK-4960: - Summary: Enable testing of different re-scaling scenarios in the AbstractStreamOperatorTestHarness to Key: FLINK-4960 URL: https://issues.apache.org/jira/browse/FLINK-4960

Re: [DISCUSS] Schedule and Scope for Flink 1.2

Hello, As Aljoshca said, the TriggerDSL depends on another PR, before it can be merged. In addition, this will change also the whole structure of how we test window/trigger related functionality, so some refactoring will be required. Kostas > On Oct 28, 2016, at 3:06 PM, Aljoscha Krettek wrote

Re: [HEADS UP] Introducing ConfigOptions

+1 for the ConfigOptions and the automatic configuration description generation once we've migrated all config keys. Cheers, Till On Fri, Oct 28, 2016 at 2:18 PM, Robert Metzger wrote: > You are right Stephan. The doc migration will be easier when we use the new > logic for all keys. > > On Fri

Re: [DISCUSS] Schedule and Scope for Flink 1.2

@Till: For the TriggerDSL we have to merge my open PR [1] for refactoring the WindowOperator/Trigger tests and then refactor the tests of the TriggerDSL to use the new test facilities introduced there. Plus then maybe some fixups. On Fri, 28 Oct 2016 at 11:17 Jark Wu wrote: > +1 to the schedule

[jira] [Created] (FLINK-4959) Write Documentation for Timely FlatMap Functions

Aljoscha Krettek created FLINK-4959: --- Summary: Write Documentation for Timely FlatMap Functions Key: FLINK-4959 URL: https://issues.apache.org/jira/browse/FLINK-4959 Project: Flink Issue Ty

[jira] [Created] (FLINK-4958) Send slot report to RM when registering

Till Rohrmann created FLINK-4958: Summary: Send slot report to RM when registering Key: FLINK-4958 URL: https://issues.apache.org/jira/browse/FLINK-4958 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Releasing Flink 1.1.4

Thanks for all your feedback. If there are no objections, I would like to stick to the mentioned issues in this thread and create RC1 as soon as they are all addressed. This will probably not be this week though, but it looks good for next week. DONE = - FLINK-4619: Answer client if savepoint

[jira] [Created] (FLINK-4957) Provide API for TimelyCoFlatMapFunction

Aljoscha Krettek created FLINK-4957: --- Summary: Provide API for TimelyCoFlatMapFunction Key: FLINK-4957 URL: https://issues.apache.org/jira/browse/FLINK-4957 Project: Flink Issue Type: Test

Re: [HEADS UP] Introducing ConfigOptions

You are right Stephan. The doc migration will be easier when we use the new logic for all keys. On Fri, Oct 28, 2016 at 4:27 AM, Jark Wu wrote: > +1 > > The automatic configuration page generating sounds really cool. > > - Jark Wu > > > 在 2016年10月28日，上午12:23，Stephan Ewen 写道： > > > > I think add

[jira] [Created] (FLINK-4956) Introduce convenient methods for testing checkpoint/restore in StreamTaskTestHarness

Stefan Richter created FLINK-4956: - Summary: Introduce convenient methods for testing checkpoint/restore in StreamTaskTestHarness Key: FLINK-4956 URL: https://issues.apache.org/jira/browse/FLINK-4956

[jira] [Created] (FLINK-4955) Add Translations Tests for KeyedStream.flatMap(TimelyFlatMapFunction)

Aljoscha Krettek created FLINK-4955: --- Summary: Add Translations Tests for KeyedStream.flatMap(TimelyFlatMapFunction) Key: FLINK-4955 URL: https://issues.apache.org/jira/browse/FLINK-4955 Project: Fl

[jira] [Created] (FLINK-4954) Remove stashing from AkkaRpcActor

Till Rohrmann created FLINK-4954: Summary: Remove stashing from AkkaRpcActor Key: FLINK-4954 URL: https://issues.apache.org/jira/browse/FLINK-4954 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

Hello, I find this feature really cool because it would allow people to tackle scenarios requiring a more advanced "join" on multiple streams as already stated above. Here you can find my few thou

trouble with type casting

I'm working on flink-4743 issue I tried write tests as Fabian suggested me https://github.com/apache/flink/pull/2686#discussion_r85218584 and figure out that there is no implicit conversion of bigDecimal to double in table api. "power" function accept only double as operands. all is okay when r

Re: [DISCUSS] Schedule and Scope for Flink 1.2

+1 to the schedule I hope to add 1) Table API: User defined table function (FLINK-4469 ) 2) Table API: Move Table API to org.apache.flink.table (FLINK-4704 ) The 1) has been discussed for a l

Re: [DISCUSS] FLIP-11: Table API Stream Aggregations

Hi , I agree that Table API should be a SQL-like API, that do not need to be strictly consistent with SQL. > - Not put the window definition into the groupBy clause. +1 Putting the window clause out of groupBy, will be easy to learn and easy to discover for users. > I like the idea of havi