Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

Thanks for the voting and comments. @Stephan - The '-XX:MaxDirectMemorySize' value should not include JVM Overhead. Thanks for correction. - 'taskmanager.memory.framework.heap' it heap memory reserved for task executor framework, which can not be allocated to task slots. I think users should be ab

FLIP-63: Rework table partition support

Hi everyone, We would like to start a discussion thread on "FLIP-63: Rework table partition support"(Design doc: [1]), where we describe how to partition support in flink and how to integrate to hive partition. This FLIP addresses: - Introduce whole story about partition support. - Introd

Re: [DISCUSS] Reducing build times

Will using more powerful for the project make it more difficult to ensure that contributor builds are still running in a reasonable time? As an example of this happening on Travis, contributors currently cannot run all e2e tests since they timeout, but on apache we have a larger timeout. On

[jira] [Created] (FLINK-13953) Facilitate enabling new Scheduler in MiniCluster Tests

Gary Yao created FLINK-13953: Summary: Facilitate enabling new Scheduler in MiniCluster Tests Key: FLINK-13953 URL: https://issues.apache.org/jira/browse/FLINK-13953 Project: Flink Issue Type: Su

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

If we later split the network memory into "shuffle" and "other network memory", I think it would make sense to split the option then. In that case "taskmanager.memory.network" would probably refer to the total network memory, which would most likely be what most users actually configure. My feelin

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

@all The FLIP document [1] has been updated. Thank you~ Xintong Song [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management On Tue, Sep 3, 2019 at 7:20 PM Zhu Zhu wrote: > Thanks Xintong for the explanation. > > For question #1, I think it

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi everyone, thanks a lot for starting this discussion Yijie. I think the Pulsar connector would be a very valuable addition since Pulsar becomes more and more popular and it would further expand Flink's interoperability. Also from a project perspective it makes sense for me to place the connector

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

@Stephan Not sure what do you mean by "just having this value". Are you suggesting that having "taskmanager.memory.network" refers to "shuffle" and "other network memory", or only "shuffle"? I guess what you mean is only "shuffle"? Because currently "taskmanager.network.memory" refers to shuffle b

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

@Zhijiang @Stephan I agree with @Xintong for the scope of the shuffle memory. but as @Zhijinag pointed out it is not easy to estimate real netty shuffle memory consumption due to the overhead. Everything that is pretty much O(1) comparing to the shuffle buffer size can be accommodated in the genera

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

Just to clarify Xintong, you suggest that Task off-heap memory represents direct and native memory. Since we don't know how the user will allocate the memory we will add this value to -XX:MaxDirectMemorySize so that the process won't fail if the user allocates only direct memory and no native memor

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

I'm quite worried that we may end up repeating history. There were already 2 attempts at contributing a pulsar connector, both of which failed because no committer was getting involved, despite the contributor opening a dedicated discussion thread about the contribution beforehand and getting

Re: [DISCUSS] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration

I guess that most things have already been said on the related discussion thread [1]. Hence I will start a vote about this FLIP. [1] https://lists.apache.org/thread.html/80bef7146f9696f35b1e50ff4acdd1cc3e87ae6f212d205aa7a72182@%3Cdev.flink.apache.org%3E Cheers, Till On Mon, Sep 2, 2019 at 10:56

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

@till > Just to clarify Xintong, you suggest that Task off-heap memory represents > direct and native memory. Since we don't know how the user will allocate > the memory we will add this value to -XX:MaxDirectMemorySize so that the > process won't fail if the user allocates only direct memory and

Re: [DISCUSS] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

I guess that most things have already been said on the related discussion thread [1]. Hence, I'm going to start the vote. [1] https://lists.apache.org/thread.html/107b15de6b8ac849610d99c4754715d2a8a2f32ddfe9f8da02af2ccc@%3Cdev.flink.apache.org%3E Cheers, Till On Tue, Sep 3, 2019 at 11:41 AM Till

Re: [DISCUSS] Releasing Flink 1.8.2

Hi, I’m just running the last tests on FLINK-13586 on Travis and them I’m merging. Best, Aljoscha > On 4. Sep 2019, at 07:37, Jark Wu wrote: > > Thanks for the work Jincheng! > > I have moved remaining major issues to 1.8.3 except FLINK-13586. > > Hi @Aljoscha Krettek

[VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration

Hi everyone, I would like to start the voting process for FLIP-61 [1], which is discussed and reached consensus in this thread [2]. Since the change is rather small I'd like to shorten the voting period to 48 hours. Hence, I'll try to close it September 6th, 11:00 am CET, unless there is an objec

[VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

Hi everyone, I would like to start the voting process for FLIP-62 [1], which is discussed and reached consensus in this thread [2]. Since the change is rather small I'd like to shorten the voting period to 48 hours. Hence, I'll try to close it September 6th, 11:00 am CET, unless there is an objec

[DISCUSS] Use static imports for test utilities or asserts.

Hi all, Is it necessary for us to replace all assert or utilities with static import ? Thank you very much.

Re: [VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration

+1 (non-binding) Thanks, Zhu Zhu Till Rohrmann 于2019年9月4日周三 下午5:05写道： > Hi everyone, > > I would like to start the voting process for FLIP-61 [1], which is > discussed and reached consensus in this thread [2]. > > Since the change is rather small I'd like to shorten the voting period to > 48 ho

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

+1 (non-binding) Thanks, Zhu Zhu Till Rohrmann 于2019年9月4日周三 下午5:06写道： > Hi everyone, > > I would like to start the voting process for FLIP-62 [1], which > is discussed and reached consensus in this thread [2]. > > Since the change is rather small I'd like to shorten the voting period to > 48 ho

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

+1 (non-binding) default 0 is really not user production friendly. Best, Jingsong Lee -- From:Zhu Zhu Send Time:2019年9月4日(星期三) 17:13 To:dev Subject:Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRes

Re: [DISCUSS] Flink client api enhancement for downstream project

Hi Kostas & Aljoscha, I notice that there is a JIRA(FLINK-13946) which could be included in this refactor thread. Since we agree on most of directions in big picture, is it reasonable that we create an umbrella issue for refactor client APIs and also linked subtasks? It would be a better way that

Re: [DISCUSS] Flink client api enhancement for downstream project

Great idea Tison! I will create the umbrella issue and post it here so that we are all on the same page! Cheers, Kostas On Wed, Sep 4, 2019 at 11:36 AM Zili Chen wrote: > > Hi Kostas & Aljoscha, > > I notice that there is a JIRA(FLINK-13946) which could be included > in this refactor thread. Si

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi David, Thank you for sharing your findings. It seems to me that there is no SQL standard regarding temporary functions. There are few systems that support it. Here are what I have found: 1. Hive: no DB qualifier allowed. Can overwrite built-in. 2. Spark: basically follows Hive ( https://docs.d

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

Hi Timo and Dawid, I discussed this offline a little bit with Jingsong and want to double check with you on the followings 2 questions. Can you please help confirm? 1. How will user set the Configurations? Users will have two ways to set an ExecutionConfig. Option 1: a) Instantiate ExecutionCon

[jira] [Created] (FLINK-13954) Clean up ExecutionEnvironment / JobSubmission code paths

Kostas Kloudas created FLINK-13954: -- Summary: Clean up ExecutionEnvironment / JobSubmission code paths Key: FLINK-13954 URL: https://issues.apache.org/jira/browse/FLINK-13954 Project: Flink

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

Hi Becket, You are right, that what we had in mind for ExecutionConfig/CheckpointConfig etc. is the option b) from your email. In the context of the FLIP-54, those objects are not Configurable. What we understood as a Configurable by the FLIP-54 are a simple pojos, that are stored under a single k

Re: [DISCUSS] Flink client api enhancement for downstream project

Here is the issue, and I will keep on updating it as I find more issues. https://issues.apache.org/jira/browse/FLINK-13954 This will also cover the refactoring of the Executors that we discussed in this thread, without any additional functionality (such as the job client). Kostas On Wed, Sep 4,

Re: [DISCUSS] Reducing build times

Yes, we can ensure the same (or better) experience for contributors. On the powerful machines, builds finish in 1.5 hours (without any caching enabled). Azure Pipelines offers 10 concurrent builds with a timeout of 6 hours for a build for open source projects. Flink needs 3.5 hours on that infra

Re: [VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration

+1 (binding) On 04/09/2019 11:13, Zhu Zhu wrote: +1 (non-binding) Thanks, Zhu Zhu Till Rohrmann 于2019年9月4日周三 下午5:05写道： Hi everyone, I would like to start the voting process for FLIP-61 [1], which is discussed and reached consensus in this thread [2]. Since the change is rather small I'd l

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

+1 (binding) On 04/09/2019 11:18, JingsongLee wrote: +1 (non-binding) default 0 is really not user production friendly. Best, Jingsong Lee -- From:Zhu Zhu Send Time:2019年9月4日(星期三) 17:13 To:dev Subject:Re: [VOTE] FLIP-62: Set d

Re: [DISCUSS] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

The issue we seem to run into again and again is that we want to try to find a value that provides a good experience when trying out Flink, but also somewhat usable for production users. We should look into solutions for this; maybe having a "recommended" value in the docs would help sufficientl

[jira] [Created] (FLINK-13955) Integrate ContinuousFileReaderOperator with StreamTask mailbox execution model

Alex created FLINK-13955: Summary: Integrate ContinuousFileReaderOperator with StreamTask mailbox execution model Key: FLINK-13955 URL: https://issues.apache.org/jira/browse/FLINK-13955 Project: Flink

Re: [DISCUSS] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

An improved documentation can definitely help. I think Arvid suggested something like this in the linked SURVEY thread and said that Kafka does something similar. The idea of different profiles sounds also promising. I guess something like this deserves a dedicated effort and someone driving it.

Re: [DISCUSS] Reducing build times

How long do we need to run all e2e tests? They are not included in the 3,5 hours I assume. Cheers, Till On Wed, Sep 4, 2019 at 11:59 AM Robert Metzger wrote: > Yes, we can ensure the same (or better) experience for contributors. > > On the powerful machines, builds finish in 1.5 hours (without

Re: [DISCUSS] Use static imports for test utilities or asserts.

Hi Yang, are you referring to Flink's checkstyle settings which forbids star imports? If you set up the checkstyle plugin [1] and activate optimize imports in IntelliJ, then the IDE will take care that everything is correct. [1] https://ci.apache.org/projects/flink/flink-docs-stable/flinkDev/ide_

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

+1 (binding) On Wed, Sep 4, 2019 at 12:43 PM Chesnay Schepler wrote: > +1 (binding) > > On 04/09/2019 11:18, JingsongLee wrote: > > +1 (non-binding) > > > > default 0 is really not user production friendly. > > > > Best, > > Jingsong Lee > > > > > > --

Re: [VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration

+1 (binding) On Wed, Sep 4, 2019 at 12:39 PM Chesnay Schepler wrote: > +1 (binding) > > On 04/09/2019 11:13, Zhu Zhu wrote: > > +1 (non-binding) > > > > Thanks, > > Zhu Zhu > > > > Till Rohrmann 于2019年9月4日周三 下午5:05写道： > > > >> Hi everyone, > >> > >> I would like to start the voting process for

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

+1 On Wed, Sep 4, 2019 at 1:38 PM Till Rohrmann wrote: > +1 (binding) > > On Wed, Sep 4, 2019 at 12:43 PM Chesnay Schepler > wrote: > > > +1 (binding) > > > > On 04/09/2019 11:18, JingsongLee wrote: > > > +1 (non-binding) > > > > > > default 0 is really not user production friendly. > > > > > >

Re: [DISCUSS] Reducing build times

e2e tests on Travis add another 4-5 hours, but we never optimized these to make use of the cached Flink artifact. On 04/09/2019 13:26, Till Rohrmann wrote: How long do we need to run all e2e tests? They are not included in the 3,5 hours I assume. Cheers, Till On Wed, Sep 4, 2019 at 11:59 AM R

Fine grained batch recovery vs. native libraries

Hi, we're testing the newly released batch recovery and are running into class loading related issues. 1) We have a per-job flink cluster 2) We use BATCH execution mode + region failover strategy Point 1) should imply single user code class loader per task manager (because there is only single p

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi all, Regarding #1 temp function <> built-in function and naming. I'm fine with temp functions should precede built-in function and can override built-in functions (we already support to override built-in function in 1.9). If we don't allow the same name as a built-in function, I'm afraid we wil

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

+1 Best, Jark > 在 2019年9月4日，19:43，David Morávek 写道： > > +1 > > On Wed, Sep 4, 2019 at 1:38 PM Till Rohrmann wrote: > >> +1 (binding) >> >> On Wed, Sep 4, 2019 at 12:43 PM Chesnay Schepler >> wrote: >> >>> +1 (binding) >>> >>> On 04/09/2019 11:18, JingsongLee wrote: +1 (non-binding)

[jira] [Created] (FLINK-13956) Add sequence file format with repeated sync blocks

Arvid Heise created FLINK-13956: --- Summary: Add sequence file format with repeated sync blocks Key: FLINK-13956 URL: https://issues.apache.org/jira/browse/FLINK-13956 Project: Flink Issue Type:

Re: Potential block size issue with S3 binary files

Hi Ken, as far as I understood, you are using the format to overcome some short comings in Flink. There is no need to even look at the data or even to create it if the join would work decently. If so, then it would make sense to keep the format, as I expect similar issues to always appear and pro

Re: Fine grained batch recovery vs. native libraries

This sounds like a serious bug, please open a JIRA ticket. On 04/09/2019 13:41, David Morávek wrote: Hi, we're testing the newly released batch recovery and are running into class loading related issues. 1) We have a per-job flink cluster 2) We use BATCH execution mode + region failover strate

[jira] [Created] (FLINK-13957) Redact passwords from dynamic properties on job submission

Matyas Orhidi created FLINK-13957: - Summary: Redact passwords from dynamic properties on job submission Key: FLINK-13957 URL: https://issues.apache.org/jira/browse/FLINK-13957 Project: Flink

[jira] [Created] (FLINK-13958) Job class loader may not be reused after batch job recovery

David Moravek created FLINK-13958: - Summary: Job class loader may not be reused after batch job recovery Key: FLINK-13958 URL: https://issues.apache.org/jira/browse/FLINK-13958 Project: Flink

Re: Fine grained batch recovery vs. native libraries

Hi Chesnay, I've created FLINK-13958 to track the issue. Thanks, D. On Wed, Sep 4, 2019 at 1:56 PM Chesnay Schepler wrote: > This sounds like a serious bug, please open a JIRA ticket. > > On 04/09/2019 13:41, David Morávek wrote: > > Hi, > > >

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Jincheng, 2. Serializability of functions: "#2 is very convenient for users" means only until they have the first backwards-compatibility issue, after that they will find it not so convinient anymore and will ask why the framework allowed storing such objects in a persistent storage. I don'

[jira] [Created] (FLINK-13959) Consolidate DetachedEnvironment and ContextEnvironment

Kostas Kloudas created FLINK-13959: -- Summary: Consolidate DetachedEnvironment and ContextEnvironment Key: FLINK-13959 URL: https://issues.apache.org/jira/browse/FLINK-13959 Project: Flink Is

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

Thanks for updating the FLIP Xintong. It looks good to me. I would be ok to start a vote for it. Best, Andrey On Wed, Sep 4, 2019 at 10:03 AM Xintong Song wrote: > @all > > The FLIP document [1] has been updated. > > Thank you~ > > Xintong Song > > > [1] > > https://cwiki.apache.org/confluence/

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi Yijie, Thanks for the interest in contributing the Pulsar connector. In general, I think having Pulsar connector with strong support is a valuable addition to Flink. So I am happy the shepherd this effort. Meanwhile, I would also like to provide some context and recent efforts on the Flink con

Re: FLIP-63: Rework table partition support

Hi Jingsong, thanks for your proposal. Could you repost this email with the subject: "[DISCUSS] FLIP-63: Rework table partition support" Some people have filters for [DISCUSS] threads and it also makes important emails more prominent visually. Thanks, Timo On 04.09.19 09:11, JingsongLee wro

[DISCUSS] Support JSON functions in Flink SQL

Hi everybody, I'd like to kick off a discussion on Support JSON functions in Flink SQL. The entire plan is divided into two steps: 1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1]. 2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL, such as JSON_TYPE in Mysql, JSO

Re: [DISCUSS] Support JSON functions in Flink SQL

Thanks Xu Forward for bring up this topic, I think the JSON functions are very useful especially for those MySQL users. I saw that you have done some work within the Apache Calcite, that’s a good start, but this is one concern from me, Flink doesn’t support JSON type internal, so how to represe

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi, Things looks interesting so far! I had one question: Where will most of the support code for this live? Will this add the required code to flink-table-common or the different runners? Can we implement this in such a way that only a minimal amount of support code is required in the parts of

Storing offsets in Kafka

Hey, I was wondering whether something has changed for KafkaConsumer, since I am using Kafka 2.0.0 with Flink and I wanted to use group offsets but there seems to be no change in the topic where Kafka stores it's offsets, after restart Flink uses the `auto.offset.reset` so it seems that there is no

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi all, thanks for the healthy discussion. It is already a very long discussion with a lot of text. So I will just post my opinion to a couple of statements: > Hive built-in functions are not part of Flink built-in functions, they are catalog functions That is not entirely true. Correct me

Re: [DISCUSS] Support JSON functions in Flink SQL

hi Danny Chan ,Thank you very much for your reply, your help can help me further improve this discussion. Best forward Danny Chan 于2019年9月4日周三 下午8:50写道： > Thanks Xu Forward for bring up this topic, I think the JSON functions are > very useful especially for those MySQL users. > > I saw that you

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

Maybe to clear up confusion about my suggestion: I would vote to keep the name of the config parameter "taskmanager.memory.network" because it is the same key as currently (good to not break things unless good reason) and there currently is no case or even a concrete follow-up where we would actua

Re: [DISCUSS] Support JSON functions in Flink SQL

Hi Forward, Thanks for bringing this discussion and preparing the nice design. I think it's nice to have the JSON functions in the next release. We have received some requirements for this feature. I can help to shepherd this JSON functions effort and will leave comments in the design doc in the

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi, Regarding the Hive & Spark support of TEMPORARY FUNCTIONS. I've just performed some experiments (hive-2.3.2 & spark 2.4.4) and I think they are very inconsistent in that manner (spark being way worse on that). Hive: You cannot overwrite all the built-in functions. I could overwrite most of t

[jira] [Created] (FLINK-13960) Provide default implementation for deprecated HighAvailabilityServices.getWebMonitorLeaderRetriever

Till Rohrmann created FLINK-13960: - Summary: Provide default implementation for deprecated HighAvailabilityServices.getWebMonitorLeaderRetriever Key: FLINK-13960 URL: https://issues.apache.org/jira/browse/FLINK-13

[DISCUSS] FLIP-64: Support for Temporary Objects in Table module

Hi all, As part of FLIP-30 a Catalog API was introduced that enables storing table meta objects permanently. At the same time the majority of current APIs create temporary objects that cannot be serialized. We shoul

[jira] [Created] (FLINK-13961) Remove obsolete abstraction JobExecutor(Service)

TisonKun created FLINK-13961: Summary: Remove obsolete abstraction JobExecutor(Service) Key: FLINK-13961 URL: https://issues.apache.org/jira/browse/FLINK-13961 Project: Flink Issue Type: Sub-tas

[jira] [Created] (FLINK-13962) Execution#taskRestore leaks if task fails before deploying

Zhu Zhu created FLINK-13962: --- Summary: Execution#taskRestore leaks if task fails before deploying Key: FLINK-13962 URL: https://issues.apache.org/jira/browse/FLINK-13962 Project: Flink Issue Type:

[jira] [Created] (FLINK-13963) Consolidate Hadoop file systems usage and Hadoop integration docs

Andrey Zagrebin created FLINK-13963: --- Summary: Consolidate Hadoop file systems usage and Hadoop integration docs Key: FLINK-13963 URL: https://issues.apache.org/jira/browse/FLINK-13963 Project: Flin

[jira] [Created] (FLINK-13964) Remove usage of deprecated methods from MiniCluster

Till Rohrmann created FLINK-13964: - Summary: Remove usage of deprecated methods from MiniCluster Key: FLINK-13964 URL: https://issues.apache.org/jira/browse/FLINK-13964 Project: Flink Issue T

Re: [DISCUSS] FLIP-55: Introduction of a Table API Java Expression DSL

Thanks for your feedback Rong. You are right, we can still have shorter names if the user feedback demands that. Adding additional shorter method names is always possible. So let's stick to lit() for now. I converted the Google document into a wiki page: https://cwiki.apache.org/confluence/dis

Re: instable checkpointing after migration to flink 1.8

Hi Stephan, sorry for late response. We indeed use timers inside a KeyedProcessFunction but the triggers of the timers are kinda evenly distributed, so not causing a firing storm. We have a custom ttl logic which is used by the deletion timer to decide whether delete a record from inmemory state or

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

+1 Best, Zhijiang -- From:Jark Wu Send Time:2019年9月4日(星期三) 13:45 To:dev Subject:Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s +1 Best, Jark > 在 2019年9月4日，19:43，David Morávek

Re: [VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration

+1 -- From:Till Rohrmann Send Time:2019年9月4日(星期三) 13:39 To:dev Cc:Zhu Zhu Subject:Re: [VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration +1 (binding) On Wed, Sep 4, 2019 at 12:39 PM Chesnay Schepler wro

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

Let's not block on config key names, just go ahead and we figure this out concurrently or on the PR later. On Wed, Sep 4, 2019 at 3:48 PM Stephan Ewen wrote: > Maybe to clear up confusion about my suggestion: > > I would vote to keep the name of the config parameter > "taskmanager.memory.networ

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Let me try to summarize and conclude the long thread so far: 1. For order of temp function v.s. built-in function: I think Dawid's point that temp function should be of fully qualified path is a better reasoning to back the newly proposed order, and i agree we don't need to follow Hive/Spark. Ho

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi David, Thanks for sharing the findings about temporary functions. Because of strong inconsistency observed in Spark, we can probably ignore it for now. For Hive, I understand one may not be able to overwrite everything, but the capability is being offered. Whether we offer this capability is t

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Thank all for the sharing thoughts. I think we have gathered some useful initial feedback from this long discussion with a couple of focal points sticking out. We will go back to do more research and adapt our proposal. Once it's ready, we will ask for a new round of review. If there is any disag

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Thanks everyone for the comments and feedback. It seems to me that the main question here is about - "how can the Flink community maintain the connector?". Here are two thoughts from myself. 1) I think how and where to host this integration is kind of less important here. I believe there can be

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Thanks for all the feedback and suggestions! As Sijie said, the goal of the connector has always been to provide users with the latest features of both systems as soon as possible. We propose to contribute the connector to Flink and hope to get more suggestions and feedback from Flink experts to e

Re: instable checkpointing after migration to flink 1.8

Hi Bekir, If it is the storage place for timers, for RocksDBStateBackend, timers can be stored in Heap or RocksDB[1][2] [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#tuning-rocksdb [2] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/s

Re: [DISCUSS] Support JSON functions in Flink SQL

We also see strong demands from our SQL users for JSON/Date related functions. Also +Anyang Hu On Wed, Sep 4, 2019 at 9:51 PM Jark Wu wrote: > Hi Forward, > > Thanks for bringing this discussion and preparing the nice design. > I think it's nice to have the JSON functions in the next release.

Re: instable checkpointing after migration to flink 1.8

Another information from our private emails there ALWAYS have Kafka AbstractCoordinator logs about lost connection to Kafka at the same time we have the checkpoints confirmed. Bekir checked the Kafka broker log, but did not find any interesting things there. Best, Congxian Congxian Qiu 于2019年9

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Timo, Yes, I think about convenient is a trade-off, for now think convenient compare with always do right thing I prefer add some limitation ensure user never encounter issues. So, #2 we are on the same page now. Best, Jincheng Aljoscha Krettek 于2019年9月4日周三 下午8:58写道： > Hi, > > Things looks

Re: [VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration

+1 zhijiang 于2019年9月5日周四 上午12:36写道： > +1 > -- > From:Till Rohrmann > Send Time:2019年9月4日(星期三) 13:39 > To:dev > Cc:Zhu Zhu > Subject:Re: [VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy > configuration > > +1 (bindin

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Aljoscha, That's a good points, so far, most of the code will live in flink-python module, and the rules and relNodes will be put into the both blink and flink planner modules, some of the common interface of required by planners will be placed in flink-table-common. I think you are right, we s

Re: Storing offsets in Kafka

Hi Dominik, There has not been any change to the offset committing logic in KafkaConsumer for a while. But the logic is a little complicated. The offset commit to Kafka is only enabled in the following two cases: 1. Flink checkpoint is enabled AND commitOffsetsOnCheckpoint is true (default value

[jira] [Created] (FLINK-13965) Keep hasDeprecatedKeys in ConfigOption and mark it with @Deprecated annotation

vinoyang created FLINK-13965: Summary: Keep hasDeprecatedKeys in ConfigOption and mark it with @Deprecated annotation Key: FLINK-13965 URL: https://issues.apache.org/jira/browse/FLINK-13965 Project: Flink

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Maybe Xuefu missed my email. Please let me know what your thoughts are on the summary, if there's still major controversy, I can take time to reevaluate that part. On Wed, Sep 4, 2019 at 2:25 PM Xuefu Z wrote: > Thank all for the sharing thoughts. I think we have gathered some useful > initial

Re: [DISCUSS] Support JSON functions in Flink SQL

+1 to add JSON support to Flink. We also see lots of requirements for JSON related functions in our internal platform. Since these are already SQL standard, I think it's a good time to add them to Flink. Best, Kurt On Thu, Sep 5, 2019 at 10:37 AM Qi Luo wrote: > We also see strong demands from

Re: Re: [DISCUSS] Support JSON functions in Flink SQL

+1 I have done similar work before. Looking forward to discussing this feature. Best wenhui winifred.wenhui.t...@gmail.com From: Kurt Young Date: 2019-09-05 14:00 To: dev CC: Anyang Hu Subject: Re: [DISCUSS] Support JSON functions in Flink SQL +1 to add JSON support to Flink. We also see lo

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi Xuefu, Just wanted to summarize my opinion on the one topic (temporary functions). My preference would be to make temporary functions always 3-part qualified (as a result that would prohibit overriding built-in functions). Having said that if the community decides that it's better to allow ove

Re: [VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration

+1 (non-binding) Zili Chen 于2019年9月5日周四 上午10:55写道： > +1 > > > zhijiang 于2019年9月5日周四 上午12:36写道： > > > +1 > > -- > > From:Till Rohrmann > > Send Time:2019年9月4日(星期三) 13:39 > > To:dev > > Cc:Zhu Zhu > > Subject:Re: [VOTE] FLIP-61 Si

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

Hi Dawid, Thanks a lot for the clarification. Got it now. A few more thoughts: 1. Naming. I agree that if the name of "Configurable" is a little misleading if we just want to use it to save POJOs. It would probably help to just name it something like "ConfigPojo". 2. Flat config map v.s. structu

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi Sijie and Yijie, Thanks for sharing your thoughts. Just want to have some update on FLIP-27. Although the FLIP wiki and discussion thread has been quiet for some time, a few committer / contributors in Flink community were actually prototyping the entire thing. We have made some good progress