Re: [VOTE] FLIP-64: Support for Temporary Objects in Table module

Thank you all for the votes. We got * five +1(binding) votes (Kurt, Aljoscha, Jark, Bowen, Timo) and one non-binding (Vino) * No -1 votes Therefore, I'm glad to announce that FLIP-64 is approved to be adopted by Apache Flink. Best, Dawid On 16/10/2019 10:32, Timo Walther wrote: > +1 >

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi Yijie, Thanks for the design document. I agree with Bowen that the catalog part needs more details. And I would suggest to separate Pulsar Catalog as another FLIP. IMO, it has little to do with source/sink. Having a separate FLIP can unblock the contribution for sink (or source) and keep the di

[jira] [Created] (FLINK-14426) Test and pass 103 tpcds tests

Leonard Xu created FLINK-14426: -- Summary: Test and pass 103 tpcds tests Key: FLINK-14426 URL: https://issues.apache.org/jira/browse/FLINK-14426 Project: Flink Issue Type: Sub-task Comp

[jira] [Created] (FLINK-14425) Pick necessary commits gently that tpc-ds tests need from flink-tpc-ds project

Leonard Xu created FLINK-14425: -- Summary: Pick necessary commits gently that tpc-ds tests need from flink-tpc-ds project Key: FLINK-14425 URL: https://issues.apache.org/jira/browse/FLINK-14425 Project: F

[jira] [Created] (FLINK-14424) Create tpc-ds end to end test

Leonard Xu created FLINK-14424: -- Summary: Create tpc-ds end to end test Key: FLINK-14424 URL: https://issues.apache.org/jira/browse/FLINK-14424 Project: Flink Issue Type: Sub-task Com

[jira] [Created] (FLINK-14423) Import teradat.tpcds as test data generator (refer to hive)

Leonard Xu created FLINK-14423: -- Summary: Import teradat.tpcds as test data generator (refer to hive) Key: FLINK-14423 URL: https://issues.apache.org/jira/browse/FLINK-14423 Project: Flink Issu

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi Bowen, Thanks for your comments. I'll add catalog details as you suggested. One more question: since we decide to not implement source part of the connector at the moment. What can users do with a Pulsar catalog? Create a table backed by Pulsar and check existing pulsar tables to see their sch

Re: [VOTE] FLINK-67: Cluster partition life-cycle

Hi Chesnay, +1 (non-binding). Very thanks for driving this. Best, Yun -- From:Chesnay Schepler Send Time:2019 Oct. 16 (Wed.) 21:01 To:dev@flink.apache.org Subject:[VOTE] FLINK-67: Cluster partiti

[jira] [Created] (FLINK-14422) Get taskmanager resource config in rest api

lining created FLINK-14422: -- Summary: Get taskmanager resource config in rest api Key: FLINK-14422 URL: https://issues.apache.org/jira/browse/FLINK-14422 Project: Flink Issue Type: Wish Co

Re: [VOTE] FLINK-67: Cluster partition life-cycle

+1 Best, Zhijiang -- From:Chesnay Schepler Send Time:2019年10月16日(星期三) 21:01 To:dev@flink.apache.org Subject:[VOTE] FLINK-67: Cluster partition life-cycle Hello, please vote for FLIP-67

Re: [VOTE] FLIP-68: Extend Core Table System with Pluggable Modules

+1 Thanks, Jark On Thu, 17 Oct 2019 at 04:44, Peter Huang wrote: > +1 Thanks > > On Wed, Oct 16, 2019 at 12:48 PM Xuefu Z wrote: > > > +1 (non-biding) > > > > On Wed, Oct 16, 2019 at 2:26 AM Timo Walther wrote: > > > > > +1 > > > > > > Thanks, > > > Timo > > > > > > > > > On 15.10.19 20:50, B

Fwd: Verifying correctness of StreamingFileSink (Kafka -> S3)

Hi Amran, I am including also the message to the public ML because this may be interesting to other users as well. You can always write your input stream to two sinks, as shown below: DataStream<...> myInputFromKafka = env.addSource(new FlinkKafkaConsumerVERSION(...)); final StreamingFileSink s

Re: [VOTE] FLIP-68: Extend Core Table System with Pluggable Modules

+1 Thanks On Wed, Oct 16, 2019 at 12:48 PM Xuefu Z wrote: > +1 (non-biding) > > On Wed, Oct 16, 2019 at 2:26 AM Timo Walther wrote: > > > +1 > > > > Thanks, > > Timo > > > > > > On 15.10.19 20:50, Bowen Li wrote: > > > Hi all, > > > > > > I'd like to kick off a voting thread for FLIP-68: Extend

Re: [VOTE] FLIP-68: Extend Core Table System with Pluggable Modules

+1 (non-biding) On Wed, Oct 16, 2019 at 2:26 AM Timo Walther wrote: > +1 > > Thanks, > Timo > > > On 15.10.19 20:50, Bowen Li wrote: > > Hi all, > > > > I'd like to kick off a voting thread for FLIP-68: Extend Core Table > System > > with Pluggable Modules [1], as we have reached consensus in [2

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi Yijie, Per the discussion, maybe you can move pulsar source to 'future work' section in the FLIP for now? Besides, the FLIP seems to be quite rough at the moment, and I'd recommend to add more details . A few questions mainly regarding the proposed pulsar catalog. - Can you provide some b

[jira] [Created] (FLINK-14421) Add 'L' when define a long value

hailong wang created FLINK-14421: Summary: Add 'L' when define a long value Key: FLINK-14421 URL: https://issues.apache.org/jira/browse/FLINK-14421 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-59: Enable execution configuration from Configuration object

Hi Kostas, can we still discuss the naming of the properties? For me, having "execution" and "exector" as prefixes might be confusing in the future and difficult to identify if you scan through a list of properties. How about `deployment` and `execution`? Or `deployer` and `exec`? Regards, T

[jira] [Created] (FLINK-14420) Add documentation for pluggable module

Bowen Li created FLINK-14420: Summary: Add documentation for pluggable module Key: FLINK-14420 URL: https://issues.apache.org/jira/browse/FLINK-14420 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-14419) Add ModuleFactory for factory discovery service

Bowen Li created FLINK-14419: Summary: Add ModuleFactory for factory discovery service Key: FLINK-14419 URL: https://issues.apache.org/jira/browse/FLINK-14419 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-14418) Create HiveModule to provide Hive built-in functions

Bowen Li created FLINK-14418: Summary: Create HiveModule to provide Hive built-in functions Key: FLINK-14418 URL: https://issues.apache.org/jira/browse/FLINK-14418 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-14417) Develop CoreModule to provide Flink built-in functions

Bowen Li created FLINK-14417: Summary: Develop CoreModule to provide Flink built-in functions Key: FLINK-14417 URL: https://issues.apache.org/jira/browse/FLINK-14417 Project: Flink Issue Type: Su

[jira] [Created] (FLINK-14416) Add Module interface and ModuleManager

Bowen Li created FLINK-14416: Summary: Add Module interface and ModuleManager Key: FLINK-14416 URL: https://issues.apache.org/jira/browse/FLINK-14416 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-59: Enable execution configuration from Configuration object

Hi all, Thanks for opening the discussion! I like the idea, so +1 from my side and actually this is aligned with our intensions for the FLIP-73 effort. For the naming convention of the parameters introduced in the FLIP, my proposal would be have the full word "execution" instead of the shorter "

[jira] [Created] (FLINK-14415) ValueLiteralExpression#equals should take array value into account

Jark Wu created FLINK-14415: --- Summary: ValueLiteralExpression#equals should take array value into account Key: FLINK-14415 URL: https://issues.apache.org/jira/browse/FLINK-14415 Project: Flink Iss

[jira] [Created] (FLINK-14414) Support string serializable for SymbolType

Jark Wu created FLINK-14414: --- Summary: Support string serializable for SymbolType Key: FLINK-14414 URL: https://issues.apache.org/jira/browse/FLINK-14414 Project: Flink Issue Type: Bug Co

Re: Mongo Connector

Hi Jingsong Lee, Thanks for the details. Were you able to achieve end-to-end exactly once support with Mongo?  Also, for doing any intermittent reads from Mongo (Kafka -> process event -> lookup Mongo -> enhance event -> Sink to Mongo), I am thinking of using Async IO (https://ci.apache.org/pr

[jira] [Created] (FLINK-14413) Some quotes from NOTICE files that cause encoding problems

Chesnay Schepler created FLINK-14413: Summary: Some quotes from NOTICE files that cause encoding problems Key: FLINK-14413 URL: https://issues.apache.org/jira/browse/FLINK-14413 Project: Flink

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

Whether the side project will be overlooked of not will depends a lot on how we integrate it with the current Flink website and documentation. I would think that a separate repository is not necessarily a big problem there. It might also help, because a link to that repo shows prominently that par

[VOTE] FLINK-67: Cluster partition life-cycle

Hello, please vote for FLIP-67 . The discussion thread can be found here . This

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

Yes, all code managed by the Flink project will be "org.apache.flink." On Wed, Oct 16, 2019 at 9:57 AM Jark Wu wrote: > I think it makes sense to keep it in a separate repo. It's a good chance to > test the pros and cons of "splitting flink repository". > > Btw, I think we will change the packag

[jira] [Created] (FLINK-14412) Rename ML Pipeline to MLPipeline

Aljoscha Krettek created FLINK-14412: Summary: Rename ML Pipeline to MLPipeline Key: FLINK-14412 URL: https://issues.apache.org/jira/browse/FLINK-14412 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-14411) OrcTableSource read error

zlzhang0122 created FLINK-14411: --- Summary: OrcTableSource read error Key: FLINK-14411 URL: https://issues.apache.org/jira/browse/FLINK-14411 Project: Flink Issue Type: Bug Affects Versions:

[jira] [Created] (FLINK-14410) Retrieve CharacterFilter once after registration

Chesnay Schepler created FLINK-14410: Summary: Retrieve CharacterFilter once after registration Key: FLINK-14410 URL: https://issues.apache.org/jira/browse/FLINK-14410 Project: Flink Issu

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

+1 for separate repositories. This is also good for the community to collect some experience for a potential repository split effort at some later point. On Wed, Oct 16, 2019 at 12:01 PM vino yang wrote: > Hi, > > Fast release cycles seems a good viewpoint to support keeping it in a > separate r

Re: [VOTE] FLIP-77: Introduce ConfigOptions with Data Types

+1 Best, tison. Aljoscha Krettek 于2019年10月16日周三 下午6:25写道： > +1 > > > On 16. Oct 2019, at 10:30, Timo Walther wrote: > > > > +1 > > > > Thanks, > > Timo > > > > On 15.10.19 17:07, Till Rohrmann wrote: > >> Sorry for the confusion. I should have checked with an external mail > >> client. Thanks

Re: [VOTE] FLIP-77: Introduce ConfigOptions with Data Types

+1 > On 16. Oct 2019, at 10:30, Timo Walther wrote: > > +1 > > Thanks, > Timo > > On 15.10.19 17:07, Till Rohrmann wrote: >> Sorry for the confusion. I should have checked with an external mail >> client. Thanks a lot for the clarification. >> >> Cheers, >> Till >> >> On Tue, Oct 15, 2019 at

Re: [DISCUSS] FLIP-65: New type inference for Table API UDFs

Hi everyone, thanks for the comments in the doc. The feedback so far was very positive. I will convert the doc into a FLIP wiki page now. If there are no further concerns, I would like to start a voting process soon. Thanks, Timo On 09.10.19 04:58, Jingsong Li wrote: Thanks Timo for your

[jira] [Created] (FLINK-14409) MapType doesn't accept any subclass of java.util.Map

Jark Wu created FLINK-14409: --- Summary: MapType doesn't accept any subclass of java.util.Map Key: FLINK-14409 URL: https://issues.apache.org/jira/browse/FLINK-14409 Project: Flink Issue Type: Bug

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

Hi, Fast release cycles seems a good viewpoint to support keeping it in a separate repository. IMO, the placement of documentation should keep consistency with the repository. Best, Vino Timo Walther 于2019年10月16日周三 下午4:02写道： > Hi Stephan, > > +1 for keeping it in a separate repository for fas

Re: [DISCUSS] FLIP policy for introducing config option keys

Hi all, I agree with Jark. Having a voting with at least 3 binding votes makes sense for API changes. It also forces people to question the introduction of another config option that might make the configuration of Flink more complicated. A FLIP is usually a bigger effort with long term impac

[jira] [Created] (FLINK-14408) In OldPlanner, UDF open method can not be invoke when SQL is optimized

hailong wang created FLINK-14408: Summary: In OldPlanner, UDF open method can not be invoke when SQL is optimized Key: FLINK-14408 URL: https://issues.apache.org/jira/browse/FLINK-14408 Project: Flink

Re: [VOTE] FLIP-68: Extend Core Table System with Pluggable Modules

+1 Thanks, Timo On 15.10.19 20:50, Bowen Li wrote: Hi all, I'd like to kick off a voting thread for FLIP-68: Extend Core Table System with Pluggable Modules [1], as we have reached consensus in [2]. The voting period will be open for at least 72 hours, ending at 7pm Oct 18 UTC. Thanks, Bowe

[jira] [Created] (FLINK-14407) YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3n failed on Travis

Till Rohrmann created FLINK-14407: - Summary: YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3n failed on Travis Key: FLINK-14407 URL: https://issues.apache.org/jira/browse/FLINK-14407 Project: F

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

Hi all, Although in the initial thread I said that, in general, I would prefer having one repository, I understand that arguments presented here and I think it makes sense for such a young project to have its own repository. So +1 from my side, with an asterisk about hoping that eventually the pr

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

I think it makes sense to keep the stateful functions code in a separate repository in the beginning as described. At a later point in time we could revisit this topic if we see that the split codebase becomes a problem or if there are other benefits such as better visibility. For the website, we

[jira] [Created] (FLINK-14406) Add metric for manage memory

lining created FLINK-14406: -- Summary: Add metric for manage memory Key: FLINK-14406 URL: https://issues.apache.org/jira/browse/FLINK-14406 Project: Flink Issue Type: Improvement Components

Re: [DISCUSS] FLIP policy for introducing config option keys

Hi all, I think config option changes deserves a voting process, but not necessarily a FLIP. My concern for always having FLIPs on config option changes is that, we might result in too many FLIPs, which makes it difficult for people who wants to track the major design changes other than pure conf

Re: [VOTE] FLIP-64: Support for Temporary Objects in Table module

+1 Thanks, Timo On 15.10.19 19:04, Bowen Li wrote: +1 On Tue, Oct 15, 2019 at 5:09 AM Jark Wu wrote: +1 from my side. Cheers, Jark On Tue, 15 Oct 2019 at 19:11, vino yang wrote: +1 Best, Vino Aljoscha Krettek 于2019年10月15日周二 下午4:31写道： +1 Best, Aljoscha On 14. Oct 2019, at 14:55,

Re: [VOTE] FLIP-77: Introduce ConfigOptions with Data Types

+1 Thanks, Timo On 15.10.19 17:07, Till Rohrmann wrote: Sorry for the confusion. I should have checked with an external mail client. Thanks a lot for the clarification. Cheers, Till On Tue, Oct 15, 2019 at 2:07 PM Jark Wu wrote: +1 It is a separate [VOTE] thread in my Mail client. Best,

Re: [VOTE] FLIP-78: Flink Python UDF Environment and Dependency Management

I'm also late to the party here :) When I saw the first draft, I was thinking how exactly the design doc would tie in with Beam. Thanks for the update. A couple of comments with this regard: Flink has provided a distributed cache mechanism and allows users to upload their files using "regist

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

Hi Stephan, +1 for keeping it in a separate repository for fast release cycles and stability until it is mature enough. But we should definitely merge it back to the core repo also for marketing reasons. IMHO side projects tend to be overlooked by the outside world even though they are great

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

I think it makes sense to keep it in a separate repo. It's a good chance to test the pros and cons of "splitting flink repository". Btw, I think we will change the package path from "com.ververica" to "org.apache.flink" even if it goes into a separate repo, right? Best, Jark On Wed, 16 Oct 2019

Re: performances of S3 writing with many buckets in parallel

Hi Enrico, Thanks for opening the discussion! One thing to note that may help s that the hadoop S3 FS tries to imitate a filesystem on top of S3: - before writing a key it checks if the "parent directory" exists by checking for a key with the prefix up to the last "/" - it creates empty marker f

Re: [VOTE] Drop Python 2 support for 1.10

+1 Bowen Li 于2019年10月16日周三 上午5:12写道： > +1 > > On Sun, Oct 13, 2019 at 10:54 PM Hequn Cheng wrote: > > > +1 > > > > Thanks a lot for driving this, Dian! > > > > On Mon, Oct 14, 2019 at 1:46 PM jincheng sun > > wrote: > > > > > +1 > > > > > > Dian Fu 于2019年10月14日周一 下午1:21写道： > > > > > > > Hi al

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

I would keep statefun in a separate repo in the beginning, for the reasons you mentioned. Best, Aljoscha > On 15. Oct 2019, at 23:40, Flavio Pompermaier wrote: > > Definitely on the same page..+1 to keep it in a separate repo (at least > until the cose becomes "stable" and widely adopted from

[jira] [Created] (FLINK-14405) Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups.

Xintong Song created FLINK-14405: Summary: Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups. Key: FLINK-14405 URL: https://issues.apache.org/jira/browse/FLINK-14405 Pr