Re: Ratelimiting in the Flink Kafka connector

2019-02-01 Thread Becket Qin
Hi Thomas, Yes, adding a rate limiting operator in front of the sink would work for record rate limiting. Another thing I am thinking is that for local throttling, it seems that throttling in sources and sinks has some subtle differences. For example, consider both source and sink as HDFS. For so

[DISCUSS] Enhance Operator API to Support Dynamically Selective Reading and EndOfInput Event

2019-02-01 Thread Guowei Ma
Hi, guys: I propose a design to enhance Stream Operator API for Batch’s requirements. This is also the Flink’s goal that Batch is a special case of Streaming. This proposal mainly contains two changes to operator api: 1. Allow "StreamOperator" can choose which input to read; 2. Notify "StreamOpera

Please add me for Flink contribution

2019-02-01 Thread abhishek sharma
Please provide me the contribution permission. email: abhion...@yahoo.com apache-username: abhioncbr Thanks Abhishek Sharma

[jira] [Created] (FLINK-11520) Triggers should be provided the window state

2019-02-01 Thread Elias Levy (JIRA)
Elias Levy created FLINK-11520: -- Summary: Triggers should be provided the window state Key: FLINK-11520 URL: https://issues.apache.org/jira/browse/FLINK-11520 Project: Flink Issue Type: Improvem

[jira] [Created] (FLINK-11519) Add function related catalog APIs

2019-02-01 Thread Xuefu Zhang (JIRA)
Xuefu Zhang created FLINK-11519: --- Summary: Add function related catalog APIs Key: FLINK-11519 URL: https://issues.apache.org/jira/browse/FLINK-11519 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-11518) Add partition related catalog APIs

2019-02-01 Thread Xuefu Zhang (JIRA)
Xuefu Zhang created FLINK-11518: --- Summary: Add partition related catalog APIs Key: FLINK-11518 URL: https://issues.apache.org/jira/browse/FLINK-11518 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-11517) Inefficient window state access when using RocksDB state backend

2019-02-01 Thread Elias Levy (JIRA)
Elias Levy created FLINK-11517: -- Summary: Inefficient window state access when using RocksDB state backend Key: FLINK-11517 URL: https://issues.apache.org/jira/browse/FLINK-11517 Project: Flink

Re: Ratelimiting in the Flink Kafka connector

2019-02-01 Thread Thomas Weise
Hi Becket, The throttling operator would suffer from the same issue of not being able to accurately count bytes. On the other hand, it can be used by composition w/o modifying existing operators. As for sinks, wouldn't an operator that adjusts the rate in front of the sink suffice? Thomas On

[jira] [Created] (FLINK-11516) Port and move some Descriptor classes to flink-table-common

2019-02-01 Thread Dian Fu (JIRA)
Dian Fu created FLINK-11516: --- Summary: Port and move some Descriptor classes to flink-table-common Key: FLINK-11516 URL: https://issues.apache.org/jira/browse/FLINK-11516 Project: Flink Issue Type

[jira] [Created] (FLINK-11515) Check and port ClientTest to new code base if necessary

2019-02-01 Thread TisonKun (JIRA)
TisonKun created FLINK-11515: Summary: Check and port ClientTest to new code base if necessary Key: FLINK-11515 URL: https://issues.apache.org/jira/browse/FLINK-11515 Project: Flink Issue Type: S

[jira] [Created] (FLINK-11514) Check and port ClusterClientTest to new code base if necessary

2019-02-01 Thread TisonKun (JIRA)
TisonKun created FLINK-11514: Summary: Check and port ClusterClientTest to new code base if necessary Key: FLINK-11514 URL: https://issues.apache.org/jira/browse/FLINK-11514 Project: Flink Issue

[jira] [Created] (FLINK-11513) Port CliFrontendSavepointTest to new code base

2019-02-01 Thread TisonKun (JIRA)
TisonKun created FLINK-11513: Summary: Port CliFrontendSavepointTest to new code base Key: FLINK-11513 URL: https://issues.apache.org/jira/browse/FLINK-11513 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-11512) Port CliFrontendModifyTest to new code base

2019-02-01 Thread TisonKun (JIRA)
TisonKun created FLINK-11512: Summary: Port CliFrontendModifyTest to new code base Key: FLINK-11512 URL: https://issues.apache.org/jira/browse/FLINK-11512 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Start new Review Process

2019-02-01 Thread Fabian Hueske
Great, thank you Robert! Am Do., 31. Jan. 2019 um 16:44 Uhr schrieb Robert Metzger < rmetz...@apache.org>: > Sorry, I haven't been active in the business of merging pull requests > recently. But you are both totally right: There is now a shiny big button > for merging pull requests. > > Regarding

Re: [DISCUSS] Contributing Chinese website and docs to Apache Flink

2019-02-01 Thread Tzu-Li (Gordon) Tai
Hi, Great seeing efforts in pushing this forward, Jark! A bit late in the party here, but I just want to also point out that: - +1 to https://flink.apache.org/zh/ as the website address - Add to the contribution checklist in PRs that both English / Chinese docs have been updated accordingly. I

Re: [DISCUSS] Contributing Chinese website and docs to Apache Flink

2019-02-01 Thread Fabian Hueske
Hi Jark, Thanks for working out a plan to integrate the Chinese translation! I would propose to first contribute and integrate the Chinese version of the main page (flink.apache.org). It is much smaller than the docs and a smaller scope might help to find tricks to ease the integration of the doc

Re: [DISCUSS] Standard / Convention for common connector metrics

2019-02-01 Thread Becket Qin
Thanks for the feedback, Konstantin, Completely agree that exposing the original metrics from the clients of other systems is a good practice and should be encouraged. Thanks, Jiangjie (Becket) Qin On Fri, Feb 1, 2019 at 4:01 PM Konstantin Knauf wrote: > Hi Becket, > > I like the idea of prov

[jira] [Created] (FLINK-11511) Remove legacy class JobAttachmentClientActor

2019-02-01 Thread TisonKun (JIRA)
TisonKun created FLINK-11511: Summary: Remove legacy class JobAttachmentClientActor Key: FLINK-11511 URL: https://issues.apache.org/jira/browse/FLINK-11511 Project: Flink Issue Type: Sub-task

Re: Request contributor permission

2019-02-01 Thread Robert Metzger
No problem :) I just granted you access to Confluence. On Fri, Feb 1, 2019 at 3:17 AM Becket Qin wrote: > Thanks Robert. > > Sorry I forgot to mention this, but yes, that is my JIRA id :) > > And my Apache confluence ID is also becket_ qin (becket@gmail.com). > Could you also help me with th

[jira] [Created] (FLINK-11510) Add the MultiFieldSumAggregator to support KeyedStream.sum(int[] positionToSums )

2019-02-01 Thread wangpeibin (JIRA)
wangpeibin created FLINK-11510: -- Summary: Add the MultiFieldSumAggregator to support KeyedStream.sum(int[] positionToSums ) Key: FLINK-11510 URL: https://issues.apache.org/jira/browse/FLINK-11510 Project

Re: [DISCUSS] Standard / Convention for common connector metrics

2019-02-01 Thread Konstantin Knauf
Hi Becket, I like the idea of providing a standard set of metrics, which sources can choose to implement/expose. In addition, I think, sources, like the Kafka or Kinesis Source, should continue to forward the original consumer metrics under their original names, so that users familiar with Kafka/K