Re: [DISCUSS] A more restrictive JIRA workflow
There are many reasons for jira, and one of the most important reasons is that flink varies greatly from version to version. However, I agree that you restrict some rights of developers, including the right to assign and edit. Robert Metzger 于2019年7月19日周五 下午10:58写道： > Hi all! > > Chesnay wrote: > > > We haven't wiped the set of contributors yet. Not sure if there's an > > easy way to remove the permissions for all of them; someone from the PMC > > may have to bite the bullet and click 600 times in a row :) > > > I don't think there's an easy way for us. > People with "Contributor" permissions have permissions such as "Closing" > and "Editing" (including "Transition", "Logging work", etc.) issues. > I would propose to either drastically reduce the number of people with > Contributor permissions, leaving just those in, who are helping to keep our > Jira in a clean shape (and this decision would be solely done at the > discretion of the PMC deleting people from the contributor group) > Another option would be to ask Infra to remove all people with Contributor > permissions from Flink (deleting and re-adding the group or so :) ) > Moving forward, the PMC would maintain a list of contributors who are > helping to keep Jira clean and well-organized. > > Tison wrote: > > > IIRC someone in this thread once mentioned that > > "Don't allow contributors to set a blocker priority." > > > Timo wrote this when he kicked off this thread. Sadly, this permission is > not provided by Jira out of the box: > > https://community.atlassian.com/t5/Jira-questions/How-to-restrict-the-setting-of-certain-priorities-to-certain/qaq-p/193230 > We will have to manually monitor tickets with "Blocker" status. > > > Fair enough, we might rephrase as > > "Pull requests belonging to unassigned Jira tickets or not authored by > > assignee will ..." > > > That's a good point. If nobody disagrees, would you like to open PR for > changing it? > > For the action "what should we do", iirc someone in this thread proposed > > closing it via flinkbot. Or leaving without review and merge could be ok > > before we implement automatic reply/reaction. > > > I proposed to automatically close. I believe that this might be too harsh. > I've implemented already an extension to Flinkbot which shows warnings for > a pull request. I just haven't found the time to test the extension and put > it in production. > > > > On Fri, Jul 19, 2019 at 5:51 AM Zili Chen wrote: > > > Fair enough, we might rephrase as > > > > "Pull requests belonging to unassigned Jira tickets or not authored by > > assignee will ..." > > > > For the action "what should we do", iirc someone in this thread proposed > > closing it via flinkbot. Or leaving without review and merge could be ok > > before we implement automatic reply/reaction. > > > > Best, > > tison. > > > > > > Zili Chen 于2019年7月19日周五 上午11:44写道： > > > >> @Jack > >> > >> From https://flink.apache.org/contributing/contribute-code.html the > >> community > >> claims > >> > >> - Only start working on the implementation if there is consensus on the > >> approach (e.g. you are assigned to the ticket) > >> > >> and accurately answer your question, > >> > >> - Pull requests belonging to unassigned Jira tickets will not be > reviewed > >> or merged by the community. > >> > >> Best, > >> tison. > >> > >> > >> Jark Wu 于2019年7月19日周五 上午11:28写道： > >> > >>> A quick question, what should we do if a developer creates a JIRA issue > >>> and > >>> then create a pull request at once without assigning? > >>> > >>> > >>> Regards, > >>> Jark > >>> > >>> On Thu, 18 Jul 2019 at 18:50, Zili Chen wrote: > >>> > >>> > Checking the result, as a discovery, I found that one can > >>> > still file a JIRA with "blocker" priority. > >>> > > >>> > IIRC someone in this thread once mentioned that > >>> > "Don't allow contributors to set a blocker priority." > >>> > > >>> > Chesnay, > >>> > > >>> > Thanks for the clarification. > >>> > > >>> > > >>> > Best, > >>> > tison. > >>> > > >>> > > >>> > Chesnay Schepler 于2019年7月18日周四 下午6:40写道： > >>> > > >>> > > We haven't wiped the set of contributors yet. Not sure if there's > an > >>> > > easy way to remove the permissions for all of them; someone from > the > >>> PMC > >>> > > may have to bite the bullet and click 600 times in a row :) > >>> > > > >>> > > On 18/07/2019 12:32, Zili Chen wrote: > >>> > > > Robert, > >>> > > > > >>> > > > Thanks for your effort. Rejecting contributor permission request > >>> > > > with a nice message and pointing them to the announcement sounds > >>> > > > reasonable. Just to be clear, we now have no person with > >>> contributor > >>> > > > role, right? > >>> > > > > >>> > > > Chesnay, > >>> > > > > >>> > > > https://flink.apache.org/contributing/contribute-code.html has > >>> been > >>> > > > updated and mentions that "Only committers can assign a Jira > >>> ticket." > >>> > > > > >>> > > > I think the corresponding update has been done. > >>> > > > > >>> > > > Best, > >>> > > >
Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog
hi, everyone I think this flip is very meaningful. it supports functions that can be shared by different catalogs and dbs, reducing the duplication of functions. Our group based on flink's sql parser module implements create function feature, stores the parsed function metadata and schema into mysql, and also customizes the catalog, customizes sql-client to support custom schemas and functions. Loaded, but the function is currently global, and is not subdivided according to catalog and db. In addition, I very much hope to participate in the development of this flip, I have been paying attention to the community, but found it is more difficult to join. thank you. Xuefu Z 于2019年9月17日周二 上午11:19写道： > Thanks to Tmo and Dawid for sharing thoughts. > > It seems to me that there is a general consensus on having temp functions > that have no namespaces and overwrite built-in functions. (As a side note > for comparability, the current user defined functions are all temporary and > having no namespaces.) > > Nevertheless, I can also see the merit of having namespaced temp functions > that can overwrite functions defined in a specific cat/db. However, this > idea appears orthogonal to the former and can be added incrementally. > > How about we first implement non-namespaced temp functions now and leave > the door open for namespaced ones for later releases as the requirement > might become more crystal? This also helps shorten the debate and allow us > to make some progress along this direction. > > As to Dawid's idea of having a dedicated cat/db to host the temporary temp > functions that don't have namespaces, my only concern is the special > treatment for a cat/db, which makes code less clean, as evident in treating > the built-in catalog currently. > > Thanks, > Xuefiu > > On Mon, Sep 16, 2019 at 5:07 PM Dawid Wysakowicz < > wysakowicz.da...@gmail.com> > wrote: > > > Hi, > > Another idea to consider on top of Timo's suggestion. How about we have a > > special namespace (catalog + database) for built-in objects? This catalog > > would be invisible for users as Xuefu was suggesting. > > > > Then users could still override built-in functions, if they fully qualify > > object with the built-in namespace, but by default the common logic of > > current dB & cat would be used. > > > > CREATE TEMPORARY FUNCTION func ... > > registers temporary function in current cat & dB > > > > CREATE TEMPORARY FUNCTION cat.db.func ... > > registers temporary function in cat db > > > > CREATE TEMPORARY FUNCTION system.system.func ... > > Overrides built-in function with temporary function > > > > The built-in/system namespace would not be writable for permanent > objects. > > WDYT? > > > > This way I think we can have benefits of both solutions. > > > > Best, > > Dawid > > > > > > On Tue, 17 Sep 2019, 07:24 Timo Walther, wrote: > > > > > Hi Bowen, > > > > > > I understand the potential benefit of overriding certain built-in > > > functions. I'm open to such a feature if many people agree. However, it > > > would be great to still support overriding catalog functions with > > > temporary functions in order to prototype a query even though a > > > catalog/database might not be available currently or should not be > > > modified yet. How about we support both cases? > > > > > > CREATE TEMPORARY FUNCTION abs > > > -> creates/overrides a built-in function and never consideres current > > > catalog and database; inconsistent with other DDL but acceptable for > > > functions I guess. > > > CREATE TEMPORARY FUNCTION cat.db.fun > > > -> creates/overrides a catalog function > > > > > > Regarding "Flink don't have any other built-in objects (tables, views) > > > except functions", this might change in the near future. Take > > > https://issues.apache.org/jira/browse/FLINK-13900 as an example. > > > > > > Thanks, > > > Timo > > > > > > On 14.09.19 01:40, Bowen Li wrote: > > > > Hi Fabian, > > > > > > > > Yes, I agree 1-part/no-override is the least favorable thus I didn't > > > > include that as a voting option, and the discussion is mainly between > > > > 1-part/override builtin and 3-part/not override builtin. > > > > > > > > Re > However, it means that temp functions are differently treated > than > > > > other db objects. > > > > IMO, the treatment difference results from the fact that functions > are > > a > > > > bit different from other objects - Flink don't have any other > built-in > > > > objects (tables, views) except functions. > > > > > > > > Cheers, > > > > Bowen > > > > > > > > > > > > > > > -- > Xuefu Zhang > > "In Honey We Trust!" >
to be contributer
Hi, I want to contribute to Apache Flink. Would you please give me the contributor permission? My JIRA ID is frank wang
Re: Add control mode for flink
I'm big +1 for this feature. 1. Limit the input qps. 2. Change log level for debug. in my team, the two examples above are needed JING ZHANG 于2021年6月8日周二 上午11:18写道： > Thanks Jiangang for bringing this up. > As mentioned in Jiangang's email, `dynamic configuration framework` > provides many useful functions in Kuaishou, because it could update job > behavior without relaunching the job. The functions are very popular in > Kuaishou, we also see similar demands in maillist [1]. > > I'm big +1 for this feature. > > Thanks Xintong and Yun for deep thoughts about the issue. I like the idea > about introducing control mode in Flink. > It takes the original issue a big step closer to essence which also > provides the possibility for more fantastic features as mentioned in > Xintong and Jark's response. > Based on the idea, there are at least two milestones to achieve the goals > which were proposed by Jiangang: > (1) Build a common control flow framework in Flink. > It focuses on control flow propagation. And, how to integrate the > common control flow framework with existing mechanisms. > (2) Builds a dynamic configuration framework which is exposed to users > directly. > We could see dynamic configuration framework is a top application on > the underlying control flow framework. > It focuses on the Public API which receives configuration updating > requests from users. Besides, it is necessary to introduce an API > protection mechanism to avoid job performance degradation caused by too > many control events. > > I suggest splitting the whole design into two after we reach a consensus > on whether to introduce this feature because these two sub-topic all need > careful design. > > > [ > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Dynamic-configuration-of-Flink-checkpoint-interval-td44059.html > ] > > Best regards, > JING ZHANG > > 刘建刚 于2021年6月8日周二 上午10:01写道： > >> Thanks Xintong Song for the detailed supplement. Since flink is >> long-running, it is similar to many services. So interacting with it or >> controlling it is a common desire. This was our initial thought when >> implementing the feature. In our inner flink, many configs used in yaml can >> be adjusted by dynamic to avoid restarting the job, for examples as follow: >> >>1. Limit the input qps. >>2. Degrade the job by sampling and so on. >>3. Reset kafka offset in certain cases. >>4. Stop checkpoint in certain cases. >>5. Control the history consuming. >>6. Change log level for debug. >> >> >> After deep discussion, we realize that a common control flow >> will benefit both users and developers. Dynamic config is just one of the >> use cases. For the concrete design and implementation, it relates with many >> components, like jobmaster, network channel, operators and so on, which >> needs deeper consideration and design. >> >> Xintong Song [via Apache Flink User Mailing List archive.] < >> ml+s2336050n44245...@n4.nabble.com> 于2021年6月7日周一 下午2:52写道： >> >>> Thanks Jiangang for bringing this up, and Steven & Peter for the >>> feedback. >>> >>> I was part of the preliminary offline discussions before this proposal >>> went public. So maybe I can help clarify things a bit. >>> >>> In short, despite the phrase "control mode" might be a bit misleading, >>> what we truly want to do from my side is to make the concept of "control >>> flow" explicit and expose it to users. >>> >>> ## Background >>> Jiangang & his colleagues at Kuaishou maintain an internal version of >>> Flink. One of their custom features is allowing dynamically changing >>> operator behaviors via the REST APIs. He's willing to contribute this >>> feature to the community, and came to Yun Gao and me for suggestions. After >>> discussion, we feel that the underlying question to be answered is how do >>> we model the control flow in Flink. Dynamically controlling jobs via REST >>> API can be one of the features built on top of the control flow, and there >>> could be others. >>> >>> ## Control flow >>> Control flow refers to the communication channels for sending >>> events/signals to/between tasks/operators, that changes Flink's behavior in >>> a way that may or may not affect the computation logic. Typical control >>> events/signals Flink currently has are watermarks and checkpoint barriers. >>> >>> In general, for modeling control flow, the following questions should be >>> considered. >>> 1. Who (which component) is responsible for generating the control >>> messages? >>> 2. Who (which component) is responsible for reacting to the messages. >>> 3. How do the messages propagate? >>> 4. When it comes to affecting the computation logics, how should the >>> control flow work together with the exact-once consistency. >>> >>> 1) & 2) may vary depending on the use cases, while 3) & 4) probably >>> share many things in common. A unified control flow model would help >>> deduplicate the common logics, allowing us to focus on the u
