cc user ML to get more attention, since the plugin will be used by Flink application developers.
Best regards, Jing On Mon, May 22, 2023 at 3:32 PM Jing Ge <j...@ververica.com> wrote: > Hi Emre, > > Thanks for clarifying it. Afaiac, it is a quite interesting proposal, > especially for Flink job developers who are heavily using the Datastream > API. Publishing the plugin in your Github would be a feasible way for the > first move. As I mentioned previously, in order to help the community > understand the plugin, you might want to describe all those attractive > features you mentioned in e.g. the readme.md with more technical details. I > personally was wondering how those connector compatibility rules will be > defined and maintained, given that almost all connectors have been > externalized. > > Best regards, > Jing > > On Mon, May 22, 2023 at 11:24 AM Kartoglu, Emre > <kar...@amazon.co.uk.invalid> wrote: > >> Hi Jing, >> >> The proposed plugin would be used by Flink application developers, when >> they are writing their Flink job. It would trigger during >> compilation/packaging and would look for known incompatibilities, bad >> practices, or bugs. >> For instance one cause of frustration for our customers is connector >> incompatibilities (specifically Kafka and Kinesis) with certain Flink >> versions. This plugin would be a quick way to update a list of known >> incompatibilities, bugs, bad practices, so customers get errors during >> compilation/packaging and not after they've deployed their Flink job. >> >> From what you're saying, the FLIP route might not be the best way to go. >> We might publish this plugin in our own GitHub namespace/group first, and >> then get community acknowledgement/support for it. I believe working with >> the Flink community on this is key as we'd need their support/opinion to do >> this the right way and reach more Flink users. >> >> Thanks >> Emre >> >> On 21/05/2023, 16:48, "Jing Ge" <j...@ververica.com.inva <mailto: >> j...@ververica.com.inva>LID> wrote: >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >> >> >> >> >> >> >> Hi Emre, >> >> >> Thanks for your proposal. It looks very interesting! Please pay attention >> that most connectors have been externalized. Will your proposed plug be >> used for building Flink Connectors or Flink itself? Furthermore, it would >> be great if you could elaborate features wrt best practices so that we >> could understand how the plugin will help us. >> >> >> Afaik, FLIP is recommended for improvement ideas that will change public >> APIs. I am not sure if a new maven plugin belongs to it. >> >> >> Best regards, >> Jing >> >> >> On Tue, May 16, 2023 at 11:29 AM Kartoglu, Emre <kar...@amazon.co.uk.inva >> <mailto:kar...@amazon.co.uk.inva>lid> >> wrote: >> >> >> > Hello all, >> > >> > Myself and 2 colleagues developed a Maven plugin (no support for Gradle >> or >> > other build tools yet) that we use internally to detect potential >> issues in >> > Flink apps at compilation/packaging stage: >> > >> > >> > * Known connector version incompatibilities – so far covering Kafka >> > and Kinesis >> > * Best practices e.g. setting operator IDs >> > >> > We’d like to make this open-source. Ideally with the Flink community’s >> > support/mention of it on the Flink website, so more people use it. >> > >> > Going forward, I believe we have at least the following options: >> > >> > * Get community support: Create a FLIP to discuss where the plugin >> > should live, what kind of problems it should detect etc. >> > * We still open-source it but without the community support (if the >> > community has objections to officially supporting it for instance). >> > >> > Just wanted to gauge the feeling/thoughts towards this tool from the >> > community before going ahead. >> > >> > Thanks, >> > Emre >> > >> > >> >> >> >>
