Thanks for the guidance Robert. I appreciate the prompt response and will share the pull requests for the ADLS support [2] next week.
Regarding the additional, I would like to contribute them to the main codebase [1] if possible I initially thought maybe it is better to start outside the core codebase but I think the adoption would be better if we have documentation on the core Flink documentation and reduce the level of effort necessary to integrate it for users. We will take time to make sure the docs and tests for the connectors are solid and then we can bring them one at a time into the core code base. [1] https://github.com/apache/flink/tree/master/flink-connectors [2] https://issues.apache.org/jira/browse/FLINK-18562 On Thu, Jul 23, 2020 at 3:41 AM Robert Metzger <rmetz...@apache.org> wrote: > Hi Israel, > > thanks a lot for reaching out! I'm very excited about your efforts to bring > additional Azure support into Flink. > There are ~50 threads on the user@ mailing list mentioning Azure -- that's > good evidence that our users use Flink in Azure. > > Regarding your questions: > > Do I need to create a FLIP [2] in order to make these changes to bring the > > new capabilities or the individual JIRA issues are sufficient? > > > For the two DataLake FS tickets, I don't see the need for a FLIP. > > I am thinking about targeting Flink versions 1.10 through 1.12 > > For new connectors like this, how many versions can/should this be > > integrated into? > > > We generally don't add new features to old releases (unless there's a very > good reason to backport the feature). Therefore, the new integrations will > all go into the next major Flink release (in this case probably Flink 1.12 > for the first tickets). > > Are there any upcoming changes to supported Java and Scala versions that I > > need to be aware of? > > > No, I'm not aware of any upcoming changes. Java 8 and Java 11 are the two > versions we test against. > > My goal is to add the first two features [FLINK-18562] and [FLINK-18568] to > > the existing file system capabilities [1] and then have the other > > connectors FLINK-1856[3-7] exist as standalone plugins. > > > I like the order in which you approach the tickets. I assigned you to the > first ticket and commented on the second one. I'm also willing to help > review your pull requests. > > What do you mean by "standalone plugins" in the context of connectors? > Would you like to contribute these connectors to the main Flink codebase, > or maintain them outside Flink but list them in flink-packages.org? > > Best, > Robert > > > On Wed, Jul 22, 2020 at 10:43 AM Israel Ekpo <israele...@gmail.com> wrote: > > > I have opened the following issues to track new efforts to bring > additional > > Azure Support in the following areas to the connectors ecosystem. > > > > My goal is to add the first two features [FLINK-18562] and [FLINK-18568] > to > > the existing file system capabilities [1] and then have the other > > connectors FLINK-1856[3-7] exist as standalone plugins. > > > > As more users adopt the additional connectors, we could incrementally > bring > > them into the core code base if necessary with sufficient support. > > > > I am new to the process so that I have a few questions: > > > > Do I need to create a FLIP [2] in order to make these changes to bring > the > > new capabilities or the individual JIRA issues are sufficient? > > > > I am thinking about targeting Flink versions 1.10 through 1.12 > > For new connectors like this, how many versions can/should this be > > integrated into? > > > > Are there any upcoming changes to supported Java and Scala versions that > I > > need to be aware of? > > > > Any ideas or suggestions you have would be great. > > > > Below is a summary of the JIRA issues that were created to track the > effort > > > > Add Support for Azure Data Lake Store Gen 2 in Flink File System > > https://issues.apache.org/jira/browse/FLINK-18562 > > > > Add Support for Azure Data Lake Store Gen 2 in Streaming File Sink > > https://issues.apache.org/jira/browse/FLINK-18568 > > > > Add Support for Azure Cosmos DB DataStream Connector > > https://issues.apache.org/jira/browse/FLINK-18563 > > > > Add Support for Azure Event Hub DataStream Connector > > https://issues.apache.org/jira/browse/FLINK-18564 > > > > Add Support for Azure Event Grid DataStream Connector > > https://issues.apache.org/jira/browse/FLINK-18565 > > > > Add Support for Azure Cognitive Search DataStream Connector > > https://issues.apache.org/jira/browse/FLINK-18566 > > > > Add Support for Azure Cognitive Search Table & SQL Connector > > https://issues.apache.org/jira/browse/FLINK-18567 > > > > > > [1] https://github.com/apache/flink/tree/master/flink-filesystems > > [2] > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > >
