+1 for Vasia’s suggestion. From a long-term perspective, the site like Spark Packages [1] would be helpful to manage external contribution.
[1] http://spark-packages.org > On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <mj...@apache.org> wrote: > > Thanks for the feedback. > > I think, the repository does not need to build on a single Flink > release. From my point of view, there should be a single parent module > that contains *independent modules* for each extension/library (there > should be no "cross dependencies" between the modules and each module > can specify the flink dependencies it needs by itself). This make is > most flexible. And if a library works on an old release, it might just > stay there as is. If a library changes (due to Flink changes), it might > just be contained multiple times for different Flink releases. > > Each module should provide a short doc (README) that shows how to use an > integrate it with Flink. Thus, the responsibility goes to the > contributor to maintain the library. If it breaks and is not maintained > any further, we can simple remove it. > > I agree, that the community might not be able to maintain those > extension/libraries right now. I would put the responsibility (more or > less completely) on the contributor and delete project that do not fix > any more. > > @Vasia: a link to a library could be included in the README. If anybody > only wants to share a library but not contribute code, the parent README > could contain a list of additional links. > > > -Matthias > > > On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote: >> How about, for now, we simply create a page where we gather links/short >> descriptions of all these contributions >> and let the maintenance and dependency management to the tool/library >> creators? >> This way we will at least have these contributions in one place and link to >> them somewhere from the website. >> >> -Vasia. >> >> On 8 October 2015 at 12:06, Maximilian Michels <m...@apache.org> wrote: >> >>> Hi Matthias, >>> >>> Thanks for bringing up this idea. Actually, it has been discussed a >>> couple of times on the mailing list whether we should have a central >>> place for third-party extensions/contributions/libraries. This could >>> either be something package-based or, like you proposed, another >>> repository. >>> >>> An external place for contributions raises a couple of questions >>> >>> - Which version should the external contributions be based on? >>> - How do we make sure, the extensions are continuously updated? >>> (dedicated maintainers or automatic compatibility checks) >>> - How do we easily plug-in the external modules into Flink? >>> >>> In the long term, we really need a solution for these questions. The >>> code base of Flink is growing and more and more packages go to >>> flink-contrib/flink-staging. I would find something packaged-based >>> better than a repository. Quite frankly, momentarily, I think >>> developing such a plugin system is out of scope for most Flink >>> developers. At the current pace of Flink development, collecting these >>> contributions externally without properly maintaining them, doesn't >>> make much sense to me. >>> >>> Cheers, >>> Max >>> >>> >>> >>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <mj...@apache.org> wrote: >>>> >>>> Hi, >>>> >>>> many people are building quite exiting stuff on top of Flink. It is hard >>>> to keep an good overview on what stuff is available and what not. What >>>> do you think about starting a second git repository "flink-external" >>>> that collects all those code? >>>> >>>> The ideas would be to collect stuff in a central point, such that people >>>> can access it easily and get an overview what is already available (this >>>> might also avoid duplicate development). It might also be a good point >>>> to show common patterns. In order to collect as much as possible, the >>>> contributing requirement (with respect to testing etc) could be lower >>>> than for Flink itself. >>>> >>>> For example, I recently started a small flink-clojure module with a >>>> simple word-count example to answer a question on SO. Including this in >>>> Flink would not be appropriate. However, for a flink-external repro it >>>> might be nice to have. >>>> >>>> What do you think about it? >>>> >>>> >>>> -Matthias >>>> >>> >> > Regards, Chiwan Park
