I had small chat with Till about how to help manage Flink ML Libraries contributions, which use Flink ML as dependencies.
I suppose if this approached is the way to go for Flink connectors, could we do the same for Flink ML libraries? - Henry On Fri, Dec 11, 2015 at 1:33 AM, Maximilian Michels <m...@apache.org> wrote: > We should have release branches which are in sync with the release > branches in the main repository. Connectors should be compatible > across minor releases. The versioning could be of the form > "flinkversion-connectorversion", e.g. 0.10-connector1. > >>The pluggable architecture is great! (why Don't we call it Flink plugins? my >>2 cents) > > We can still change the name. IMHO "Plugins" is a bit broad since this > is currently only targeted at the connectors included in Flink. > >>Would we loose test coverage by putting the connectors into a separate >>repository/maven project? > > Not necessarily. Two possibilities: > > 1) Run a connectors test jar during the normal Travis tests in the > main repository > 2) Trigger a Travis test run at the connectors repository upon a > commit into the main repository > > Option 1 seems like the better alternative because we would > immediately see if a change breaks the connectors. Of course, if > changes are made in the connectors repository, we would also run tests > with the main repository. > > On Thu, Dec 10, 2015 at 11:00 PM, jun aoki <ja...@apache.org> wrote: >> The pluggable architecture is great! (why Don't we call it Flink plugins? >> my 2 cents) >> It will be nice if we come up with an idea of what directory structure >> should look like before start dumping connectors (plugins). >> Also wonder what to do with versioning. >> At some point, for example, Twitter v1 connector could be compatible with >> flink 0.10 but Flume v2 connector could be compatible with trunk, etc. It >> should be taken consideration either in the directory structure or >> branching strategy. >> >> On Thu, Dec 10, 2015 at 7:12 AM, Aljoscha Krettek <aljos...@apache.org> >> wrote: >> >>> We would need to have a stable interface between the connectors and flink >>> and have very good checks that ensure that we don’t inadvertently break >>> things. >>> >>> > On 10 Dec 2015, at 15:45, Fabian Hueske <fhue...@gmail.com> wrote: >>> > >>> > Sounds like a good idea to me. >>> > >>> > +1 >>> > >>> > Fabian >>> > >>> > 2015-12-10 15:31 GMT+01:00 Maximilian Michels <m...@apache.org>: >>> > >>> >> Hi squirrels, >>> >> >>> >> By this time, we have numerous connectors which let you insert data >>> >> into Flink or output data from Flink. >>> >> >>> >> On the streaming side we have >>> >> >>> >> - RollingSink >>> >> - Flume >>> >> - Kafka >>> >> - Nifi >>> >> - RabbitMQ >>> >> - Twitter >>> >> >>> >> On the batch side we have >>> >> >>> >> - Avro >>> >> - Hadoop compatibility >>> >> - HBase >>> >> - HCatalog >>> >> - JDBC >>> >> >>> >> >>> >> Many times we would have liked to release updates to the connectors or >>> >> even create new ones in between Flink releases. This is currently not >>> >> possible because the connectors are part of the main repository. >>> >> >>> >> Therefore, I have created a new repository at >>> >> https://git-wip-us.apache.org/repos/asf/flink-connectors.git. The idea >>> >> is to externalize the connectors to this repository. We can then >>> >> update and release them independently of the main Flink repository. I >>> >> think this will give us more flexibility in the development process. >>> >> >>> >> What do you think about this idea? >>> >> >>> >> Cheers, >>> >> Max >>> >> >>> >>> >> >> >> -- >> -jun
