Hello Ilya, Thank you for your email. Yes, before we remove any integration we will move them to another repository and release.
We are thinking we will start with Pub/Sub Streamer implementation and continue to move other integration as mentioned in the following doc. https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations Regards, Saikat On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev <ilya.kasnach...@gmail.com> wrote: > Hello! > > My take on this: we want to offload some of integrations, but please commit > and release them to another project first, and only then we will decide to > remove them from our repository. > > Please also make sure they are available with same artifact name and API. > If this has to be changed, let's wait for AI 3.0 > > Regards, > -- > Ilya Kasnacheev > > > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra <saikat.mai...@gmail.com>: > > > Hello Emmanouil, > > > > As discussed earlier, I discussed with Apache Bahir community and they > are > > interested to have Apache Ignite extensions as part of Apache Bahir > > project. > > > > Can you please share Pub/Sub streamer implementation details with Apache > > Bahir community and request for feedback. > > > > https://bahir.apache.org/ > > > > I have also requested for contributor access in Jira for Apache Bahir > > project so that I can create issues and assign to myself. > > > > I can help with code reviews as well. > > > > Here is the email thread for reference > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > > > > Regards, > > Saikat > > > > > > > > > > > > > > > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <saikat.mai...@gmail.com> > > wrote: > > > > > Hello, > > > > > > As discussed I have created following discussion threads on our > migration > > > proposals for Apache Ignite extensions: > > > > > > > > > > > > http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html > > > > > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > > > > > > I will share update as I hear more information. > > > > > > Regards, > > > Saikat > > > > > > > > > > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <saikat.mai...@gmail.com > > > > > wrote: > > > > > >> Hello Denis, > > >> > > >> Sure, sounds good. I will create a separate discussion thread to get > > >> community feedback on whether we > > >> should join the Bahir or kick-off "Ignite Extensions" project. > > >> > > >> Regards, > > >> Saikat > > >> > > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <dma...@apache.org> > wrote: > > >> > > >>> Folks, > > >>> > > >>> The concept of Apache Bahir is precisely what we need! Saikat, thanks > > for > > >>> doing the research. Why I believe the * idea* fits us well: > > >>> > > >>> - All integrations can be stored in separate Github repositories > and > > >>> have their dev lifecycles. We've not obliged to couple all the > > >>> integrations > > >>> in a single repo. For instance, Spark can be located in a > dedicated > > >>> repo > > >>> while streaming integrations might be in a single one. It's up to > > us. > > >>> - All the repositories and integrations belong to ASF. We're not > > >>> dumping > > >>> them on Github but rather keep supporting and developing in > > >>> accordance with > > >>> ASF vision and practices. > > >>> - There is a way to reward contributors via committership and PMC > > >>> membership. You won't get this if a project is just one of the > > >>> millions of > > >>> Github projects. > > >>> > > >>> Saikat, as Ignite PMC member, could you please kick-off a separate > > >>> dev-list > > >>> discussion to involve more community members and referring to this > > >>> thread? > > >>> I think the primary question the community needs to answer whether we > > >>> should join the Bahir or kick-off "Ignite Extensions" project? > > >>> > > >>> - > > >>> Denis > > >>> > > >>> > > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev < > > zaleslaw....@gmail.com> > > >>> wrote: > > >>> > > >>> > Maybe we could move all our Streaming Integrations there, but what > is > > >>> about > > >>> > maintaining and committer permissions to the new repositories? > > >>> > I see the list of the committers and PMC members there > > >>> > https://bahir.apache.org/community-members/ > > >>> > > > >>> > Could somebody from Ignite community be added to this list as a > > >>> > PMC/committer to maintain Ignite integrations? > > >>> > If yes, I'd happy to join this project and collaborate with these > > guys > > >>> > together > > >>> > If no, and we should start from the zero level with external PRs - > > >>> hmmm, > > >>> > it's better to have our own external repository with ApacheBahirr > > >>> ideology. > > >>> > > > >>> > Also, I agree, that all connectors could be moved there (in > > >>> ApacheBahirr > > >>> > like repository), but not all modules from the page > > >>> > > > >>> > > > >>> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > >>> > because > > >>> > they use different parts of core modules and could be developed > with > > >>> > different velocity. > > >>> > > > >>> > In reality, before creation of new repositories we need wide > > discussion > > >>> > based on the document mentioned above. > > >>> > > > >>> > P.S. Of course, we could experiment in the next release 2.8 with > one > > >>> or two > > >>> > integrations like twitter or ZeroMQ > > >>> > Also, I'd like Denis Magda idea for the separate repositories in > > Apache > > >>> > Github to maintain them by anybody. > > >>> > > > >>> > > > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra < > saikat.mai...@gmail.com > > >: > > >>> > > > >>> > > Hi Denis, Emmanouil > > >>> > > > > >>> > > Thank you for your email. I think creating a separate integration > > >>> project > > >>> > > in github and also proposing it as an apache incubator project > will > > >>> be a > > >>> > > good move. The other separate integration modules > > >>> > > > > >>> > > > > >>> > > > >>> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > >>> > > can > > >>> > > be moved to this new apache incubator project. > > >>> > > > > >>> > > There are similar integration available for Flink and Spark in > > Apache > > >>> > Bahir > > >>> > > https://bahir.apache.org/ > > >>> > > > > >>> > > Do you think we should reach out to Apache Bahir community with a > > >>> > proposal > > >>> > > or we should plan to create a separate Apache Incubator project? > > >>> > > > > >>> > > Regards, > > >>> > > Saikat > > >>> > > > > >>> > > > > >>> > > > > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras < > > >>> > > gkatzio...@gmail.com> > > >>> > > wrote: > > >>> > > > > >>> > > > Hi all! > > >>> > > > > > >>> > > > Based on the discussions most probably this is going to be on > > >>> another > > >>> > > > GitHub repo. Therefore I will prepare a standalone project with > > the > > >>> > > Pub/Sub > > >>> > > > module and once the repository is created a pull request shall > be > > >>> > issued. > > >>> > > > If there is anything else I can do in order to facilitate this > > >>> please > > >>> > let > > >>> > > > me know. > > >>> > > > > > >>> > > > If a ticket is created for this issue, I shall be able to > create > > a > > >>> pull > > >>> > > > request and thus a build will take place on ignite's CI. > > >>> > > > If everyone is aligned with adding a Pub/Sub implementation > may I > > >>> > create > > >>> > > a > > >>> > > > ticket on JIRA? > > >>> > > > > > >>> > > > My next question has to do on how Ignite configures the build > > jobs > > >>> in > > >>> > > Team > > >>> > > > City. The unit tests for my implementation need a local Pub/Sub > > >>> server > > >>> > up > > >>> > > > and running (they provide one for testing purposes). Will this > > take > > >>> > > effect > > >>> > > > in a form of a build script inside my implementation or is it > > >>> something > > >>> > > > that should be configured on Team City? > > >>> > > > > > >>> > > > Kind regards, > > >>> > > > Emmanouil > > >>> > > > > > >>> > > > > > >>> > > > *Emmanouil Gkatziouras* > > >>> > > > https://egkatzioura.com/ | > > >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > >>> > > > https://github.com/gkatzioura > > >>> > > > > > >>> > > > > > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dma...@apache.org> > > >>> wrote: > > >>> > > > > > >>> > > > > Emmanouil, Saikat, > > >>> > > > > > > >>> > > > > After contemplating on this, let me propose another way with > > this > > >>> > > > > particular integration that is aligned with our > modularization > > >>> > > endeavor. > > >>> > > > > The modularization [1] defines the Ignite core, Ignite > modules, > > >>> and > > >>> > > > > independent integrations. The first two are to be located in > > >>> Ignite > > >>> > > > > repositories and to be tested/updated by the community for > > every > > >>> > > release. > > >>> > > > > The modules are a subset of existing integrations that are > > >>> adopted > > >>> > > widely > > >>> > > > > and require community attention. > > >>> > > > > > > >>> > > > > As for the independent integrations [1], those are supposed > to > > be > > >>> > > located > > >>> > > > > in a separate GitHub repo. I would advise us to start > > >>> implementing > > >>> > this > > >>> > > > > concept with the Pub/Sub integration. Even though it will be > > >>> located > > >>> > > in a > > >>> > > > > separate Github repo, it will be listed among officially > > >>> available > > >>> > > > > integrations on the Ignite website featuring Emmanouil as a > > >>> > > contributor. > > >>> > > > > > > >>> > > > > What do you think? > > >>> > > > > > > >>> > > > > > > >>> > > > > [1] > > >>> > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > - > > >>> > > > > Denis > > >>> > > > > > > >>> > > > > > > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra < > > >>> > saikat.mai...@gmail.com > > >>> > > > > > >>> > > > > wrote: > > >>> > > > > > > >>> > > > > > Hi Emmanouil, > > >>> > > > > > > > >>> > > > > > The changes looks good to me. I wanted to check if you are > > also > > >>> > able > > >>> > > to > > >>> > > > > > validate that the once data is added to Ignite cluster you > > are > > >>> also > > >>> > > > able > > >>> > > > > to > > >>> > > > > > access it using another Ignite client or using rest > > endpoints. > > >>> > > > > > > > >>> > > > > > We use Teamcity for CI process, the details are as follows > > >>> > > > > > > > >>> > > > > > Trigger validation of those test suites that have been > > >>> affected by > > >>> > > your > > >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: < > > >>> > > > > http://204.14.53.153/ > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > - Locate a test suite you have to check, press button > > named > > >>> > "..." > > >>> > > > that > > >>> > > > > > is located on the left of "Run" button. "Run custom > build" > > >>> > dialog > > >>> > > > will > > >>> > > > > > appear; > > >>> > > > > > - Go to "Changes" tab and choose > > >>> > "pull/<pull-request-number>/head" > > >>> > > > in > > >>> > > > > > "Build branch" dropdown list; > > >>> > > > > > - Press "Run build" button and monitor tests results. > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > @Roman rsht...@yahoo.com - can you also please take a > look? > > >>> > > > > > > > >>> > > > > > Regards, > > >>> > > > > > Saikat > > >>> > > > > > > > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras < > > >>> > > > > > gkatzio...@gmail.com> > > >>> > > > > > wrote: > > >>> > > > > > > > >>> > > > > > > Hi @Saikat Maitra > > >>> > > > > > > > > >>> > > > > > > I have my implementation on an Ignite fork on my personal > > >>> GitHub > > >>> > > > > account > > >>> > > > > > > [1] > > >>> > > > > > > It is based on the Kafka implementation which is of a > > polling > > >>> > > nature. > > >>> > > > > > > GCP does not provide a service like Kafka. Instead it > > >>> provides us > > >>> > > > with > > >>> > > > > > > Pub/Sub which is just like Kafka. > > >>> > > > > > > Therefore using the Kafka streamer as a guideline made > more > > >>> sense > > >>> > > to > > >>> > > > > me. > > >>> > > > > > > > > >>> > > > > > > Apart from the implementation I have also added a unit > > test. > > >>> > > > > > > It definitely needs some polishing and a double check on > my > > >>> side > > >>> > to > > >>> > > > > make > > >>> > > > > > > the review smooth. > > >>> > > > > > > For the tests a local Pub/Sub [2] server has to be setup > > >>> however > > >>> > I > > >>> > > am > > >>> > > > > not > > >>> > > > > > > sure which is the best way to do so with regards to your > > >>> CI/CD. > > >>> > > > > > > > > >>> > > > > > > [1] > > >>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > > >>> > > > > > > [2] > > >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ > > >>> > > > > > > > > >>> > > > > > > *Emmanouil Gkatziouras* > > >>> > > > > > > https://egkatzioura.com/ | > > >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > >>> > > > > > > https://github.com/gkatzioura > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra < > > >>> > > saikat.mai...@gmail.com > > >>> > > > > > > >>> > > > > > > wrote: > > >>> > > > > > > > > >>> > > > > > > > Hello Emmanouil, > > >>> > > > > > > > > > >>> > > > > > > > Can you please share more info about the > implementation? > > >>> > > > > > > > > > >>> > > > > > > > We have similar implementation for data streamers with > > >>> Kafka > > >>> > and > > >>> > > > > > RocketMQ > > >>> > > > > > > > > > >>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer > > >>> > > > > > > > > > >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer > > >>> > > > > > > > > > >>> > > > > > > > Please review and share if you have any questions. > > >>> > > > > > > > > > >>> > > > > > > > Regards, > > >>> > > > > > > > Saikat > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras < > > >>> > > > > > > > gkatzio...@gmail.com> > > >>> > > > > > > > wrote: > > >>> > > > > > > > > > >>> > > > > > > > > Hi all, > > >>> > > > > > > > > > > >>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub > [1] > > >>> > > > > > > > > Yes it is indeed cloud specific. It is the messaging > > >>> service > > >>> > > > > provided > > >>> > > > > > > by > > >>> > > > > > > > > GCP. > > >>> > > > > > > > > I have made the integration with it, but I haven't > > >>> published > > >>> > it > > >>> > > > yet > > >>> > > > > > (I > > >>> > > > > > > am > > >>> > > > > > > > > waiting for the discussion/ticket and proceed with > the > > >>> > > > > contribution). > > >>> > > > > > > > > > > >>> > > > > > > > > Kind regards, > > >>> > > > > > > > > Emmanouil > > >>> > > > > > > > > > > >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview > > >>> > > > > > > > > > > >>> > > > > > > > > *Emmanouil Gkatziouras* > > >>> > > > > > > > > https://egkatzioura.com/ | > > >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > >>> > > > > > > > > https://github.com/gkatzioura > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda < > > >>> dma...@apache.org > > >>> > > > > >>> > > > > wrote: > > >>> > > > > > > > > > > >>> > > > > > > > > > Hello Emmanouil, > > >>> > > > > > > > > > > > >>> > > > > > > > > > Are you referring to any specific implementation > > >>> tailored > > >>> > > for a > > >>> > > > > > > cloud? > > >>> > > > > > > > If > > >>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is > > there > > >>> any > > >>> > > > Github > > >>> > > > > > > > source > > >>> > > > > > > > > > code? > > >>> > > > > > > > > > > > >>> > > > > > > > > > Here is a list of existing integrations with > > streaming > > >>> and > > >>> > > > > pub/sub > > >>> > > > > > > > > > frameworks: > > >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview > > >>> > > > > > > > > > > > >>> > > > > > > > > > with the source code in Ignite master: > > >>> > > > > > > > > > > https://github.com/apache/ignite/tree/master/modules > > >>> > > > > > > > > > > > >>> > > > > > > > > > Check out those resources to see what are the most > > >>> suitable > > >>> > > > > Ignite > > >>> > > > > > > APIs > > >>> > > > > > > > > for > > >>> > > > > > > > > > such integrations. > > >>> > > > > > > > > > > > >>> > > > > > > > > > @Saikat Maitra <saikat.mai...@gmail.com> , @Roman > < > > >>> > > > > > rsht...@yahoo.com > > >>> > > > > > > > > > >>> > > > > > > > , > > >>> > > > > > > > > as > > >>> > > > > > > > > > streaming technologies contributors/maintainers, > > could > > >>> you > > >>> > > > please > > >>> > > > > > > join > > >>> > > > > > > > > this > > >>> > > > > > > > > > thread? > > >>> > > > > > > > > > > > >>> > > > > > > > > > - > > >>> > > > > > > > > > Denis > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil > > Gkatziouras < > > >>> > > > > > > > > > gkatzio...@gmail.com> > > >>> > > > > > > > > > wrote: > > >>> > > > > > > > > > > > >>> > > > > > > > > > > Hello everyone, > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > I started using Ignite lately. Part of my work > > >>> involves > > >>> > > > > Pub/Sub, > > >>> > > > > > > > thus I > > >>> > > > > > > > > > > created a pub/sub streamer. > > >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka. > > >>> > > > > > > > > > > Here's how the streamer works. Every node creates > > an > > >>> > > instance > > >>> > > > > of > > >>> > > > > > a > > >>> > > > > > > > > > Pub/Sub > > >>> > > > > > > > > > > client. > > >>> > > > > > > > > > > The clients subscribe to the same topic using the > > >>> same > > >>> > > > > > subscription > > >>> > > > > > > > > name. > > >>> > > > > > > > > > > A message is being sent and one of the nodes will > > >>> receive > > >>> > > the > > >>> > > > > > > > message, > > >>> > > > > > > > > > > process it and then forward it to Ignite using > the > > >>> > > extractor > > >>> > > > > > > > provided. > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > Any thoughts? > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > *Emmanouil Gkatziouras* > > >>> > > > > > > > > > > https://egkatzioura.com/ | > > >>> > > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > >>> > > > > > > > > > > https://github.com/gkatzioura > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >> > > >
