Hi all! This is an update. The integration is tested with mocks based on the original Ignite repo [1]. I am currently waiting for the first commits on the new integration project to take place in order to create my pull request accordingly to the initial commits. In order to simulate the Pub/Sub server in a test I used a test server implementation [2]. Unfortunately they don't have this util uploaded in a maven repo, thus it needs to be hosted on a maven repo (Apache License 2.0). Is there a repo for utilities that Ignite uses for the test/build? I have also created a ticket [3]
Kind regards [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub [2] https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator [3] https://issues.apache.org/jira/browse/IGNITE-12262 *Emmanouil Gkatziouras* https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/ https://github.com/gkatzioura On Fri, 25 Oct 2019 at 03:35, Saikat Maitra <saikat.mai...@gmail.com> wrote: > Hello Ilya, > > Thank you for your email. Yes, before we remove any integration we will > move them to another repository and release. > > We are thinking we will start with Pub/Sub Streamer implementation and > continue to move other integration as mentioned in the following doc. > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > Regards, > Saikat > > > On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev < > ilya.kasnach...@gmail.com> > wrote: > > > Hello! > > > > My take on this: we want to offload some of integrations, but please > commit > > and release them to another project first, and only then we will decide > to > > remove them from our repository. > > > > Please also make sure they are available with same artifact name and API. > > If this has to be changed, let's wait for AI 3.0 > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra <saikat.mai...@gmail.com>: > > > > > Hello Emmanouil, > > > > > > As discussed earlier, I discussed with Apache Bahir community and they > > are > > > interested to have Apache Ignite extensions as part of Apache Bahir > > > project. > > > > > > Can you please share Pub/Sub streamer implementation details with > Apache > > > Bahir community and request for feedback. > > > > > > https://bahir.apache.org/ > > > > > > I have also requested for contributor access in Jira for Apache Bahir > > > project so that I can create issues and assign to myself. > > > > > > I can help with code reviews as well. > > > > > > Here is the email thread for reference > > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > > > > > > Regards, > > > Saikat > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <saikat.mai...@gmail.com > > > > > wrote: > > > > > > > Hello, > > > > > > > > As discussed I have created following discussion threads on our > > migration > > > > proposals for Apache Ignite extensions: > > > > > > > > > > > > > > > > > > http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html > > > > > > > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > > > > > > > > I will share update as I hear more information. > > > > > > > > Regards, > > > > Saikat > > > > > > > > > > > > > > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra < > saikat.mai...@gmail.com > > > > > > > wrote: > > > > > > > >> Hello Denis, > > > >> > > > >> Sure, sounds good. I will create a separate discussion thread to get > > > >> community feedback on whether we > > > >> should join the Bahir or kick-off "Ignite Extensions" project. > > > >> > > > >> Regards, > > > >> Saikat > > > >> > > > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <dma...@apache.org> > > wrote: > > > >> > > > >>> Folks, > > > >>> > > > >>> The concept of Apache Bahir is precisely what we need! Saikat, > thanks > > > for > > > >>> doing the research. Why I believe the * idea* fits us well: > > > >>> > > > >>> - All integrations can be stored in separate Github repositories > > and > > > >>> have their dev lifecycles. We've not obliged to couple all the > > > >>> integrations > > > >>> in a single repo. For instance, Spark can be located in a > > dedicated > > > >>> repo > > > >>> while streaming integrations might be in a single one. It's up > to > > > us. > > > >>> - All the repositories and integrations belong to ASF. We're not > > > >>> dumping > > > >>> them on Github but rather keep supporting and developing in > > > >>> accordance with > > > >>> ASF vision and practices. > > > >>> - There is a way to reward contributors via committership and > PMC > > > >>> membership. You won't get this if a project is just one of the > > > >>> millions of > > > >>> Github projects. > > > >>> > > > >>> Saikat, as Ignite PMC member, could you please kick-off a separate > > > >>> dev-list > > > >>> discussion to involve more community members and referring to this > > > >>> thread? > > > >>> I think the primary question the community needs to answer whether > we > > > >>> should join the Bahir or kick-off "Ignite Extensions" project? > > > >>> > > > >>> - > > > >>> Denis > > > >>> > > > >>> > > > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev < > > > zaleslaw....@gmail.com> > > > >>> wrote: > > > >>> > > > >>> > Maybe we could move all our Streaming Integrations there, but > what > > is > > > >>> about > > > >>> > maintaining and committer permissions to the new repositories? > > > >>> > I see the list of the committers and PMC members there > > > >>> > https://bahir.apache.org/community-members/ > > > >>> > > > > >>> > Could somebody from Ignite community be added to this list as a > > > >>> > PMC/committer to maintain Ignite integrations? > > > >>> > If yes, I'd happy to join this project and collaborate with these > > > guys > > > >>> > together > > > >>> > If no, and we should start from the zero level with external PRs > - > > > >>> hmmm, > > > >>> > it's better to have our own external repository with ApacheBahirr > > > >>> ideology. > > > >>> > > > > >>> > Also, I agree, that all connectors could be moved there (in > > > >>> ApacheBahirr > > > >>> > like repository), but not all modules from the page > > > >>> > > > > >>> > > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > > >>> > because > > > >>> > they use different parts of core modules and could be developed > > with > > > >>> > different velocity. > > > >>> > > > > >>> > In reality, before creation of new repositories we need wide > > > discussion > > > >>> > based on the document mentioned above. > > > >>> > > > > >>> > P.S. Of course, we could experiment in the next release 2.8 with > > one > > > >>> or two > > > >>> > integrations like twitter or ZeroMQ > > > >>> > Also, I'd like Denis Magda idea for the separate repositories in > > > Apache > > > >>> > Github to maintain them by anybody. > > > >>> > > > > >>> > > > > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra < > > saikat.mai...@gmail.com > > > >: > > > >>> > > > > >>> > > Hi Denis, Emmanouil > > > >>> > > > > > >>> > > Thank you for your email. I think creating a separate > integration > > > >>> project > > > >>> > > in github and also proposing it as an apache incubator project > > will > > > >>> be a > > > >>> > > good move. The other separate integration modules > > > >>> > > > > > >>> > > > > > >>> > > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > > >>> > > can > > > >>> > > be moved to this new apache incubator project. > > > >>> > > > > > >>> > > There are similar integration available for Flink and Spark in > > > Apache > > > >>> > Bahir > > > >>> > > https://bahir.apache.org/ > > > >>> > > > > > >>> > > Do you think we should reach out to Apache Bahir community > with a > > > >>> > proposal > > > >>> > > or we should plan to create a separate Apache Incubator > project? > > > >>> > > > > > >>> > > Regards, > > > >>> > > Saikat > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras < > > > >>> > > gkatzio...@gmail.com> > > > >>> > > wrote: > > > >>> > > > > > >>> > > > Hi all! > > > >>> > > > > > > >>> > > > Based on the discussions most probably this is going to be on > > > >>> another > > > >>> > > > GitHub repo. Therefore I will prepare a standalone project > with > > > the > > > >>> > > Pub/Sub > > > >>> > > > module and once the repository is created a pull request > shall > > be > > > >>> > issued. > > > >>> > > > If there is anything else I can do in order to facilitate > this > > > >>> please > > > >>> > let > > > >>> > > > me know. > > > >>> > > > > > > >>> > > > If a ticket is created for this issue, I shall be able to > > create > > > a > > > >>> pull > > > >>> > > > request and thus a build will take place on ignite's CI. > > > >>> > > > If everyone is aligned with adding a Pub/Sub implementation > > may I > > > >>> > create > > > >>> > > a > > > >>> > > > ticket on JIRA? > > > >>> > > > > > > >>> > > > My next question has to do on how Ignite configures the build > > > jobs > > > >>> in > > > >>> > > Team > > > >>> > > > City. The unit tests for my implementation need a local > Pub/Sub > > > >>> server > > > >>> > up > > > >>> > > > and running (they provide one for testing purposes). Will > this > > > take > > > >>> > > effect > > > >>> > > > in a form of a build script inside my implementation or is it > > > >>> something > > > >>> > > > that should be configured on Team City? > > > >>> > > > > > > >>> > > > Kind regards, > > > >>> > > > Emmanouil > > > >>> > > > > > > >>> > > > > > > >>> > > > *Emmanouil Gkatziouras* > > > >>> > > > https://egkatzioura.com/ | > > > >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > >>> > > > https://github.com/gkatzioura > > > >>> > > > > > > >>> > > > > > > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dma...@apache.org > > > > > >>> wrote: > > > >>> > > > > > > >>> > > > > Emmanouil, Saikat, > > > >>> > > > > > > > >>> > > > > After contemplating on this, let me propose another way > with > > > this > > > >>> > > > > particular integration that is aligned with our > > modularization > > > >>> > > endeavor. > > > >>> > > > > The modularization [1] defines the Ignite core, Ignite > > modules, > > > >>> and > > > >>> > > > > independent integrations. The first two are to be located > in > > > >>> Ignite > > > >>> > > > > repositories and to be tested/updated by the community for > > > every > > > >>> > > release. > > > >>> > > > > The modules are a subset of existing integrations that are > > > >>> adopted > > > >>> > > widely > > > >>> > > > > and require community attention. > > > >>> > > > > > > > >>> > > > > As for the independent integrations [1], those are supposed > > to > > > be > > > >>> > > located > > > >>> > > > > in a separate GitHub repo. I would advise us to start > > > >>> implementing > > > >>> > this > > > >>> > > > > concept with the Pub/Sub integration. Even though it will > be > > > >>> located > > > >>> > > in a > > > >>> > > > > separate Github repo, it will be listed among officially > > > >>> available > > > >>> > > > > integrations on the Ignite website featuring Emmanouil as a > > > >>> > > contributor. > > > >>> > > > > > > > >>> > > > > What do you think? > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > [1] > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > - > > > >>> > > > > Denis > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra < > > > >>> > saikat.mai...@gmail.com > > > >>> > > > > > > >>> > > > > wrote: > > > >>> > > > > > > > >>> > > > > > Hi Emmanouil, > > > >>> > > > > > > > > >>> > > > > > The changes looks good to me. I wanted to check if you > are > > > also > > > >>> > able > > > >>> > > to > > > >>> > > > > > validate that the once data is added to Ignite cluster > you > > > are > > > >>> also > > > >>> > > > able > > > >>> > > > > to > > > >>> > > > > > access it using another Ignite client or using rest > > > endpoints. > > > >>> > > > > > > > > >>> > > > > > We use Teamcity for CI process, the details are as > follows > > > >>> > > > > > > > > >>> > > > > > Trigger validation of those test suites that have been > > > >>> affected by > > > >>> > > your > > > >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: < > > > >>> > > > > http://204.14.53.153/ > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > - Locate a test suite you have to check, press button > > > named > > > >>> > "..." > > > >>> > > > that > > > >>> > > > > > is located on the left of "Run" button. "Run custom > > build" > > > >>> > dialog > > > >>> > > > will > > > >>> > > > > > appear; > > > >>> > > > > > - Go to "Changes" tab and choose > > > >>> > "pull/<pull-request-number>/head" > > > >>> > > > in > > > >>> > > > > > "Build branch" dropdown list; > > > >>> > > > > > - Press "Run build" button and monitor tests results. > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > @Roman rsht...@yahoo.com - can you also please take a > > look? > > > >>> > > > > > > > > >>> > > > > > Regards, > > > >>> > > > > > Saikat > > > >>> > > > > > > > > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras < > > > >>> > > > > > gkatzio...@gmail.com> > > > >>> > > > > > wrote: > > > >>> > > > > > > > > >>> > > > > > > Hi @Saikat Maitra > > > >>> > > > > > > > > > >>> > > > > > > I have my implementation on an Ignite fork on my > personal > > > >>> GitHub > > > >>> > > > > account > > > >>> > > > > > > [1] > > > >>> > > > > > > It is based on the Kafka implementation which is of a > > > polling > > > >>> > > nature. > > > >>> > > > > > > GCP does not provide a service like Kafka. Instead it > > > >>> provides us > > > >>> > > > with > > > >>> > > > > > > Pub/Sub which is just like Kafka. > > > >>> > > > > > > Therefore using the Kafka streamer as a guideline made > > more > > > >>> sense > > > >>> > > to > > > >>> > > > > me. > > > >>> > > > > > > > > > >>> > > > > > > Apart from the implementation I have also added a unit > > > test. > > > >>> > > > > > > It definitely needs some polishing and a double check > on > > my > > > >>> side > > > >>> > to > > > >>> > > > > make > > > >>> > > > > > > the review smooth. > > > >>> > > > > > > For the tests a local Pub/Sub [2] server has to be > setup > > > >>> however > > > >>> > I > > > >>> > > am > > > >>> > > > > not > > > >>> > > > > > > sure which is the best way to do so with regards to > your > > > >>> CI/CD. > > > >>> > > > > > > > > > >>> > > > > > > [1] > > > >>> > > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > > > >>> > > > > > > [2] > > > >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ > > > >>> > > > > > > > > > >>> > > > > > > *Emmanouil Gkatziouras* > > > >>> > > > > > > https://egkatzioura.com/ | > > > >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > >>> > > > > > > https://github.com/gkatzioura > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra < > > > >>> > > saikat.mai...@gmail.com > > > >>> > > > > > > > >>> > > > > > > wrote: > > > >>> > > > > > > > > > >>> > > > > > > > Hello Emmanouil, > > > >>> > > > > > > > > > > >>> > > > > > > > Can you please share more info about the > > implementation? > > > >>> > > > > > > > > > > >>> > > > > > > > We have similar implementation for data streamers > with > > > >>> Kafka > > > >>> > and > > > >>> > > > > > RocketMQ > > > >>> > > > > > > > > > > >>> > > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer > > > >>> > > > > > > > > > > >>> > > > > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer > > > >>> > > > > > > > > > > >>> > > > > > > > Please review and share if you have any questions. > > > >>> > > > > > > > > > > >>> > > > > > > > Regards, > > > >>> > > > > > > > Saikat > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil > Gkatziouras < > > > >>> > > > > > > > gkatzio...@gmail.com> > > > >>> > > > > > > > wrote: > > > >>> > > > > > > > > > > >>> > > > > > > > > Hi all, > > > >>> > > > > > > > > > > > >>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub > > [1] > > > >>> > > > > > > > > Yes it is indeed cloud specific. It is the > messaging > > > >>> service > > > >>> > > > > provided > > > >>> > > > > > > by > > > >>> > > > > > > > > GCP. > > > >>> > > > > > > > > I have made the integration with it, but I haven't > > > >>> published > > > >>> > it > > > >>> > > > yet > > > >>> > > > > > (I > > > >>> > > > > > > am > > > >>> > > > > > > > > waiting for the discussion/ticket and proceed with > > the > > > >>> > > > > contribution). > > > >>> > > > > > > > > > > > >>> > > > > > > > > Kind regards, > > > >>> > > > > > > > > Emmanouil > > > >>> > > > > > > > > > > > >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview > > > >>> > > > > > > > > > > > >>> > > > > > > > > *Emmanouil Gkatziouras* > > > >>> > > > > > > > > https://egkatzioura.com/ | > > > >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > >>> > > > > > > > > https://github.com/gkatzioura > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda < > > > >>> dma...@apache.org > > > >>> > > > > > >>> > > > > wrote: > > > >>> > > > > > > > > > > > >>> > > > > > > > > > Hello Emmanouil, > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > Are you referring to any specific implementation > > > >>> tailored > > > >>> > > for a > > > >>> > > > > > > cloud? > > > >>> > > > > > > > If > > > >>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is > > > there > > > >>> any > > > >>> > > > Github > > > >>> > > > > > > > source > > > >>> > > > > > > > > > code? > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > Here is a list of existing integrations with > > > streaming > > > >>> and > > > >>> > > > > pub/sub > > > >>> > > > > > > > > > frameworks: > > > >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > with the source code in Ignite master: > > > >>> > > > > > > > > > > > https://github.com/apache/ignite/tree/master/modules > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > Check out those resources to see what are the > most > > > >>> suitable > > > >>> > > > > Ignite > > > >>> > > > > > > APIs > > > >>> > > > > > > > > for > > > >>> > > > > > > > > > such integrations. > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > @Saikat Maitra <saikat.mai...@gmail.com> , > @Roman > > < > > > >>> > > > > > rsht...@yahoo.com > > > >>> > > > > > > > > > > >>> > > > > > > > , > > > >>> > > > > > > > > as > > > >>> > > > > > > > > > streaming technologies contributors/maintainers, > > > could > > > >>> you > > > >>> > > > please > > > >>> > > > > > > join > > > >>> > > > > > > > > this > > > >>> > > > > > > > > > thread? > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > - > > > >>> > > > > > > > > > Denis > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil > > > Gkatziouras < > > > >>> > > > > > > > > > gkatzio...@gmail.com> > > > >>> > > > > > > > > > wrote: > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > Hello everyone, > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > I started using Ignite lately. Part of my work > > > >>> involves > > > >>> > > > > Pub/Sub, > > > >>> > > > > > > > thus I > > > >>> > > > > > > > > > > created a pub/sub streamer. > > > >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka. > > > >>> > > > > > > > > > > Here's how the streamer works. Every node > creates > > > an > > > >>> > > instance > > > >>> > > > > of > > > >>> > > > > > a > > > >>> > > > > > > > > > Pub/Sub > > > >>> > > > > > > > > > > client. > > > >>> > > > > > > > > > > The clients subscribe to the same topic using > the > > > >>> same > > > >>> > > > > > subscription > > > >>> > > > > > > > > name. > > > >>> > > > > > > > > > > A message is being sent and one of the nodes > will > > > >>> receive > > > >>> > > the > > > >>> > > > > > > > message, > > > >>> > > > > > > > > > > process it and then forward it to Ignite using > > the > > > >>> > > extractor > > > >>> > > > > > > > provided. > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > Any thoughts? > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > *Emmanouil Gkatziouras* > > > >>> > > > > > > > > > > https://egkatzioura.com/ | > > > >>> > > > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > >>> > > > > > > > > > > https://github.com/gkatzioura > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >> > > > > > >
