Maybe we could move all our Streaming Integrations there, but what is about maintaining and committer permissions to the new repositories? I see the list of the committers and PMC members there https://bahir.apache.org/community-members/
Could somebody from Ignite community be added to this list as a PMC/committer to maintain Ignite integrations? If yes, I'd happy to join this project and collaborate with these guys together If no, and we should start from the zero level with external PRs - hmmm, it's better to have our own external repository with ApacheBahirr ideology. Also, I agree, that all connectors could be moved there (in ApacheBahirr like repository), but not all modules from the page https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations because they use different parts of core modules and could be developed with different velocity. In reality, before creation of new repositories we need wide discussion based on the document mentioned above. P.S. Of course, we could experiment in the next release 2.8 with one or two integrations like twitter or ZeroMQ Also, I'd like Denis Magda idea for the separate repositories in Apache Github to maintain them by anybody. чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <saikat.mai...@gmail.com>: > Hi Denis, Emmanouil > > Thank you for your email. I think creating a separate integration project > in github and also proposing it as an apache incubator project will be a > good move. The other separate integration modules > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > can > be moved to this new apache incubator project. > > There are similar integration available for Flink and Spark in Apache Bahir > https://bahir.apache.org/ > > Do you think we should reach out to Apache Bahir community with a proposal > or we should plan to create a separate Apache Incubator project? > > Regards, > Saikat > > > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras < > gkatzio...@gmail.com> > wrote: > > > Hi all! > > > > Based on the discussions most probably this is going to be on another > > GitHub repo. Therefore I will prepare a standalone project with the > Pub/Sub > > module and once the repository is created a pull request shall be issued. > > If there is anything else I can do in order to facilitate this please let > > me know. > > > > If a ticket is created for this issue, I shall be able to create a pull > > request and thus a build will take place on ignite's CI. > > If everyone is aligned with adding a Pub/Sub implementation may I create > a > > ticket on JIRA? > > > > My next question has to do on how Ignite configures the build jobs in > Team > > City. The unit tests for my implementation need a local Pub/Sub server up > > and running (they provide one for testing purposes). Will this take > effect > > in a form of a build script inside my implementation or is it something > > that should be configured on Team City? > > > > Kind regards, > > Emmanouil > > > > > > *Emmanouil Gkatziouras* > > https://egkatzioura.com/ | > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > https://github.com/gkatzioura > > > > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dma...@apache.org> wrote: > > > > > Emmanouil, Saikat, > > > > > > After contemplating on this, let me propose another way with this > > > particular integration that is aligned with our modularization > endeavor. > > > The modularization [1] defines the Ignite core, Ignite modules, and > > > independent integrations. The first two are to be located in Ignite > > > repositories and to be tested/updated by the community for every > release. > > > The modules are a subset of existing integrations that are adopted > widely > > > and require community attention. > > > > > > As for the independent integrations [1], those are supposed to be > located > > > in a separate GitHub repo. I would advise us to start implementing this > > > concept with the Pub/Sub integration. Even though it will be located > in a > > > separate Github repo, it will be listed among officially available > > > integrations on the Ignite website featuring Emmanouil as a > contributor. > > > > > > What do you think? > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > > > > > > > > > > > - > > > Denis > > > > > > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <saikat.mai...@gmail.com > > > > > wrote: > > > > > > > Hi Emmanouil, > > > > > > > > The changes looks good to me. I wanted to check if you are also able > to > > > > validate that the once data is added to Ignite cluster you are also > > able > > > to > > > > access it using another Ignite client or using rest endpoints. > > > > > > > > We use Teamcity for CI process, the details are as follows > > > > > > > > Trigger validation of those test suites that have been affected by > your > > > > changes on TeamCity <http://ci.ignite.apache.org/>: < > > > http://204.14.53.153/ > > > > > > > > > > > > > - Locate a test suite you have to check, press button named "..." > > that > > > > is located on the left of "Run" button. "Run custom build" dialog > > will > > > > appear; > > > > - Go to "Changes" tab and choose "pull/<pull-request-number>/head" > > in > > > > "Build branch" dropdown list; > > > > - Press "Run build" button and monitor tests results. > > > > > > > > > > > > > > > > @Roman rsht...@yahoo.com - can you also please take a look? > > > > > > > > Regards, > > > > Saikat > > > > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras < > > > > gkatzio...@gmail.com> > > > > wrote: > > > > > > > > > Hi @Saikat Maitra > > > > > > > > > > I have my implementation on an Ignite fork on my personal GitHub > > > account > > > > > [1] > > > > > It is based on the Kafka implementation which is of a polling > nature. > > > > > GCP does not provide a service like Kafka. Instead it provides us > > with > > > > > Pub/Sub which is just like Kafka. > > > > > Therefore using the Kafka streamer as a guideline made more sense > to > > > me. > > > > > > > > > > Apart from the implementation I have also added a unit test. > > > > > It definitely needs some polishing and a double check on my side to > > > make > > > > > the review smooth. > > > > > For the tests a local Pub/Sub [2] server has to be setup however I > am > > > not > > > > > sure which is the best way to do so with regards to your CI/CD. > > > > > > > > > > [1] > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > > > > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ > > > > > > > > > > *Emmanouil Gkatziouras* > > > > > https://egkatzioura.com/ | > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > > https://github.com/gkatzioura > > > > > > > > > > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra < > saikat.mai...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > Hello Emmanouil, > > > > > > > > > > > > Can you please share more info about the implementation? > > > > > > > > > > > > We have similar implementation for data streamers with Kafka and > > > > RocketMQ > > > > > > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer > > > > > > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer > > > > > > > > > > > > Please review and share if you have any questions. > > > > > > > > > > > > Regards, > > > > > > Saikat > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras < > > > > > > gkatzio...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1] > > > > > > > Yes it is indeed cloud specific. It is the messaging service > > > provided > > > > > by > > > > > > > GCP. > > > > > > > I have made the integration with it, but I haven't published it > > yet > > > > (I > > > > > am > > > > > > > waiting for the discussion/ticket and proceed with the > > > contribution). > > > > > > > > > > > > > > Kind regards, > > > > > > > Emmanouil > > > > > > > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview > > > > > > > > > > > > > > *Emmanouil Gkatziouras* > > > > > > > https://egkatzioura.com/ | > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > > > > https://github.com/gkatzioura > > > > > > > > > > > > > > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <dma...@apache.org> > > > wrote: > > > > > > > > > > > > > > > Hello Emmanouil, > > > > > > > > > > > > > > > > Are you referring to any specific implementation tailored > for a > > > > > cloud? > > > > > > If > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there any > > Github > > > > > > source > > > > > > > > code? > > > > > > > > > > > > > > > > Here is a list of existing integrations with streaming and > > > pub/sub > > > > > > > > frameworks: > > > > > > > > https://apacheignite-mix.readme.io/docs/overview > > > > > > > > > > > > > > > > with the source code in Ignite master: > > > > > > > > https://github.com/apache/ignite/tree/master/modules > > > > > > > > > > > > > > > > Check out those resources to see what are the most suitable > > > Ignite > > > > > APIs > > > > > > > for > > > > > > > > such integrations. > > > > > > > > > > > > > > > > @Saikat Maitra <saikat.mai...@gmail.com> , @Roman < > > > > rsht...@yahoo.com > > > > > > > > > > > > , > > > > > > > as > > > > > > > > streaming technologies contributors/maintainers, could you > > please > > > > > join > > > > > > > this > > > > > > > > thread? > > > > > > > > > > > > > > > > - > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras < > > > > > > > > gkatzio...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > > > I started using Ignite lately. Part of my work involves > > > Pub/Sub, > > > > > > thus I > > > > > > > > > created a pub/sub streamer. > > > > > > > > > Pub/Sub is pretty close to Kafka. > > > > > > > > > Here's how the streamer works. Every node creates an > instance > > > of > > > > a > > > > > > > > Pub/Sub > > > > > > > > > client. > > > > > > > > > The clients subscribe to the same topic using the same > > > > subscription > > > > > > > name. > > > > > > > > > A message is being sent and one of the nodes will receive > the > > > > > > message, > > > > > > > > > process it and then forward it to Ignite using the > extractor > > > > > > provided. > > > > > > > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > > > > > > > *Emmanouil Gkatziouras* > > > > > > > > > https://egkatzioura.com/ | > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > > > > > > https://github.com/gkatzioura > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
