Hi Emmanouil, I have fixed the Flink module testsuites and have setup new project in teamcity.
https://ci.ignite.apache.org/project.html?projectId=IgniteExtensions&tab=projectOverview I am looking into setting up the build. Regards, Saikat On Sat, Nov 23, 2019 at 9:52 AM Saikat Maitra <saikat.mai...@gmail.com> wrote: > Hi Emmanouil, > > Thank you for your email. Yes, the plan is once the Flink PR is merged it > will provide the Licence, parent POMs etc and you can rebase from master > branch and apply the changes on top of it. > > I am getting some issues with test failures in local with > GridTestProperties as the test.properties is not present in this new > project but should be available from dependencies. > > Once I address this issue, I will go ahead and merge the changes and then > we can take it from there. > > Regards, > Saikat > > On Fri, Nov 22, 2019 at 5:23 PM Emmanouil Gkatziouras < > gkatzio...@gmail.com> wrote: > >> Hi all, >> >> I made my first pull request [1]. Since this project is brand new (no >> parent poms, licensing), I reckoned it was best to use Saikat's branch on >> Flink. >> I suppose the Flink branch will be merged first. If not please give me >> guidelines on how I should proceed next. >> >> Kind regards >> Emmanouil >> >> [1] https://github.com/apache/ignite-extensions/pull/2 >> >> *Emmanouil Gkatziouras* >> https://egkatzioura.com/ | >> https://www.linkedin.com/in/gkatziourasemmanouil/ >> https://github.com/gkatzioura >> >> >> On Fri, 22 Nov 2019 at 20:55, Denis Magda <dma...@apache.org> wrote: >> >> > Awesome, ping us whenever you're ready! >> > >> > - >> > Denis >> > >> > >> > On Fri, Nov 22, 2019 at 12:52 PM Emmanouil Gkatziouras < >> > gkatzio...@gmail.com> >> > wrote: >> > >> > > Hi all! >> > > >> > > I am sorry for being late on that. I was trying to refactor the test >> in >> > > order not to be in need of any external tools or spinning up a server. >> > > I did forked the new repo and indeed my changes there, so a pull >> request >> > is >> > > a matter of time! >> > > >> > > Kind regards >> > > >> > > *Emmanouil Gkatziouras* >> > > https://egkatzioura.com/ | >> > > https://www.linkedin.com/in/gkatziourasemmanouil/ >> > > https://github.com/gkatzioura >> > > >> > > >> > > On Fri, 22 Nov 2019 at 20:45, Denis Magda <dma...@apache.org> wrote: >> > > >> > > > Hi Emmanouil, >> > > > >> > > > Do you have any questions or need any support from the community? >> > > > >> > > > - >> > > > Denis >> > > > >> > > > >> > > > On Sun, Nov 10, 2019 at 3:07 PM Saikat Maitra < >> saikat.mai...@gmail.com >> > > >> > > > wrote: >> > > > >> > > >> Hi Emmanouil, >> > > >> >> > > >> Can you please take a looks at dev utils, if this is something you >> are >> > > >> looking for >> > > >> https://github.com/apache/ignite/tree/master/modules/dev-utils? >> > > >> >> > > >> IMO, if you can release Pub/Sub server in maven and then use it as >> > > >> dependency, that would be great. >> > > >> >> > > >> Regards, >> > > >> Saikat >> > > >> >> > > >> On Sun, Nov 10, 2019 at 5:00 PM Saikat Maitra < >> > saikat.mai...@gmail.com> >> > > >> wrote: >> > > >> >> > > >> > Hi Emmanouil, >> > > >> > >> > > >> > The master branch has been initialized. I have raised a PR to >> > migrate >> > > >> > Flink module to ignite-extensions. >> > > >> > >> > > >> > PR https://github.com/apache/ignite-extensions/pull/1 >> > > >> > Jira : https://issues.apache.org/jira/browse/IGNITE-12356 >> > > >> > >> > > >> > Regards, >> > > >> > Saikat >> > > >> > >> > > >> > >> > > >> > On Sun, Nov 10, 2019 at 3:39 PM Emmanouil Gkatziouras < >> > > >> > gkatzio...@gmail.com> wrote: >> > > >> > >> > > >> >> Hi all! >> > > >> >> >> > > >> >> This is an update. The integration is tested with mocks based on >> > the >> > > >> >> original Ignite repo [1]. >> > > >> >> I am currently waiting for the first commits on the new >> integration >> > > >> >> project >> > > >> >> to take place in order to create my pull request accordingly to >> the >> > > >> >> initial >> > > >> >> commits. >> > > >> >> In order to simulate the Pub/Sub server in a test I used a test >> > > server >> > > >> >> implementation [2]. Unfortunately they don't have this util >> > uploaded >> > > >> in a >> > > >> >> maven repo, thus it needs to be hosted on a maven repo (Apache >> > > License >> > > >> >> 2.0). Is there a repo for utilities that Ignite uses for the >> > > >> test/build? >> > > >> >> I have also created a ticket [3] >> > > >> >> >> > > >> >> Kind regards >> > > >> >> >> > > >> >> [1] >> > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub >> > > >> >> [2] >> https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator >> > > >> >> [3] https://issues.apache.org/jira/browse/IGNITE-12262 >> > > >> >> >> > > >> >> *Emmanouil Gkatziouras* >> > > >> >> https://egkatzioura.com/ | >> > > >> >> https://www.linkedin.com/in/gkatziourasemmanouil/ >> > > >> >> https://github.com/gkatzioura >> > > >> >> >> > > >> >> >> > > >> >> On Fri, 25 Oct 2019 at 03:35, Saikat Maitra < >> > saikat.mai...@gmail.com >> > > > >> > > >> >> wrote: >> > > >> >> >> > > >> >> > Hello Ilya, >> > > >> >> > >> > > >> >> > Thank you for your email. Yes, before we remove any >> integration >> > we >> > > >> will >> > > >> >> > move them to another repository and release. >> > > >> >> > >> > > >> >> > We are thinking we will start with Pub/Sub Streamer >> > implementation >> > > >> and >> > > >> >> > continue to move other integration as mentioned in the >> following >> > > doc. >> > > >> >> > >> > > >> >> > >> > > >> >> > >> > > >> >> >> > > >> >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >> > > >> >> > >> > > >> >> > Regards, >> > > >> >> > Saikat >> > > >> >> > >> > > >> >> > >> > > >> >> > On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev < >> > > >> >> > ilya.kasnach...@gmail.com> >> > > >> >> > wrote: >> > > >> >> > >> > > >> >> > > Hello! >> > > >> >> > > >> > > >> >> > > My take on this: we want to offload some of integrations, >> but >> > > >> please >> > > >> >> > commit >> > > >> >> > > and release them to another project first, and only then we >> > will >> > > >> >> decide >> > > >> >> > to >> > > >> >> > > remove them from our repository. >> > > >> >> > > >> > > >> >> > > Please also make sure they are available with same artifact >> > name >> > > >> and >> > > >> >> API. >> > > >> >> > > If this has to be changed, let's wait for AI 3.0 >> > > >> >> > > >> > > >> >> > > Regards, >> > > >> >> > > -- >> > > >> >> > > Ilya Kasnacheev >> > > >> >> > > >> > > >> >> > > >> > > >> >> > > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra < >> > > >> saikat.mai...@gmail.com>: >> > > >> >> > > >> > > >> >> > > > Hello Emmanouil, >> > > >> >> > > > >> > > >> >> > > > As discussed earlier, I discussed with Apache Bahir >> community >> > > and >> > > >> >> they >> > > >> >> > > are >> > > >> >> > > > interested to have Apache Ignite extensions as part of >> Apache >> > > >> Bahir >> > > >> >> > > > project. >> > > >> >> > > > >> > > >> >> > > > Can you please share Pub/Sub streamer implementation >> details >> > > with >> > > >> >> > Apache >> > > >> >> > > > Bahir community and request for feedback. >> > > >> >> > > > >> > > >> >> > > > https://bahir.apache.org/ >> > > >> >> > > > >> > > >> >> > > > I have also requested for contributor access in Jira for >> > Apache >> > > >> >> Bahir >> > > >> >> > > > project so that I can create issues and assign to myself. >> > > >> >> > > > >> > > >> >> > > > I can help with code reviews as well. >> > > >> >> > > > >> > > >> >> > > > Here is the email thread for reference >> > > >> >> > > > >> > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html >> > > >> >> > > > >> > > >> >> > > > Regards, >> > > >> >> > > > Saikat >> > > >> >> > > > >> > > >> >> > > > >> > > >> >> > > > >> > > >> >> > > > >> > > >> >> > > > >> > > >> >> > > > >> > > >> >> > > > >> > > >> >> > > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra < >> > > >> >> saikat.mai...@gmail.com >> > > >> >> > > >> > > >> >> > > > wrote: >> > > >> >> > > > >> > > >> >> > > > > Hello, >> > > >> >> > > > > >> > > >> >> > > > > As discussed I have created following discussion >> threads on >> > > our >> > > >> >> > > migration >> > > >> >> > > > > proposals for Apache Ignite extensions: >> > > >> >> > > > > >> > > >> >> > > > > >> > > >> >> > > > > >> > > >> >> > > > >> > > >> >> > > >> > > >> >> > >> > > >> >> >> > > >> >> > > >> > >> http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html >> > > >> >> > > > > >> > > >> >> > > > > >> > > >> https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html >> > > >> >> > > > > >> > > >> >> > > > > I will share update as I hear more information. >> > > >> >> > > > > >> > > >> >> > > > > Regards, >> > > >> >> > > > > Saikat >> > > >> >> > > > > >> > > >> >> > > > > >> > > >> >> > > > > >> > > >> >> > > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra < >> > > >> >> > saikat.mai...@gmail.com >> > > >> >> > > > >> > > >> >> > > > > wrote: >> > > >> >> > > > > >> > > >> >> > > > >> Hello Denis, >> > > >> >> > > > >> >> > > >> >> > > > >> Sure, sounds good. I will create a separate discussion >> > > thread >> > > >> to >> > > >> >> get >> > > >> >> > > > >> community feedback on whether we >> > > >> >> > > > >> should join the Bahir or kick-off "Ignite Extensions" >> > > project. >> > > >> >> > > > >> >> > > >> >> > > > >> Regards, >> > > >> >> > > > >> Saikat >> > > >> >> > > > >> >> > > >> >> > > > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda < >> > > >> dma...@apache.org> >> > > >> >> > > wrote: >> > > >> >> > > > >> >> > > >> >> > > > >>> Folks, >> > > >> >> > > > >>> >> > > >> >> > > > >>> The concept of Apache Bahir is precisely what we need! >> > > >> Saikat, >> > > >> >> > thanks >> > > >> >> > > > for >> > > >> >> > > > >>> doing the research. Why I believe the * idea* fits us >> > well: >> > > >> >> > > > >>> >> > > >> >> > > > >>> - All integrations can be stored in separate Github >> > > >> >> repositories >> > > >> >> > > and >> > > >> >> > > > >>> have their dev lifecycles. We've not obliged to >> couple >> > > all >> > > >> >> the >> > > >> >> > > > >>> integrations >> > > >> >> > > > >>> in a single repo. For instance, Spark can be >> located >> > in >> > > a >> > > >> >> > > dedicated >> > > >> >> > > > >>> repo >> > > >> >> > > > >>> while streaming integrations might be in a single >> one. >> > > >> It's >> > > >> >> up >> > > >> >> > to >> > > >> >> > > > us. >> > > >> >> > > > >>> - All the repositories and integrations belong to >> ASF. >> > > >> We're >> > > >> >> not >> > > >> >> > > > >>> dumping >> > > >> >> > > > >>> them on Github but rather keep supporting and >> > developing >> > > >> in >> > > >> >> > > > >>> accordance with >> > > >> >> > > > >>> ASF vision and practices. >> > > >> >> > > > >>> - There is a way to reward contributors via >> > > committership >> > > >> and >> > > >> >> > PMC >> > > >> >> > > > >>> membership. You won't get this if a project is just >> > one >> > > of >> > > >> >> the >> > > >> >> > > > >>> millions of >> > > >> >> > > > >>> Github projects. >> > > >> >> > > > >>> >> > > >> >> > > > >>> Saikat, as Ignite PMC member, could you please >> kick-off a >> > > >> >> separate >> > > >> >> > > > >>> dev-list >> > > >> >> > > > >>> discussion to involve more community members and >> > referring >> > > to >> > > >> >> this >> > > >> >> > > > >>> thread? >> > > >> >> > > > >>> I think the primary question the community needs to >> > answer >> > > >> >> whether >> > > >> >> > we >> > > >> >> > > > >>> should join the Bahir or kick-off "Ignite Extensions" >> > > >> project? >> > > >> >> > > > >>> >> > > >> >> > > > >>> - >> > > >> >> > > > >>> Denis >> > > >> >> > > > >>> >> > > >> >> > > > >>> >> > > >> >> > > > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev < >> > > >> >> > > > zaleslaw....@gmail.com> >> > > >> >> > > > >>> wrote: >> > > >> >> > > > >>> >> > > >> >> > > > >>> > Maybe we could move all our Streaming Integrations >> > there, >> > > >> but >> > > >> >> > what >> > > >> >> > > is >> > > >> >> > > > >>> about >> > > >> >> > > > >>> > maintaining and committer permissions to the new >> > > >> repositories? >> > > >> >> > > > >>> > I see the list of the committers and PMC members >> there >> > > >> >> > > > >>> > https://bahir.apache.org/community-members/ >> > > >> >> > > > >>> > >> > > >> >> > > > >>> > Could somebody from Ignite community be added to >> this >> > > list >> > > >> as >> > > >> >> a >> > > >> >> > > > >>> > PMC/committer to maintain Ignite integrations? >> > > >> >> > > > >>> > If yes, I'd happy to join this project and >> collaborate >> > > with >> > > >> >> these >> > > >> >> > > > guys >> > > >> >> > > > >>> > together >> > > >> >> > > > >>> > If no, and we should start from the zero level with >> > > >> external >> > > >> >> PRs >> > > >> >> > - >> > > >> >> > > > >>> hmmm, >> > > >> >> > > > >>> > it's better to have our own external repository with >> > > >> >> ApacheBahirr >> > > >> >> > > > >>> ideology. >> > > >> >> > > > >>> > >> > > >> >> > > > >>> > Also, I agree, that all connectors could be moved >> there >> > > (in >> > > >> >> > > > >>> ApacheBahirr >> > > >> >> > > > >>> > like repository), but not all modules from the page >> > > >> >> > > > >>> > >> > > >> >> > > > >>> > >> > > >> >> > > > >>> >> > > >> >> > > > >> > > >> >> > > >> > > >> >> > >> > > >> >> >> > > >> >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >> > > >> >> > > > >>> > because >> > > >> >> > > > >>> > they use different parts of core modules and could >> be >> > > >> >> developed >> > > >> >> > > with >> > > >> >> > > > >>> > different velocity. >> > > >> >> > > > >>> > >> > > >> >> > > > >>> > In reality, before creation of new repositories we >> need >> > > >> wide >> > > >> >> > > > discussion >> > > >> >> > > > >>> > based on the document mentioned above. >> > > >> >> > > > >>> > >> > > >> >> > > > >>> > P.S. Of course, we could experiment in the next >> release >> > > 2.8 >> > > >> >> with >> > > >> >> > > one >> > > >> >> > > > >>> or two >> > > >> >> > > > >>> > integrations like twitter or ZeroMQ >> > > >> >> > > > >>> > Also, I'd like Denis Magda idea for the separate >> > > >> repositories >> > > >> >> in >> > > >> >> > > > Apache >> > > >> >> > > > >>> > Github to maintain them by anybody. >> > > >> >> > > > >>> > >> > > >> >> > > > >>> > >> > > >> >> > > > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra < >> > > >> >> > > saikat.mai...@gmail.com >> > > >> >> > > > >: >> > > >> >> > > > >>> > >> > > >> >> > > > >>> > > Hi Denis, Emmanouil >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > > Thank you for your email. I think creating a >> separate >> > > >> >> > integration >> > > >> >> > > > >>> project >> > > >> >> > > > >>> > > in github and also proposing it as an apache >> > incubator >> > > >> >> project >> > > >> >> > > will >> > > >> >> > > > >>> be a >> > > >> >> > > > >>> > > good move. The other separate integration modules >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > >> > > >> >> > > > >>> >> > > >> >> > > > >> > > >> >> > > >> > > >> >> > >> > > >> >> >> > > >> >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >> > > >> >> > > > >>> > > can >> > > >> >> > > > >>> > > be moved to this new apache incubator project. >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > > There are similar integration available for Flink >> and >> > > >> Spark >> > > >> >> in >> > > >> >> > > > Apache >> > > >> >> > > > >>> > Bahir >> > > >> >> > > > >>> > > https://bahir.apache.org/ >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > > Do you think we should reach out to Apache Bahir >> > > >> community >> > > >> >> > with a >> > > >> >> > > > >>> > proposal >> > > >> >> > > > >>> > > or we should plan to create a separate Apache >> > Incubator >> > > >> >> > project? >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > > Regards, >> > > >> >> > > > >>> > > Saikat >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil >> > Gkatziouras < >> > > >> >> > > > >>> > > gkatzio...@gmail.com> >> > > >> >> > > > >>> > > wrote: >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > > > Hi all! >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > > Based on the discussions most probably this is >> > going >> > > to >> > > >> >> be on >> > > >> >> > > > >>> another >> > > >> >> > > > >>> > > > GitHub repo. Therefore I will prepare a >> standalone >> > > >> project >> > > >> >> > with >> > > >> >> > > > the >> > > >> >> > > > >>> > > Pub/Sub >> > > >> >> > > > >>> > > > module and once the repository is created a pull >> > > >> request >> > > >> >> > shall >> > > >> >> > > be >> > > >> >> > > > >>> > issued. >> > > >> >> > > > >>> > > > If there is anything else I can do in order to >> > > >> facilitate >> > > >> >> > this >> > > >> >> > > > >>> please >> > > >> >> > > > >>> > let >> > > >> >> > > > >>> > > > me know. >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > > If a ticket is created for this issue, I shall >> be >> > > able >> > > >> to >> > > >> >> > > create >> > > >> >> > > > a >> > > >> >> > > > >>> pull >> > > >> >> > > > >>> > > > request and thus a build will take place on >> > ignite's >> > > >> CI. >> > > >> >> > > > >>> > > > If everyone is aligned with adding a Pub/Sub >> > > >> >> implementation >> > > >> >> > > may I >> > > >> >> > > > >>> > create >> > > >> >> > > > >>> > > a >> > > >> >> > > > >>> > > > ticket on JIRA? >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > > My next question has to do on how Ignite >> configures >> > > the >> > > >> >> build >> > > >> >> > > > jobs >> > > >> >> > > > >>> in >> > > >> >> > > > >>> > > Team >> > > >> >> > > > >>> > > > City. The unit tests for my implementation need >> a >> > > local >> > > >> >> > Pub/Sub >> > > >> >> > > > >>> server >> > > >> >> > > > >>> > up >> > > >> >> > > > >>> > > > and running (they provide one for testing >> > purposes). >> > > >> Will >> > > >> >> > this >> > > >> >> > > > take >> > > >> >> > > > >>> > > effect >> > > >> >> > > > >>> > > > in a form of a build script inside my >> > implementation >> > > or >> > > >> >> is it >> > > >> >> > > > >>> something >> > > >> >> > > > >>> > > > that should be configured on Team City? >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > > Kind regards, >> > > >> >> > > > >>> > > > Emmanouil >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > > *Emmanouil Gkatziouras* >> > > >> >> > > > >>> > > > https://egkatzioura.com/ | >> > > >> >> > > > >>> > > > >> https://www.linkedin.com/in/gkatziourasemmanouil/ >> > > >> >> > > > >>> > > > https://github.com/gkatzioura >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda < >> > > >> >> dma...@apache.org >> > > >> >> > > >> > > >> >> > > > >>> wrote: >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > > > Emmanouil, Saikat, >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > After contemplating on this, let me propose >> > another >> > > >> way >> > > >> >> > with >> > > >> >> > > > this >> > > >> >> > > > >>> > > > > particular integration that is aligned with >> our >> > > >> >> > > modularization >> > > >> >> > > > >>> > > endeavor. >> > > >> >> > > > >>> > > > > The modularization [1] defines the Ignite >> core, >> > > >> Ignite >> > > >> >> > > modules, >> > > >> >> > > > >>> and >> > > >> >> > > > >>> > > > > independent integrations. The first two are >> to be >> > > >> >> located >> > > >> >> > in >> > > >> >> > > > >>> Ignite >> > > >> >> > > > >>> > > > > repositories and to be tested/updated by the >> > > >> community >> > > >> >> for >> > > >> >> > > > every >> > > >> >> > > > >>> > > release. >> > > >> >> > > > >>> > > > > The modules are a subset of existing >> integrations >> > > >> that >> > > >> >> are >> > > >> >> > > > >>> adopted >> > > >> >> > > > >>> > > widely >> > > >> >> > > > >>> > > > > and require community attention. >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > As for the independent integrations [1], those >> > are >> > > >> >> supposed >> > > >> >> > > to >> > > >> >> > > > be >> > > >> >> > > > >>> > > located >> > > >> >> > > > >>> > > > > in a separate GitHub repo. I would advise us >> to >> > > start >> > > >> >> > > > >>> implementing >> > > >> >> > > > >>> > this >> > > >> >> > > > >>> > > > > concept with the Pub/Sub integration. Even >> though >> > > it >> > > >> >> will >> > > >> >> > be >> > > >> >> > > > >>> located >> > > >> >> > > > >>> > > in a >> > > >> >> > > > >>> > > > > separate Github repo, it will be listed among >> > > >> officially >> > > >> >> > > > >>> available >> > > >> >> > > > >>> > > > > integrations on the Ignite website featuring >> > > >> Emmanouil >> > > >> >> as a >> > > >> >> > > > >>> > > contributor. >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > What do you think? >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > [1] >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > >> > > >> >> > > > >>> >> > > >> >> > > > >> > > >> >> > > >> > > >> >> > >> > > >> >> >> > > >> >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > - >> > > >> >> > > > >>> > > > > Denis >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra >> < >> > > >> >> > > > >>> > saikat.mai...@gmail.com >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > > > wrote: >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > > Hi Emmanouil, >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > > The changes looks good to me. I wanted to >> check >> > > if >> > > >> you >> > > >> >> > are >> > > >> >> > > > also >> > > >> >> > > > >>> > able >> > > >> >> > > > >>> > > to >> > > >> >> > > > >>> > > > > > validate that the once data is added to >> Ignite >> > > >> cluster >> > > >> >> > you >> > > >> >> > > > are >> > > >> >> > > > >>> also >> > > >> >> > > > >>> > > > able >> > > >> >> > > > >>> > > > > to >> > > >> >> > > > >>> > > > > > access it using another Ignite client or >> using >> > > rest >> > > >> >> > > > endpoints. >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > > We use Teamcity for CI process, the details >> are >> > > as >> > > >> >> > follows >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > > Trigger validation of those test suites that >> > have >> > > >> been >> > > >> >> > > > >>> affected by >> > > >> >> > > > >>> > > your >> > > >> >> > > > >>> > > > > > changes on TeamCity < >> > > http://ci.ignite.apache.org/>: >> > > >> < >> > > >> >> > > > >>> > > > > http://204.14.53.153/ >> > > >> >> > > > >>> > > > > > > >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > > - Locate a test suite you have to check, >> > press >> > > >> >> button >> > > >> >> > > > named >> > > >> >> > > > >>> > "..." >> > > >> >> > > > >>> > > > that >> > > >> >> > > > >>> > > > > > is located on the left of "Run" button. >> "Run >> > > >> custom >> > > >> >> > > build" >> > > >> >> > > > >>> > dialog >> > > >> >> > > > >>> > > > will >> > > >> >> > > > >>> > > > > > appear; >> > > >> >> > > > >>> > > > > > - Go to "Changes" tab and choose >> > > >> >> > > > >>> > "pull/<pull-request-number>/head" >> > > >> >> > > > >>> > > > in >> > > >> >> > > > >>> > > > > > "Build branch" dropdown list; >> > > >> >> > > > >>> > > > > > - Press "Run build" button and monitor >> tests >> > > >> >> results. >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > > @Roman rsht...@yahoo.com - can you also >> please >> > > >> take a >> > > >> >> > > look? >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > > Regards, >> > > >> >> > > > >>> > > > > > Saikat >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil >> > > >> Gkatziouras >> > > >> >> < >> > > >> >> > > > >>> > > > > > gkatzio...@gmail.com> >> > > >> >> > > > >>> > > > > > wrote: >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > > > Hi @Saikat Maitra >> > > >> >> > > > >>> > > > > > > >> > > >> >> > > > >>> > > > > > > I have my implementation on an Ignite >> fork on >> > > my >> > > >> >> > personal >> > > >> >> > > > >>> GitHub >> > > >> >> > > > >>> > > > > account >> > > >> >> > > > >>> > > > > > > [1] >> > > >> >> > > > >>> > > > > > > It is based on the Kafka implementation >> which >> > > is >> > > >> of >> > > >> >> a >> > > >> >> > > > polling >> > > >> >> > > > >>> > > nature. >> > > >> >> > > > >>> > > > > > > GCP does not provide a service like Kafka. >> > > >> Instead >> > > >> >> it >> > > >> >> > > > >>> provides us >> > > >> >> > > > >>> > > > with >> > > >> >> > > > >>> > > > > > > Pub/Sub which is just like Kafka. >> > > >> >> > > > >>> > > > > > > Therefore using the Kafka streamer as a >> > > guideline >> > > >> >> made >> > > >> >> > > more >> > > >> >> > > > >>> sense >> > > >> >> > > > >>> > > to >> > > >> >> > > > >>> > > > > me. >> > > >> >> > > > >>> > > > > > > >> > > >> >> > > > >>> > > > > > > Apart from the implementation I have also >> > > added a >> > > >> >> unit >> > > >> >> > > > test. >> > > >> >> > > > >>> > > > > > > It definitely needs some polishing and a >> > double >> > > >> >> check >> > > >> >> > on >> > > >> >> > > my >> > > >> >> > > > >>> side >> > > >> >> > > > >>> > to >> > > >> >> > > > >>> > > > > make >> > > >> >> > > > >>> > > > > > > the review smooth. >> > > >> >> > > > >>> > > > > > > For the tests a local Pub/Sub [2] server >> has >> > to >> > > >> be >> > > >> >> > setup >> > > >> >> > > > >>> however >> > > >> >> > > > >>> > I >> > > >> >> > > > >>> > > am >> > > >> >> > > > >>> > > > > not >> > > >> >> > > > >>> > > > > > > sure which is the best way to do so with >> > > regards >> > > >> to >> > > >> >> > your >> > > >> >> > > > >>> CI/CD. >> > > >> >> > > > >>> > > > > > > >> > > >> >> > > > >>> > > > > > > [1] >> > > >> >> > > > >>> > > >> > > >> >> > >> https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub >> > > >> >> > > > >>> > > > > > > [2] >> > > >> >> > > > >>> >> > https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ >> > > >> >> > > > >>> > > > > > > >> > > >> >> > > > >>> > > > > > > *Emmanouil Gkatziouras* >> > > >> >> > > > >>> > > > > > > https://egkatzioura.com/ | >> > > >> >> > > > >>> > > > > > > >> > > >> https://www.linkedin.com/in/gkatziourasemmanouil/ >> > > >> >> > > > >>> > > > > > > https://github.com/gkatzioura >> > > >> >> > > > >>> > > > > > > >> > > >> >> > > > >>> > > > > > > >> > > >> >> > > > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat >> Maitra < >> > > >> >> > > > >>> > > saikat.mai...@gmail.com >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > > > > wrote: >> > > >> >> > > > >>> > > > > > > >> > > >> >> > > > >>> > > > > > > > Hello Emmanouil, >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > Can you please share more info about the >> > > >> >> > > implementation? >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > We have similar implementation for data >> > > >> streamers >> > > >> >> > with >> > > >> >> > > > >>> Kafka >> > > >> >> > > > >>> > and >> > > >> >> > > > >>> > > > > > RocketMQ >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > https://apacheignite-mix.readme.io/docs/kafka-streamer >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > Please review and share if you have any >> > > >> questions. >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > Regards, >> > > >> >> > > > >>> > > > > > > > Saikat >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM >> Emmanouil >> > > >> >> > Gkatziouras < >> > > >> >> > > > >>> > > > > > > > gkatzio...@gmail.com> >> > > >> >> > > > >>> > > > > > > > wrote: >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > > Hi all, >> > > >> >> > > > >>> > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > By mentioning Pub/Sub I mean Google >> > Cloud's >> > > >> >> Pub/Sub >> > > >> >> > > [1] >> > > >> >> > > > >>> > > > > > > > > Yes it is indeed cloud specific. It is >> > the >> > > >> >> > messaging >> > > >> >> > > > >>> service >> > > >> >> > > > >>> > > > > provided >> > > >> >> > > > >>> > > > > > > by >> > > >> >> > > > >>> > > > > > > > > GCP. >> > > >> >> > > > >>> > > > > > > > > I have made the integration with it, >> but >> > I >> > > >> >> haven't >> > > >> >> > > > >>> published >> > > >> >> > > > >>> > it >> > > >> >> > > > >>> > > > yet >> > > >> >> > > > >>> > > > > > (I >> > > >> >> > > > >>> > > > > > > am >> > > >> >> > > > >>> > > > > > > > > waiting for the discussion/ticket and >> > > proceed >> > > >> >> with >> > > >> >> > > the >> > > >> >> > > > >>> > > > > contribution). >> > > >> >> > > > >>> > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > Kind regards, >> > > >> >> > > > >>> > > > > > > > > Emmanouil >> > > >> >> > > > >>> > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > [1] >> > > >> >> https://cloud.google.com/pubsub/docs/overview >> > > >> >> > > > >>> > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > *Emmanouil Gkatziouras* >> > > >> >> > > > >>> > > > > > > > > https://egkatzioura.com/ | >> > > >> >> > > > >>> > > > > > > > > >> > > >> >> https://www.linkedin.com/in/gkatziourasemmanouil/ >> > > >> >> > > > >>> > > > > > > > > https://github.com/gkatzioura >> > > >> >> > > > >>> > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis >> > Magda < >> > > >> >> > > > >>> dma...@apache.org >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > > > > wrote: >> > > >> >> > > > >>> > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > Hello Emmanouil, >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > Are you referring to any specific >> > > >> >> implementation >> > > >> >> > > > >>> tailored >> > > >> >> > > > >>> > > for a >> > > >> >> > > > >>> > > > > > > cloud? >> > > >> >> > > > >>> > > > > > > > If >> > > >> >> > > > >>> > > > > > > > > > I'm not mistaken you were mentioning >> > > >> AWS/GCE. >> > > >> >> Is >> > > >> >> > > > there >> > > >> >> > > > >>> any >> > > >> >> > > > >>> > > > Github >> > > >> >> > > > >>> > > > > > > > source >> > > >> >> > > > >>> > > > > > > > > > code? >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > Here is a list of existing >> integrations >> > > >> with >> > > >> >> > > > streaming >> > > >> >> > > > >>> and >> > > >> >> > > > >>> > > > > pub/sub >> > > >> >> > > > >>> > > > > > > > > > frameworks: >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> https://apacheignite-mix.readme.io/docs/overview >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > with the source code in Ignite >> master: >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > https://github.com/apache/ignite/tree/master/modules >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > Check out those resources to see >> what >> > are >> > > >> the >> > > >> >> > most >> > > >> >> > > > >>> suitable >> > > >> >> > > > >>> > > > > Ignite >> > > >> >> > > > >>> > > > > > > APIs >> > > >> >> > > > >>> > > > > > > > > for >> > > >> >> > > > >>> > > > > > > > > > such integrations. >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > @Saikat Maitra < >> > saikat.mai...@gmail.com> >> > > , >> > > >> >> > @Roman >> > > >> >> > > < >> > > >> >> > > > >>> > > > > > rsht...@yahoo.com >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > > , >> > > >> >> > > > >>> > > > > > > > > as >> > > >> >> > > > >>> > > > > > > > > > streaming technologies >> > > >> >> contributors/maintainers, >> > > >> >> > > > could >> > > >> >> > > > >>> you >> > > >> >> > > > >>> > > > please >> > > >> >> > > > >>> > > > > > > join >> > > >> >> > > > >>> > > > > > > > > this >> > > >> >> > > > >>> > > > > > > > > > thread? >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > - >> > > >> >> > > > >>> > > > > > > > > > Denis >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM >> > Emmanouil >> > > >> >> > > > Gkatziouras < >> > > >> >> > > > >>> > > > > > > > > > gkatzio...@gmail.com> >> > > >> >> > > > >>> > > > > > > > > > wrote: >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > > Hello everyone, >> > > >> >> > > > >>> > > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > > I started using Ignite lately. >> Part >> > of >> > > my >> > > >> >> work >> > > >> >> > > > >>> involves >> > > >> >> > > > >>> > > > > Pub/Sub, >> > > >> >> > > > >>> > > > > > > > thus I >> > > >> >> > > > >>> > > > > > > > > > > created a pub/sub streamer. >> > > >> >> > > > >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka. >> > > >> >> > > > >>> > > > > > > > > > > Here's how the streamer works. >> Every >> > > node >> > > >> >> > creates >> > > >> >> > > > an >> > > >> >> > > > >>> > > instance >> > > >> >> > > > >>> > > > > of >> > > >> >> > > > >>> > > > > > a >> > > >> >> > > > >>> > > > > > > > > > Pub/Sub >> > > >> >> > > > >>> > > > > > > > > > > client. >> > > >> >> > > > >>> > > > > > > > > > > The clients subscribe to the same >> > topic >> > > >> >> using >> > > >> >> > the >> > > >> >> > > > >>> same >> > > >> >> > > > >>> > > > > > subscription >> > > >> >> > > > >>> > > > > > > > > name. >> > > >> >> > > > >>> > > > > > > > > > > A message is being sent and one of >> > the >> > > >> nodes >> > > >> >> > will >> > > >> >> > > > >>> receive >> > > >> >> > > > >>> > > the >> > > >> >> > > > >>> > > > > > > > message, >> > > >> >> > > > >>> > > > > > > > > > > process it and then forward it to >> > > Ignite >> > > >> >> using >> > > >> >> > > the >> > > >> >> > > > >>> > > extractor >> > > >> >> > > > >>> > > > > > > > provided. >> > > >> >> > > > >>> > > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > > Any thoughts? >> > > >> >> > > > >>> > > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > > *Emmanouil Gkatziouras* >> > > >> >> > > > >>> > > > > > > > > > > https://egkatzioura.com/ | >> > > >> >> > > > >>> > > > > > > > > > > >> > > >> >> > > https://www.linkedin.com/in/gkatziourasemmanouil/ >> > > >> >> > > > >>> > > > > > > > > > > https://github.com/gkatzioura >> > > >> >> > > > >>> > > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > > >> > > >> >> > > > >>> > > > > > > > > >> > > >> >> > > > >>> > > > > > > > >> > > >> >> > > > >>> > > > > > > >> > > >> >> > > > >>> > > > > > >> > > >> >> > > > >>> > > > > >> > > >> >> > > > >>> > > > >> > > >> >> > > > >>> > > >> > > >> >> > > > >>> > >> > > >> >> > > > >>> >> > > >> >> > > > >> >> > > >> >> > > > >> > > >> >> > > >> > > >> >> > >> > > >> >> >> > > >> > >> > > >> >> > > > >> > > >> > >> >
