Hi,
First things first, let me clarify that I didn't request the feature myself
so I'm not advocating it ;-)
We have a vague proposition in a JIRA ticket, and my intention is to try
give it some objective essence. The team can happily close the ticket with
a "Won't Fix" if we feel that there's no compelling benefit.
That said, the more I think about it, the more I believe Reactive Streams
and Ignite are a valuable combination.
*#1 Compute Jobs*
Reactive Streams would give compute jobs the ability to return partial
results as they are being generated. Basically an API like:
Observable<String> ob = grid.compute().submit(Abc.class,
"foo").toObservable();
would return an Observable to which we can then subscribe to get results as
they are being generated:
ob.subscribe((result) -> System.out.println("Got a result " + result));
The client would get results as they are being emitted by the compute job
server-side.
Of course this is the simplest example. Once we have an Observable we can
do lots and lots of streaming operations on it: merge, drop, folds, etc.
and even combine those operations with other compute jobs. Full set of
operators supported by RxJava:
http://reactivex.io/documentation/operators.html.
So basically a client could send 2, 3, 4... compute jobs and combine their
results together in a streaming fashion:
Observable<String> ob1 = grid.compute().submit(Abc.class,
"foo").toObservable();
Observable<String> ob2 = grid.compute().submit(Def.class,
"foo").toObservable();
Observable<String> ob3 = grid.compute().submit(Ghi.class,
"foo").toObservable();
ob1.flatMap((result) => Observable.just("Observable 1 emitted: " +
result))
.merge(ob2, ob3.skipLast(2))
.subscribe((result) -> System.out.println("Got a result " +
result));
This example would submit 3 jobs, and would combine the streams of all 3
into a single one, but applying a transformation to ob1 and skipping the
last 2 items emitted from ob3.
*#2 Data Streamers*
Similar to above, but for data streamers. Currently our streaming logic is
simple: get a message, transform it and add it to cache. Some streamers
allow ignoring messages, but that's all.
But maybe a user doesn't want to store every single incoming item in the
cache. Maybe they want to store a sliding window result, i.e. receive for 5
seconds, calculate moving averages and store that in the cache.
Or maybe they want to combine multiple data streams together, e.g. combine
3 types of MQTT messages within a window of 10 seconds and only store the
result of an aggregation.
---
I also mentioned messaging and events in my initial email. The rationale
would be similar to the above.
P.S.: For my crude API examples, I'm using RxJava – as that's what I'm
familiar with – not the Reactive Streams API.
P.S. 2: We should put all ideas on a Wiki page – even if we decide to
discard the implementation for now.
Regards,
*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
On Mon, Sep 21, 2015 at 4:49 PM, Dmitriy Setrakyan <dsetrak...@apache.org>
wrote:
> From my standpoint, I would like to see some API examples. How do you
> envision reactive stream support in Ignite at the API level?
>
> D.
>
> On Sun, Sep 20, 2015 at 9:59 PM, Lalit Kumar Jha <lalitj....@gmail.com>
> wrote:
>
> > Yes, lets first explore how reactive-streams can be useful for Ignite.
> >
> > Adding one more candidate to list http://projectreactor.io/docs/, as
> this
> > project:
> > Is for fast-data applications
> > Team are core contributors to the Reactive Streams Specification
> > <https://github.com/reactive-streams/reactive-streams>
> >
> >
> >
> > On Sun, Sep 20, 2015 at 10:33 PM, Sergi Vladykin <
> sergi.vlady...@gmail.com
> > >
> > wrote:
> >
> > > Guys,
> > >
> > > I think you are digging too deep in implementation possibilities right
> > > away, may be lets start from simpler things?
> > > I don't know much about reactive streams and probably not everyone in
> the
> > > community that educated as well :)
> > >
> > > It would be nice to start with a quick overview of what are they and
> why
> > in
> > > makes sense to
> > > to add such a functionality to Ignite? What real use cases do you see?
> > How
> > > can Ignite
> > > be useful for reactive streams users and vice versa?
> > >
> > > Lets get clear answers to these questions first to avoid writing
> useless
> > > code.
> > >
> > > Sergi
> > >
> > > 2015-09-20 18:55 GMT+03:00 Raul Kripalani <r...@evosent.com>:
> > >
> > > > Yeah, the idea would be to wrap RxJava 1.0 with the adapter and only
> > use
> > > > the Reactive Streams API where possible from within Ignite.
> > > >
> > > > Nevertheless, the ReactiveX API would also be accessible in case we
> > want
> > > to
> > > > do stuff with Hystrix too.
> > > >
> > > > Raúl.
> > > > On 20 Sep 2015 16:39, "Vishal Garg" <gargv...@gmail.com> wrote:
> > > >
> > > > > https://github.com/ReactiveX/RxJava/wiki/R
> > > > > eactive-Streams
> > > > > +1 on your approach ( if we are wrapping 1.0 with adapters??)
> > > > > Raul it looks like there is a path to convergence in RxJava 2.0,
> but
> > > > above
> > > > > link mentions that you could use the standard + integration modules
> > > with
> > > > > wrappers on core 1.0 Apis. Where the modules could be Akka or
> RxJava
> > > > > Vishal
> > > > >
> > > > >
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Sep 20, 2015, at 7:03 AM, Gianfranco Murador <
> > > > > murador.gianfra...@gmail.com> wrote:
> > > > > >
> > > > > > Hy Raul, it could be a valid extension of the Service Grid of
> > Ignite
> > > (
> > > > > if I
> > > > > > understand the point), imho.
> > > > > >
> > > > > > 2015-09-20 14:37 GMT+02:00 Raul Kripalani <ra...@apache.org>:
> > > > > >
> > > > > >> Hey Igniters,
> > > > > >>
> > > > > >> Lalit Kumar Jha expressed interest [1] in contributing to ticket
> > > > > IGNITE-815
> > > > > >> [2] which aims to bring in Reactive Streams to Ignite.
> > > > > >>
> > > > > >> I wanted to start a brainstorm about which functionalities we'd
> > like
> > > > to
> > > > > >> cover.
> > > > > >>
> > > > > >> Off the top of my head, I can think of merging, filtering,
> > sorting,
> > > > > >> batching, grouping, etc. on Data Streams, Ignite Messages and
> > > Events.
> > > > > Also,
> > > > > >> integration with Continuous Queries would be useful.
> > > > > >>
> > > > > >> Furthermore, I wanted to ask the community if at this point we
> > feel
> > > > > >> comfortable with Reactive Streams per-se or we prefer to go with
> > > > RxJava
> > > > > for
> > > > > >> now. Reactive Streams is a proposed standard whose main Java
> > > > > implementation
> > > > > >> and is slowly gaining traction (e.g. vert.x [3] and MongoDB
> > > > integrations
> > > > > >> [4]), whereas RxJava is an already mature and production-tested
> > > > > >> implementation of the initial ReactiveX design by Microsoft [5]
> > with
> > > > > 7000+
> > > > > >> stars on GitHub.
> > > > > >>
> > > > > >> RxJava is the foundation of Hystrix [6], a powerful library for
> > > > latency
> > > > > and
> > > > > >> fault tolerance management. RxJava does not implement Reactive
> > > Streams
> > > > > as
> > > > > >> of 1.x (as they predated it), but they've vowed support for 2.x
> > [7].
> > > > > >> Moreover, there's already an adapter which passes the TCK [8].
> > > > > >>
> > > > > >> My take is as follows. I'd like to follow the Reactive Streams
> > > > standard,
> > > > > >> but I wouldn't like to lose support for Hystrix (hopefully it'll
> > > > > eventually
> > > > > >> be migrated to Reactive Streams too). So an option is to go with
> > the
> > > > > RxJava
> > > > > >> + adapter design for now to get the best of both worlds. Once
> > RxJava
> > > > > 2.0 is
> > > > > >> out, we can drop the adapter out of the picture.
> > > > > >>
> > > > > >> Thoughts?
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Hello-Apache-Ignite-Dev-tp3137p3315.html
> > > > > >> [2] https://issues.apache.org/jira/browse/IGNITE-815
> > > > > >> [3] http://vertx.io/docs/vertx-reactive-streams/java/
> > > > > >> [4]
> https://mongodb.github.io/mongo-java-driver-reactivestreams/
> > > > > >> [5] https://msdn.microsoft.com/en-us/data/gg577609.aspx
> > > > > >> [6] https://github.com/Netflix/Hystrix
> > > > > >> [7] https://github.com/ReactiveX/RxJava/wiki/Reactive-Streams
> > > > > >> [8] https://github.com/ReactiveX/RxJavaReactiveStreams
> > > > > >>
> > > > > >> Regards,
> > > > > >>
> > > > > >> *Raúl Kripalani*
> > > > > >> Apache Camel PMC Member & Committer | Enterprise Architect, Open
> > > > Source
> > > > > >> Integration specialist
> > > > > >> http://about.me/raulkripalani |
> > > > > http://www.linkedin.com/in/raulkripalani
> > > > > >> http://blog.raulkr.net | twitter: @raulvk
> > > > > >>
> > > > >
> > > >
> > >
> >
>
