Hi All, I think it would be most helpful to note the distinction between the parallelism aspect and the bridge to domain classes aspect (currently done with reflection in the proposed github repo.)
It seems (to me) that in between the ForkJoin framework already in Java (a low-level library) and up to Apache Spark (an lowel-level set of classes and high-level application-server-like code base), there are a ton of options already out there. I am not sure what yet another framework would do that is not already there. Maybe the distinguishing factor here is the use of reflection? What about annotations? That seems to be more modern approach (Java 5! :-) than the typing of method names in code (as currently done in the repo) which is a nightmare to maintain especially when you are in an evolving code base and refactoring all the time. Maybe an interesting angle would be decorating domain classes with annotations and submitting those to fork/join. Just thinkin' aloud... Gary On Mon, Jun 12, 2017 at 4:29 PM, Matt Sicker <boa...@gmail.com> wrote: > I'd be interested to see where this leads to. It could end up as a sort of > Commons Parallel library. Besides providing an execution API, there could > be plenty of support utilities that tend to be found in all the > *Util(s)/*Helper classes in projects like all the ones I mentioned earlier > (basically all sorts of Hadoop-related projects and other distributed > systems here). > > Really, there's so many ways that such a project could head, I'd like to > hear more ideas on what to focus on. > > On 12 June 2017 at 18:19, Gary Gregory <garydgreg...@gmail.com> wrote: > > > The upshot is that there has to be a way to do this with some custom code > > to at least have the ability to 'fast path' the code without reflection. > > Using lambdas should make this fairly syntactically unobtrusive. > > > > On Mon, Jun 12, 2017 at 4:02 PM, Arun Mohan <strider90a...@gmail.com> > > wrote: > > > > > Yes, reflection is not very performant but I don't think I have any > other > > > choice since the library has to inspect the object supplied by the > client > > > at runtime to pick out the methods to be invoked using > CompletableFuture. > > > But the performance penalty paid for using reflection will be more than > > > offset by the savings of parallel method execution, more so as the no > of > > > methods executed in parallel increases. > > > > > > On Mon, Jun 12, 2017 at 3:21 PM, Gary Gregory <garydgreg...@gmail.com> > > > wrote: > > > > > > > On a lower-level, if you want to use this for lower-level services > > (where > > > > there is no network latency for example), you will need to avoid > using > > > > reflection to get the best performance. > > > > > > > > Gary > > > > > > > > On Mon, Jun 12, 2017 at 3:15 PM, Arun Mohan <strider90a...@gmail.com > > > > > > wrote: > > > > > > > > > Hi Gary, > > > > > > > > > > Thanks for your response. You have some valid and interesting > points > > > :-) > > > > > Of course you are right that Spark is much more mature. Thanks for > > your > > > > > insight. > > > > > It will be interesting indeed to find out if the core > parallelization > > > > > engine of Spark can be isolated like you suggest. > > > > > > > > > > I started working on this project because I felt that there was no > > good > > > > > library for parallelizing method calls which can be plugged in > easily > > > > into > > > > > an existing java project. Ultimately, if such a solution can be > > > > > incorporated in the Apache Commons, it would be a useful addition > to > > > the > > > > > Commons repository. > > > > > > > > > > Thanks, > > > > > Arun > > > > > > > > > > > > > > > > > > > > On Mon, Jun 12, 2017 at 3:01 PM, Gary Gregory < > > garydgreg...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Arun, > > > > > > > > > > > > Sure, and that is to be expected, Spark is more mature than a > four > > > > class > > > > > > prototype. What I am trying to get to is that in order for the > > > library > > > > to > > > > > > be useful, you will end up with more in a first release, and > after > > a > > > > > couple > > > > > > more releases, there will be more and more. Would Spark not have > in > > > its > > > > > > guts the same kind of code your are proposing here? By extension, > > > will > > > > > you > > > > > > not end up with more framework-like (Spark-like) code and > solutions > > > as > > > > > > found in Spark? I am just playing devil's advocate here ;-) > > > > > > > > > > > > > > > > > > What would be interesting would be to find out if there is a core > > > part > > > > of > > > > > > Spark that is separable and ex tractable into a Commons > component. > > > > Since > > > > > > Spark has a proven track record, it is more likely, that such a > > > library > > > > > > would be generally useful than one created from scratch that does > > not > > > > > > integrate with anything else. Again, please do not take any of > this > > > > > > personally, I am just playing here :-) > > > > > > > > > > > > Gary > > > > > > > > > > > > > > > > > > On Mon, Jun 12, 2017 at 2:29 PM, Matt Sicker <boa...@gmail.com> > > > wrote: > > > > > > > > > > > > > I already see a huge difference here: Spark requires a bunch of > > > > > > > infrastructure to be set up, while this library is just a > > library. > > > > > > Similar > > > > > > > to Kafka Streams versus Spark Streaming or Flink or Storm or > > Samza > > > or > > > > > the > > > > > > > others. > > > > > > > > > > > > > > On 12 June 2017 at 16:28, Gary Gregory <garydgreg...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > > > On Mon, Jun 12, 2017 at 2:26 PM, Arun Mohan < > > > > strider90a...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > Good afternoon. > > > > > > > > > > > > > > > > > > I have been working on a java generic parallel execution > > > library > > > > > > which > > > > > > > > will > > > > > > > > > allow clients to execute methods in parallel irrespective > of > > > the > > > > > > number > > > > > > > > of > > > > > > > > > method arguments, type of method arguments, return type of > > the > > > > > method > > > > > > > > etc. > > > > > > > > > > > > > > > > > > Here is the link to the source code: > > > > > > > > > https://github.com/striderarun/parallel-execution-engine > > > > > > > > > > > > > > > > > > The project is in a nascent state and I am the only > > contributor > > > > so > > > > > > > far. I > > > > > > > > > am new to the Apache community and I would like to bring > this > > > > > project > > > > > > > > into > > > > > > > > > Apache and improve, expand and build a developer community > > > around > > > > > it. > > > > > > > > > > > > > > > > > > I think this project can be a sub project of Apache Commons > > > since > > > > > it > > > > > > > > > provides generic components for parallelizing any kind of > > > > methods. > > > > > > > > > > > > > > > > > > Can somebody please guide me or suggest what other options > I > > > can > > > > > > > explore > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > Hi Arun, > > > > > > > > > > > > > > > > Thank you for your proposal. > > > > > > > > > > > > > > > > How would this be different from Apache Spark? > > > > > > > > > > > > > > > > Thank you, > > > > > > > > Gary > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Arun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Matt Sicker <boa...@gmail.com> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Matt Sicker <boa...@gmail.com> >
