On a lower-level, if you want to use this for lower-level services (where there is no network latency for example), you will need to avoid using reflection to get the best performance.
Gary On Mon, Jun 12, 2017 at 3:15 PM, Arun Mohan <strider90a...@gmail.com> wrote: > Hi Gary, > > Thanks for your response. You have some valid and interesting points :-) > Of course you are right that Spark is much more mature. Thanks for your > insight. > It will be interesting indeed to find out if the core parallelization > engine of Spark can be isolated like you suggest. > > I started working on this project because I felt that there was no good > library for parallelizing method calls which can be plugged in easily into > an existing java project. Ultimately, if such a solution can be > incorporated in the Apache Commons, it would be a useful addition to the > Commons repository. > > Thanks, > Arun > > > > On Mon, Jun 12, 2017 at 3:01 PM, Gary Gregory <garydgreg...@gmail.com> > wrote: > > > Hi Arun, > > > > Sure, and that is to be expected, Spark is more mature than a four class > > prototype. What I am trying to get to is that in order for the library to > > be useful, you will end up with more in a first release, and after a > couple > > more releases, there will be more and more. Would Spark not have in its > > guts the same kind of code your are proposing here? By extension, will > you > > not end up with more framework-like (Spark-like) code and solutions as > > found in Spark? I am just playing devil's advocate here ;-) > > > > > > What would be interesting would be to find out if there is a core part of > > Spark that is separable and ex tractable into a Commons component. Since > > Spark has a proven track record, it is more likely, that such a library > > would be generally useful than one created from scratch that does not > > integrate with anything else. Again, please do not take any of this > > personally, I am just playing here :-) > > > > Gary > > > > > > On Mon, Jun 12, 2017 at 2:29 PM, Matt Sicker <boa...@gmail.com> wrote: > > > > > I already see a huge difference here: Spark requires a bunch of > > > infrastructure to be set up, while this library is just a library. > > Similar > > > to Kafka Streams versus Spark Streaming or Flink or Storm or Samza or > the > > > others. > > > > > > On 12 June 2017 at 16:28, Gary Gregory <garydgreg...@gmail.com> wrote: > > > > > > > On Mon, Jun 12, 2017 at 2:26 PM, Arun Mohan <strider90a...@gmail.com > > > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > Good afternoon. > > > > > > > > > > I have been working on a java generic parallel execution library > > which > > > > will > > > > > allow clients to execute methods in parallel irrespective of the > > number > > > > of > > > > > method arguments, type of method arguments, return type of the > method > > > > etc. > > > > > > > > > > Here is the link to the source code: > > > > > https://github.com/striderarun/parallel-execution-engine > > > > > > > > > > The project is in a nascent state and I am the only contributor so > > > far. I > > > > > am new to the Apache community and I would like to bring this > project > > > > into > > > > > Apache and improve, expand and build a developer community around > it. > > > > > > > > > > I think this project can be a sub project of Apache Commons since > it > > > > > provides generic components for parallelizing any kind of methods. > > > > > > > > > > Can somebody please guide me or suggest what other options I can > > > explore > > > > ? > > > > > > > > > > > > > Hi Arun, > > > > > > > > Thank you for your proposal. > > > > > > > > How would this be different from Apache Spark? > > > > > > > > Thank you, > > > > Gary > > > > > > > > > > > > > > > > > > Thanks, > > > > > Arun > > > > > > > > > > > > > > > > > > > > > -- > > > Matt Sicker <boa...@gmail.com> > > > > > >
