Hi Gary, Thanks for your response. You have some valid and interesting points :-) Of course you are right that Spark is much more mature. Thanks for your insight. It will be interesting indeed to find out if the core parallelization engine of Spark can be isolated like you suggest.
I started working on this project because I felt that there was no good library for parallelizing method calls which can be plugged in easily into an existing java project. Ultimately, if such a solution can be incorporated in the Apache Commons, it would be a useful addition to the Commons repository. Thanks, Arun On Mon, Jun 12, 2017 at 3:01 PM, Gary Gregory <garydgreg...@gmail.com> wrote: > Hi Arun, > > Sure, and that is to be expected, Spark is more mature than a four class > prototype. What I am trying to get to is that in order for the library to > be useful, you will end up with more in a first release, and after a couple > more releases, there will be more and more. Would Spark not have in its > guts the same kind of code your are proposing here? By extension, will you > not end up with more framework-like (Spark-like) code and solutions as > found in Spark? I am just playing devil's advocate here ;-) > > > What would be interesting would be to find out if there is a core part of > Spark that is separable and ex tractable into a Commons component. Since > Spark has a proven track record, it is more likely, that such a library > would be generally useful than one created from scratch that does not > integrate with anything else. Again, please do not take any of this > personally, I am just playing here :-) > > Gary > > > On Mon, Jun 12, 2017 at 2:29 PM, Matt Sicker <boa...@gmail.com> wrote: > > > I already see a huge difference here: Spark requires a bunch of > > infrastructure to be set up, while this library is just a library. > Similar > > to Kafka Streams versus Spark Streaming or Flink or Storm or Samza or the > > others. > > > > On 12 June 2017 at 16:28, Gary Gregory <garydgreg...@gmail.com> wrote: > > > > > On Mon, Jun 12, 2017 at 2:26 PM, Arun Mohan <strider90a...@gmail.com> > > > wrote: > > > > > > > Hi All, > > > > > > > > Good afternoon. > > > > > > > > I have been working on a java generic parallel execution library > which > > > will > > > > allow clients to execute methods in parallel irrespective of the > number > > > of > > > > method arguments, type of method arguments, return type of the method > > > etc. > > > > > > > > Here is the link to the source code: > > > > https://github.com/striderarun/parallel-execution-engine > > > > > > > > The project is in a nascent state and I am the only contributor so > > far. I > > > > am new to the Apache community and I would like to bring this project > > > into > > > > Apache and improve, expand and build a developer community around it. > > > > > > > > I think this project can be a sub project of Apache Commons since it > > > > provides generic components for parallelizing any kind of methods. > > > > > > > > Can somebody please guide me or suggest what other options I can > > explore > > > ? > > > > > > > > > > Hi Arun, > > > > > > Thank you for your proposal. > > > > > > How would this be different from Apache Spark? > > > > > > Thank you, > > > Gary > > > > > > > > > > > > > > Thanks, > > > > Arun > > > > > > > > > > > > > > > -- > > Matt Sicker <boa...@gmail.com> > > >
