On Thu, Jan 29, 2009 at 7:28 PM, Greg Harman <ghar...@gmail.com> wrote:
> > Hank: > > I have looked at TC in the past, and took another look today at your > suggestion. Terracotta certainly seems to have promise feature-wise, > but I have to admit it's a "heavier" solution than I had been thinking > of, and there are probably all sorts of gotchas (and reviewing old > threads on the topic, a few are hinted at) when it comes to > distributing some of Clojure's concurrency data structures in the > shared-memory approach. > Of course until someone tries we won't know where the limitations are for sure. But my thought is that even if terracotta turns out not to be ideal as a mechanism to implement everything, as I see it, the terracotta feature set (or hazelcast) is critical to anything like this, and I can't see re-engineering all that work because you need/want certain things that may need to be done outside it. Regarding it being "heavier", my feeling is that for what is really needed, it really is a "heavy" problem. Hank > > Kevin: > > I hadn't heard of Hadoop before, but at first glance it's exactly what > I'm looking for. (Thanks!) The problem I am working on is a processing > pipeline for massive data sets, and that seems to be the advertised > use case for Hadoop. > > On Jan 29, 7:14 pm, Kevin Downey <redc...@gmail.com> wrote: > > have you looked at the available java frameworks like hadoop? there is > > also some kind of java interface to erlang > > instead of reinventing the wheel again... > > > > > > > > On Thu, Jan 29, 2009 at 6:15 AM, Greg Harman <ghar...@gmail.com> wrote: > > > > > One of Clojure's big selling points (obviously) is the support for > > > concurrent programming. However, the performance gains you get with > > > this concurrency hits a scalability wall once you're fully utilizing > > > all the cores available in your server. The next step, of course, is > > > to add additional servers. > > > > > So, I've been mulling over the idea of putting together a framework > > > for distributed applications. I think it should deal with issues such > > > as: > > > > > * Locating, assessing, and registering CPUs > > > * Division of large jobs into smaller sub-jobs > > > * Dispatching of jobs, including optimization planning (send more jobs > > > to faster CPUs) > > > * Failure recovery > > > * Seamless code integration (so that this can be retrofit as an > > > optimization) both "horizontally" and "vertically" > > > * Horizontal = take one time-consuming task and parallelize it > > > * Vertical = take two sequential tasks and pipeline them across > > > different servers > > > > > Is anybody already working on something similar? Is this already on > > > the Clojure language roadmap as something to be added after 1.0? > > > > > Any design advice or feature requests if I were to put something like > > > this together? > > > > > -Greg > > > > -- > > And what is good, Phaedrus, > > And what is not good— > > Need we ask anyone to tell us these things? > > > -- blog: whydoeseverythingsuck.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~----------~----~----~----~------~----~------~--~---
