Konrad Hinsen <konrad.hin...@fastmail.net> writes: > Thanks for the link! Judging from the example in the README, it's a > library for task farming in Clojure. While that's a limited form of > parallelism, there are still lots of applications where it is useful, > so I'd say this library is definitely worth a closer look. However, it > doesn't seem to deal with distributed data.
Distributed data is hard, though, partly because kind of distribution you need depends on your calculation. Every time I've had to do a distributed calculations, I've always just used the filesystem for data. I see a lot of frameworks that assume the data is small and can be entirely contained in the "message," while I need some kind of data affinity. (I do model estimation on large data sets, so I'd like to send a lump of data to different nodes, leave it there, then exchange parameter vectors and error scores with a controller.) In today's world, I've found I get more done faster with a single 8-core machine with a lot of RAM (96 GB now; at a previous employer I had access to a 512 GB monster) than I would with a farm of machines with only 4 GB or 8 GB, so I'm back to concurrency. Of course, that's just because my data is large, but not too large. >> I come from the scientific computing community .. the likes of >> Computation Fluid Dynamics and related topics.. large matrix >> operations and such stuff.. > > My background is somewhat similar: molecular simulations and analysis > of large data sets. I did astronomy, but mostly small-scale stuff. Integration, cascade calculations, the like. These days, though, I'm doing finance, mortgages in particular. That's a field that's been fun for the past few years. -Johann -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
