On Monday, 14 March 2016 13:49:36 UTC+8, Leif wrote: > > > I also think the core.matrix crusade is unnecessary. Here are my two > cents: > > 1. No one jumps in every time someone writes a new web routing library > saying "No! You'll fragment the clojure > web routing community! Use compojure!" I think we should pick and > choose based on our needs. >
Hi Leif, Absolutely agree that we should choose libraries based on our needs. However there is huge value to having standard abstractions that everyone can code to if you want to create an ecosystem of compatible tools. Consider these examples: a) The Clojure "sequence" abstraction b) The Ring handler / middleware abstraction c) Transducers All of these present a common abstraction that can support many different implementations and use cases. core.matrix does exactly the same for array programming. If someone reinvented transducers with a completely different API / syntax but a clever new implementation (e.g. for distributed parallel processing or something like that) then people would quite rightly say things like "why don't you just make that a transducer? It would be much better if we can use that with all our existing transducer code". All I'm really asking is that people use the core.matrix abstractions when doing numerical work, and we can all interoperate much more smoothly. For individual end users it isn't so much of an issue, however it is a *big* problem if tools / higher level libraries start adopting different APIs and representations. Then it isn't "choose the right library for your needs" but "choose between different competing stacks of libraries that don't interoperate". That's when fragmentation becomes a real problem. If you are interested in this topic, it is well worth reading about "The Curse of Lisp". 2. We should build on well-tested, stable APIs, true. Dragan has built on > top of BLAS (basically stable for over > 35 years) and LAPACK (25-35 years depending on how you count). > There is a set of BLAS-like API functions in core.matrix already. See: https://github.com/mikera/core.matrix/blob/develop/src/main/clojure/clojure/core/matrix/blas.cljc Having said that, I don't personally think the BLAS API is a particularly good fit for Clojure (it depends on mutability, and I think it is a pretty clumsy design by modern API standards). But if you simply want to copy the syntax, it's certainly trivial to do in core.matrix. Note that none of these (core.matrix blas namespace, Neanderthal) are actually real BLAS (which is a native API that you can't call directly from Clojure). They are all just wrappers that adopt a similar syntax and delegate to the underlying implementation after a bit of parameter manipulation / data marshalling. > 3. Dragan has no need or desire to implement the core.matrix API, but I'm > sure someone that wants native speed > *and* compatibility with core.matrix will do so when the need arises. > 4. If you want accessibility to the widest possible community of numerical > programmers, bear in mind that most > of them aren't clojure or even java programmers anyway. BLAS and > LAPACK are the way to make them feel > at home. Pure-java numerical programming is a rather strange > cul-de-sac community already. > 5. Numerical programming is a field where you, unfortunately, have to drop > layers of abstraction more > frequently than other fields. I'd rather drop down into ATLAS than > <<insert random java backend with its own > matrix class here>>. > > In short, having two libraries that do the same thing is not a problem, > and if it becomes a problem, I think we as a community can deal with it > fairly quickly. > > --Leif > An important point to note is that they don't do the same thing at all: core.matrix is an API providing an general purpose array programming abstraction with pluggable implementation support. Neanderthal is a specific implementation tied to native BLAS/ATLAS. They should ideally work in harmony, not be seen as alternatives. Neanderthal is more closely comparable to Vectorz, which *is* a matrix implementation (and I think it matches or beats Neanderthal in performance for virtually every operation *apart* from large matrix multiplication for which ATLAS is obviously fantastic for). Ultimately, I'm trying to encourage people to work towards an ecosystem for Clojure(Script) that rivals the Python or R ecosystems. It's still going to take a fair bit of work to get there, but is perfectly feasible. I think we need: a) core.matrix as a common abstraction that people can use to develop user level code as well as higher level libraries (environments like Incanter, data processing tools, deep learning etc.) b) A really good pure JVM implementation (currently Vectorz / vectorz-clj is great for this) c) A really good native implementation backed by BLAS/ATLAS (Clatrix currently works, but this could be Neanderthal) d) A really good ClojureScript implementation (I know some folks have got core.matrix working in ClojureScript with thi-ng/ndarray which could be a good option) e) Some other specialised implementations for specific use cases (e.g. for interfacing with Spark) If anyone has other ideas / a better strategy I'd love to hear, and I welcome a good constructive debate. I'm not precious about any of my own contributions. But I do genuinely think this is the best way forward for Clojure data science overall, based on where we are right now. > On Sunday, March 13, 2016 at 8:19:25 PM UTC-4, Dragan Djuric wrote: >> >> On Monday, March 14, 2016 at 12:28:24 AM UTC+1, Mikera wrote: >> >>> It would be great if Neanderthal simply implemented the core.matrix >>> protocols, then people could use it as a core.matrix implementation for >>> situations where it makes sense. I really think it is an architectural >>> dead-end for Neanderthal to develop a separate API. You'll simply get less >>> users for Neanderthal and fragment the Clojure library ecosystem which >>> doesn't help anyone. >>> >> >> Mike, I explained that many times in detail what's wrong with >> core.matrix, and I think it is a bit funny that you jump in every time >> Neanderthal is mentioned with the same dreams about core.matrix, without >> even trying Neanderthal, or discussing the issues that I raised. Every time >> your answer is that core.matrix is fine for *YOUR* use cases. That's fine >> with me and I support your choice, but core.matrix fell short for *MY* use >> cases, and after detailed inspection I decided it was unsalvageable. If I >> thought I could improve it, it would have been easier for me to do that >> than to spend my time fiddling with JNI and GPU minutes. >> >> I understand your emotions about core.matrix, and I empathize with you. I >> support your contributions to Clojure open-source space, and am glad if >> core.matrix is a fine solution for a number of people. Please also >> understand that it is not a solution to every problem, and that it can also >> be an obstacle, when it fells short in a challenge. >> >> >>> In the absence of that, we'll just need to develop separate BLAS >>> implementations for core.matrix. >>> >> >> I support you. If you do a good job, I might even learn something now and >> improve Neanderthal. >> >> >>> Would be great if you could implement the core.matrix protocols and >>> solve this issue. It really isn't much work, I'd even be happy to do it >>> myself if Neanderthal worked on Windows (last time I tried it doesn't). >>> >> >> I am happy that it is not much work, since it will be easy for you or >> someone else to implement it ;) Contrary to what you said on slack, I am >> *not against it*. I said that many times. Go for it. The only thing that I >> said is that *I* do not have time for that nor I have any use of >> core.matrix. >> >> Regarding Windows - Neanderthal works on Windows. I know this because a >> student of mine compiled it (he's experimenting with an alternative GPU >> backend for Neanderthal and prefers to work on Windows). As I explained to >> you in the issue that you raised on GitHub last year, You have to install >> ATLAS on your machine, and Neanderthal has nothing un-Windowsy in its code. >> There is nothing Neanderthal specific there, it is all about comiling >> ATLAS. Follow any ATLAS or Nympu + ATLAS or R + ATLAS guide for >> instructions. Many people did that installation, so I doubt it'd be a real >> obstacle for you. >> >> >> >>> On Sunday, 13 March 2016 23:34:23 UTC+8, Dragan Djuric wrote: >>>> >>>> I am soon going to release a new version of Neanderthal. >>>> >>>> I reinstalled ATLAS, so I decided to also update benchmarks with >>>> threaded ATLAS bindings. >>>> >>>> The results are still the same for doubles on one core: 10x faster than >>>> Vectorz and 2x faster than JBlas. >>>> >>>> The page now covers some more cases: multicore ATLAS (on my 4-core >>>> i7-4790k) and floats. Neanderthal is 60 times faster with multi-threaded >>>> ATLAS and floats than Vectorz (core.matrix). >>>> >>>> For the rest of the results, please follow the link. This will work >>>> with older versions of Neanderthal. >>>> >>>> >>>> https://www.reddit.com/r/Clojure/comments/4a8o9n/new_matrix_multiplication_benchmarks_neanderthal/ >>>> http://neanderthal.uncomplicate.org/articles/benchmarks.html >>>> >>> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
