Re: parallel sequence side-effect processor

Thanks everyone. There's a lot here--I'm still digesting it all. It's clear that a lot may depend on the underlying collections, and that, among other things, foldmap is worth considering for benchmarking in performance-critical code, as is mapping a vector across multiple sequences to combin

Re: parallel sequence side-effect processor

Just for the reference, this is the code with neanderthal, so we can get the difference when running with primitives. No native code has been used here, just pure Clojure. (def sv1 (sv v1)) (def sv2 (sv v2)) (defn side-effect-4 ^double [^double a ^double b] 1.0) (with-progress-reporting (quick

Re: parallel sequence side-effect processor

Francis, The times you got also heavily depend on the actual side-effect function, which in this case is much faster when called with one arg, instead of with varargs, that fluokitten need here. If we give fluokitten a function that does not create a sequence for multiple arguments, it is much

Re: parallel sequence side-effect processor

That's why I still think that in this particular case, there is no new sequence creation (by fluokitten). yes, it does call first/next but those do not require (significant) new memory or copying. They reuse the memory of the underlying vectors, if I understood that well. Or there is something else

Re: parallel sequence side-effect processor

sequence taps into the underlying TransfomerIterator which applies a transducer while iterating across multiple iterators at the same time. sequence wraps that into a chunked sequence, but you could tap directly into TransformerIterator/createMulti without using the outer sequence. Nothing in

Re: parallel sequence side-effect processor

Just a general note of caution. range is a highly specialized (and optimized) impl for both reducing and seq traversal. Also note that seqs on vecs are about the next-most optimized seq (because they also don't allocate or cache - rather they just pull from the vector nodes as the backing store

Re: parallel sequence side-effect processor

BTW I noticed that sequence got hotter and eventually became the fastest, but it took many more runs: (time (dorun (sequence (map side-effect) col1 col2))) "Elapsed time: 31.321698 msecs" => nil (time (dorun (sequence (map side-effect) col1 col2))) "Elapsed time: 15.492247 msecs" => nil (time (do

Re: parallel sequence side-effect processor

Well, you pay a cost whenever you use seqs instead of reduce, so it/s strange to think of doseq as "as fast as possible". If it/s argument is a true collection, its IReduce is usually faster than its seq. If it is a sequence already, its IReduce is usually just using seqs anyway. Let's get some

Re: parallel sequence side-effect processor

Francis is correct, and I want to make a point about something he said. If you are mapping over two collections at once, you will either be using Java iterators, or you will be creating lazy sequences when you walk over the input collections. Sure, how many lazy sequences are create here and there

Re: parallel sequence side-effect processor

Now I am on the REPL, and the solution is straightforward: (foldmap op nil println [1 2 3] [4 5 6]) gives: 1 4 2 5 3 6 nil The first function is a folding function. In this case we can use op, a monoid operation. Since nil is also a monoid, everything will be folded to nil. The second part is

Re: parallel sequence side-effect processor

Thanks very much Francis. So it sounds as if, if my data would already be in a vector, list or sequence (maybe a lazy sequence), doseq will process that structure as fast as possible, but there are other structures that might have faster internal reduce operations. -- You received this mess

Re: parallel sequence side-effect processor

> Francis, thanks for separating out the different types of intermediate > collections. I'm not entirely clear on type 1. It sounds like those are > just collections that are already there before processing them. Or are you > saying that Clojure has to convert them to a seq as such? Why wou

Re: parallel sequence side-effect processor

Thanks everyone--I'm very much appreciating the discussion, though I'm not sure I follow every point. Dragan, thank you very much for suggesting (and writing) foldmap. Very nice. I certainly don't mind using a library, though I still think there ought to be something like what I've describe

Re: parallel sequence side-effect processor

There are a few typos and defn is missing from fb in my last message, but I hope it is still readable. Sorry, I am typing this on a mobile device while watching the last episode of The Man in the High Castle :) Also, I am talking about the code I wrote years ago from the top of my mind without

Re: parallel sequence side-effect processor

A couple of things: 1. How fold/foldmap and any other function works, depends on the actual type. For example, if you look at https://github.com/uncomplicate/neanderthal/blob/master/src/clojure/uncomplicate/neanderthal/impl/fluokitten.clj#L396 you can see that there no intermediate allocations

Re: parallel sequence side-effect processor

Well, hold on, it is creating seqs for its input collection (allocation item 1 in my list), but it is still better than map because it is not making an intermediate lazy-seqs of output (item 4). (I must amend my earlier message, (map f coll1 col2) only pays 1 and 4) On Friday, September 23, 20

Re: parallel sequence side-effect processor

Yeah, I have to call you out on this one Dragan. I ran the following code: (ns fold-test (:require [uncomplicate.fluokitten.core :refer [foldmap]])) (defn fa [& args] (println "fa " args) 1) (defn fb [& args] (println "fb " args) 1) (defn test-fold [] (foldmap fa nil fb [1 2 3] [4 5 6])) (test

Re: parallel sequence side-effect processor

There are a few intermediate collections here: 1. The source coll may produce a seq object. How costly this is depends on the type of coll and the quality of its iterator/ireduce/seq implementations. 2. You may need to collect multiple source colls into a tuple-like thing to pro

Re: parallel sequence side-effect processor

fluokitten does not have anything to do with haskell, or how haskell does those things. It is completely tailored to clojure. Regarding marsi0's task, now I see that, actually, foldmap is what should be used, and not fold. so, it's something like (foldmap dummy-nil-accumulating-fn nil side-effe

Re: parallel sequence side-effect processor

How is this done, I searched the source, but there was so much indirection I can't find it. I'm familiar with how Haskell does rfold vs lfold, but one of those does create allocations, they may be in the form of closures, but they are allocations none the less. So does fold use iterators or somethi

Re: parallel sequence side-effect processor

fluokitten's fold is MUCH better than (map f a b) because it does NOT create intermediate collections. just use (fold f a b) and it would fold everything into one thing (in this case nil). If f is a function with side effects, it will invoke them. No intermediate collection is created AND the f

Re: parallel sequence side-effect processor

But that would print 1 to 32 items depending on a ton of things. No the correct way to write code like that is (doseq [s coll] (println s)) If you need to iterate over multiple items then use (map f a b). Also I would suggest benchmarking things, sure (map f coll) is slower than a transducer,

Re: parallel sequence side-effect processor

I've run into this when I wanted to print each item in a seq. Since println returns nil, I just did (first (keep println coll)). --Needed the first here because keep returns a lazy seq. On Friday, September 23, 2016 at 12:02:09 AM UTC-4, Mars0i wrote: > > This is almost the same as an issue I ra

Re: parallel sequence side-effect processor

How is fluokitten's fold any better than using seqs like (map f a b) would? Both create intermediate collections. On Fri, Sep 23, 2016 at 11:40 AM, Dragan Djuric wrote: > If you do not insist on vanilla clojure, but can use a library, fold from > fluokitten might enable you to do this. It is sim

Re: parallel sequence side-effect processor

If you do not insist on vanilla clojure, but can use a library, fold from fluokitten might enable you to do this. It is similar to reduce, but accepts multiple arguments. Give it a vararg folding function that prints what you need and ignores the first parameter, and you'd get what you asked fo

Re: parallel sequence side-effect processor

On Friday, September 23, 2016 at 11:11:07 AM UTC-5, Alan Thompson wrote: > > ​Huh. I was also unaware of the run! function.​ > > I suppose you could always write it like this: > > (def x (vec (range 3))) > (def y (vec (reverse x))) > > (run! > (fn [[x y]] (println x y)) > > (map vector x y)) >

Re: parallel sequence side-effect processor

​Huh. I was also unaware of the run! function.​ I suppose you could always write it like this: (def x (vec (range 3))) (def y (vec (reverse x))) (run! (fn [[x y]] (println x y)) (map vector x y)) > lein run 0 2 1 1 2 0 although the plain old for loop with dotimes looks simpler: (doti

Re: parallel sequence side-effect processor

On Friday, September 23, 2016 at 1:47:45 AM UTC-5, Francis Avila wrote: > > It's not crystal clear to me what you are after, either from this post or > the one you link to. I think you want a map that does not produce > intermediate collections and accepts multiple colls as input at a time? Ye

parallel sequence side-effect processor

It's not crystal clear to me what you are after, either from this post or the one you link to. I think you want a map that does not produce intermediate collections and accepts multiple colls as input at a time? Do you have some pseudocode example so we can be precise? What about (run! my-side-

parallel sequence side-effect processor

This is almost the same as an issue I raised in this group over a year and a half ago, here . I suggested that Clojure should include function with map's syntax but that was executed only for s