Re: Unexpected performace of transducers

Yes. > On 17 Dec 2017, at 15:56, Matan Safriel wrote: > > Thanks Gary, Alex, Didier, I learned a lot from your replies. > As per iota, I assume we're talking about this library of course. > > > On Sun, Dec 17, 2017 at 2:40 PM, Gary Verhaegen > wrote: > The iota library implements a reducible

Re: Unexpected performace of transducers

Thanks Gary, Alex, Didier, I learned a lot from your replies. As per iota, I assume we're talking about this library of course . On Sun, Dec 17, 2017 at 2:40 PM, Gary Verhaegen wrote: The iota library implements a reducible object on top of files. It may be wor

Re: Unexpected performace of transducers

The iota library implements a reducible object on top of files. It may be worth trying out for your use-case. > On 17 Dec 2017, at 00:32, Alex Miller wrote: > > > >> On Saturday, December 16, 2017 at 2:39:14 PM UTC-6, Matan wrote: >> Hi, >> >> As this thread seems to have been going down th

Re: Unexpected performace of transducers

On Saturday, December 16, 2017 at 2:39:14 PM UTC-6, Matan wrote: > > Hi, > > As this thread seems to have been going down this path, I am joining it > after having spent some time fiddling the source code of some clojure.core > transducers and familiarizing with how to create, compose and use

Re: Unexpected performace of transducers

| I didn't fully grasp the difference between self-reducible collections v.s. other ones (in this context, and in general). Can you please delineate? I think this means that collections which implement CollReduce have an implementation of reduce which is faster than just calling first and next

Re: Unexpected performace of transducers

Hi, As this thread seems to have been going down this path, I am joining it after having spent some time fiddling the source code of some clojure.core transducers and familiarizing with how to create, compose and use transducers in transducing processes. By the way I think the reference

Re: Unexpected performace of transducers

> On 28 Nov 2017, at 02:54, Alex Miller wrote: > > I would say transducers are preferable when: > > 1) you have reducible collections > 2) you have a lot of pipelined transformations (transducers handle these in > one pass with no intermediate data) > 3) the number of elements is "large" (this

Re: Unexpected performace of transducers

> On 28 Nov 2017, at 02:33, Jiacai Liu wrote: > > > Also, most of the performance boost from transducers is due to less garbage > > being created, and some times the heap of the JVM is so large you'll never > > see much change from switching to transducers. > > Thanks for this tip. I seldom

Re: Unexpected performace of transducers

Thanks Alex. Good summary. I think this can be added to https://clojure.org/reference/transducers On Tue, Nov 28, 2017 at 10:54 AM, Alex Miller wrote: > I would say transducers are preferable when: > > 1) you have reducible collections > 2) you have a lot of pipelined transformations (transducer

Re: Unexpected performace of transducers

I would say transducers are preferable when: 1) you have reducible collections 2) you have a lot of pipelined transformations (transducers handle these in one pass with no intermediate data) 3) the number of elements is "large" (this amplifies the memory and perf savings from #2) 4) you put to p

Re: Unexpected performace of transducers

> Also, most of the performance boost from transducers is due to less garbage being created, and some times the heap of the JVM is so large you'll never see much change from switching to transducers. Thanks for this tip. I seldom use transducers in my daily work, and I was convinced transducer

Re: Unexpected performace of transducers

Chunked seqs are annoyingly fast. :) On Monday, November 27, 2017 at 3:07:10 PM UTC-6, tbc++ wrote: > > >> Also, I think the transducer version should always be faster, no matter > >> the > size of the source collection (no threshold). > > It's a bit more complicated than that, mostly because tr

Re: Unexpected performace of transducers

>> Also, I think the transducer version should always be faster, no matter the size of the source collection (no threshold). It's a bit more complicated than that, mostly because transducer pipelines require about 2 allocations per step during creation. Also, most of the performance boost from tra

Re: Unexpected performace of transducers

Jiacai – I saw you updated the gist. Just in case it passed you by: performance profits from the source collection being reducible. So pouring ‘dataset’ into a vector beforehand should speed up the processing quite a bit. Also, I think the transducer version should always be faster, no matter the