I have added this to the Tupelo library <https://github.com/cloojure/tupelo> as `tupelo.lazy/join`:
API docs: http://cloojure.github.io/doc/tupelo/tupelo.lazy.html source: (defn join "Lazily concatenates a sequence-of-sequences into a flat sequence." [sequences] (lazy-seq (when-let [seq-of-seqs (seq sequences)] (concat (first seq-of-seqs) (join (rest seq-of-seqs)))))) with tests: (dotest (is= [] (lazy/join [[]])) (is= [1] (lazy/join [[1]])) (is= [1 2 3 ] (lazy/join [[1] [2 3]])) (is= [1 2 3 4 5 6] (lazy/join [[1] [2 3] [4 5 6]])) (is= [1 2 3 4 5 6] (lazy/join [[] [1] [] [2 3] [4 5 6] []]))) On Wed, Jul 18, 2018 at 3:08 AM, Nicola Mometto <brobro...@gmail.com> wrote: > Yes, you’re right that neither directly handles `concat`, however CLJ-1583 > improves the behaviour of both your examples and is _necessary_ to fix any > `apply` related laziness issues > > user=> (first (apply concat (map #(do (println %) [%]) (list 1 2 3 4 5)))) > 1 > 2 > 3 > 1 > user=> (first (mapcat #(do (println %) [%]) (list 1 2 3 4 5))) > 1 > 2 > 3 > 1 > > While CLJ-1218 in conjunction with CLJ-1583 “fixes” the mapcat example: > > user=> (first (mapcat #(do (println %) [%]) (list 1 2 3 4 5))) > 1 > 1 > > So to make all your examples as lazy as possible we need a combination of > CLJ-1218, CLJ-1583 and a `join`-like `concat` > > > > > On 18 Jul 2018, at 09:24, Mark Engelberg <mark.engelb...@gmail.com> wrote: > > Thanks, I hadn't seen those issues. > https://dev.clojure.org/jira/browse/CLJ-1218 talks about mapcat, and > https://dev.clojure.org/jira/browse/CLJ-1583 talks about apply. Those are > aspects of the problem, for sure, but fixing those two issues would not > solve the problem with (apply concat ...), I don't think, because another > facet of the problem is that concat's args are of the form [x y & zs]. > > Gary Fredericks recognizes this problem in the comments for CLJ-1218: "I > realized that concat could actually be made lazier without changing its > semantics, if it had a single [& args] clause that was then implemented > similarly to join above." > > But for the most part, the comments are focused on fixing mapcat by > implementing a new function, join, rather than fixing the problem with > concat directly. Seems better to fix concat, if possible. > > I'm truly astonished I haven't gotten bitten by this before. This is a > pattern I use frequently in my code; I guess I just never had deep enough > recursion for it to slow things down enough for me to realize what was > happening. > > > > On Wed, Jul 18, 2018 at 12:53 AM, Nicola Mometto <brobro...@gmail.com> > wrote: > >> This behaviour is known and there are a couple of tickets about it : >> >> https://dev.clojure.org/jira/browse/CLJ-1583 >> https://dev.clojure.org/jira/browse/CLJ-1218 >> >> On Wed, 18 Jul 2018, 08:28 Mark Engelberg, <mark.engelb...@gmail.com> >> wrote: >> >>> I'm kind of surprised I haven't run across this before, but tonight I >>> was debugging a function that was doing an explosion of computation to >>> return the first value of a lazy sequence, and I was able to reduce the >>> problem down to this example: >>> >>> > (first (apply concat (map #(do (println %) [%]) (list 1 2 3 4 5)))) >>> 1 >>> 2 >>> 3 >>> 4 >>> 1 >>> >>> The last 1 is the return value, but notice that it realized 4 values in >>> order to return the 1. This has nothing to do with chunked sequences, by >>> the way -- a list is an unchunked sequence. It appears to be that the way >>> concat is written, it realizes the first two elements, and then another two >>> elements in a recursive call before the lazy-seq kicks in. >>> >>> In the function in question, the "apply concat" was part of a recursion, >>> causing that explosion of realizing values (four at each level of the >>> recursion, it would seem) to get at the first element. >>> >>> Note that this affects mapcat as well, which relies on concat under the >>> hood: >>> > (first (mapcat #(do (println %) [%]) (list 1 2 3 4 5))) >>> 1 >>> 2 >>> 3 >>> 4 >>> 1 >>> >>> I don't see a quick fix other than writing my own improved >>> concat/mapcat. Do you? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clojure@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+unsubscr...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
