This might be a bug in java.jdbc. The code that passes the result set seq
to the function is:
(^{:once true} fn* [rset]
((^{:once true} fn* [rs]
(result-set-fn (if as-arrays?
(cons (first rs)
(map row-fn (rest rs)))
(map row-fn rs))))
(result-set-seq rset opts)))
I'm wondering if this function holds onto the head of the seq, since it's
bound to "rs".
On 20 June 2017 at 00:20, Luke Burton <luke_bur...@me.com> wrote:
>
> Anyone have any insights here? Really the most important thing I'm trying
> to learn is 2) how to identify when a lazy seq head is being retained,
> other than waiting for it to become bad enough that your program OOMs.
>
>
> On Jun 16, 2017, at 6:14 PM, Luke Burton <luke_bur...@me.com> wrote:
>
>
> Riddle me this:
>
> https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e
>
> Wherein I synthesize a large table in Postgres, then attempt to lazily
> load the table, discarding each row as I receive it. I tried *many*
> permutations and experiments, but settled on these two tests to illustrate
> my point. Which is that I simply can't get it to work with
> clojure.java.jdbc.
>
> test1, according to all my research and reading of the source code
> involved, should consume the query results lazily. It does not, and I can't
> for the life of me figure out why. Traffic starts to stream in, and the
> heap is overwhelmed almost immediately. I've deliberately set the heap to 1
> GB.
>
> test2 uses a technique I borrowed wholesale from Ghadi Shayban in JDBC-99
> <https://dev.clojure.org/jira/browse/JDBC-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs>,
> which is to have ResultSet implement IReduceInit. It consumes a nominal
> amount of memory. I've verified it's actually doing something by putting
> counters in, and using YourKit to watch about 20 MB/s of traffic streaming
> into the JVM. It's brilliant, it doesn't even break 200 MB total heap usage.
>
> I used YourKit to track where the memory is being retained for test1.
> Initially I made the mistake of not setting the fetchSize, so I saw an
> ArrayList inside the driver holding the reference. The driver
> documentation <https://jdbc.postgresql.org/documentation/head/query.html>
> confirms
> that autoCommit must be disabled and the fetchSize set to some non-zero
> number.
>
> After making that change, YourKit confirmed that the GC root holding all
> the memory was the stack local variable "rs". At least I think it did, as a
> non-expert in this domain. I tried disassembling the functions using
> no.disassemble and the IntelliJ decompiler but I'm not really at the point
> where I understand what to look for.
>
> So my questions are:
>
> 1) what am I doing wrong with clojure.java.jdbc?
>
> Note some things I've already tried:
>
> * using row-fn instead of result-set-fn
> * using prepared statements
> * explicitly setting auto-commit false on the connection
> * declaring my result-set-fn with (^{:once true} *fn […]) (I did not see a
> change in the disassembly when using this)
> * probably other things I am forgetting
>
> 2) in these situations where you suspect that the head of a lazy sequence
> is being retained, how do you reason about it? I'm kind of lucky this one
> blew the heap so quickly, who knows how much of my production code might
> burning memory unnecessarily but not quite as fatally. Do you disassemble
> the functions and observe some smoking gun? How do you peek under the
> covers to see where the problem is?
>
> Luke.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
James Reeves
booleanknot.com
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
