The problem is likely in the MySQL's JDBC driver. Some retrieve the
entire result set from the database on statement execute while others
are more true to the notion of a remote database cursor.
The JDBC API has a workaround for this problem: Use
Statement.setFetchSize(int) to limit the number of rows the JDBC
driver will try to retrieve from the database at any one time.
On May 14, 4:13 pm, Ian Eure <ian.e...@gmail.com> wrote:
> I'm trying to process mid/large result sets with Clojure, and not
> having any success.
>
> (ns foo
> (:require [clojure.contrib.sql :as sql]))
>
> (def *db* {:classname "com.mysql.jdbc.Driver"
> :subprotocol "mysql"
> :subname "//DSN"
> :user "read"
> :password "swordfish"})
>
> (sql/with-connection *db*
> (sql/with-query-results rs ["SELECT * FROM Foo;"]
> (prn "Done")))
>
> When I run this at the REPL, it dies with:
> java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0)
>
> I don't understand why this is OOMing. Even if it were trying to
> buffer the whole result set - which I _really_ hope it's not - the
> default 128mb JVM memory limit should be enough to hold it.
>
> My dataset is 155k rows, each anywhere from 300 bytes to 1kb. It
> should be 50-75mb for the whole thing. This is test data, and my full
> dataset is 25m rows, so increasing the memory limit isn't a good
> solution for me.
>
> Any ideas?
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To post to this group, send email to clojure@googlegroups.com
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
