Hi Bob, Right now the API only allows for single puts, and fsyncing is all-or-nothing. However, this is just an artifact of my major use case for the library, which relies on upstream batching of tasks. I'm planning an 0.1.1 release which has an explicit `sync` method, and support for sync-intervals (i.e. sync twice a second) and sync-thresholds (i.e. sync every ten puts or takes). The use case you describe could be achieved by disabling automatic syncing, and doing a series of puts and takes followed by a call to `sync`.
If you have thoughts or suggestions on how this can be more useful for you, please let me know. Zach On Fri, Feb 7, 2014 at 5:26 AM, Bob Hutchison <hutch-li...@recursive.ca>wrote: > > On Feb 6, 2014, at 6:45 PM, Zach Tellman <z...@factual.com> wrote: > > At Factual we get a lot of data thrown at us, and often don't have control > over the rate at which it comes in. As such, it's preferable that our > buffer isn't bounded by the process' memory, since a temporary blip in > throughput may cause GC pauses, OOM exceptions, and other things that will > only exacerbate the problem. It's also preferable that if the process > dies, we won't lose any data which hasn't yet escaped the process. A > disk-backed queue satisfies both of these requirements. > > As such, I'm happy to announce that we're open sourcing 'durable-queue': > https://github.com/Factual/durable-queue. It's a small, fast, > pure-Clojure implementation that in our production systems is responsible > for processing billions of entries daily. We believe it has broad > applications, and are excited to see how others will use it. > > > What excellent timing! I've been looking at ZeroMQ, RabbitMQ, and Kafka > for the last week or so. ZMQ is awfully attractive for what I'm trying to > do, but there are a few things it doesn't do that I need done. I had begun > thinking of building something similar on top of Redis. > > You mention the idea of batching to reduce the impact of fsync. Is there > an API for batching puts? Is there a way to batch a complete! and put! new > tasks to the queue? > > One pattern that keeps coming up is: > - take a single task from the queue > - execute the task, which might generate a set of new tasks to be > queued on the same queue (and likely on other queues too) > - signal completion, and put the new tasks > > Cheers, > Bob > > > Zach > > P.S. If this sort of work is interesting to you, Factual is hiring: > https://groups.google.com/forum/#!searchin/clojure/factual/clojure/8bPIEnNpfyQ/lvv-9gkVozAJ > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to a topic in the > Google Groups "Clojure" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/clojure/4tZFWdMKvjw/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
