Hi, Zach. I was trying to benchmark at different values of the :fysnc-* parameters, and I noticed that it didn't matter what value of :fsync-interval I set, the performance was constant, and about what it is with both :fsync-put? and :fsync-take? disabled.
Any suggestions on how to test if data is actually being synced to disk at my specified interval? Please forgive my suspicious nature, Leif On Friday, March 7, 2014 4:21:44 PM UTC-5, Zach Tellman wrote: > > I added the above-described features a few weeks back, but only got around > to marking 0.1.1 today. Fsync batching is described at the end of the > README, let me know if you have any questions. > > On Friday, February 7, 2014 11:52:11 AM UTC-8, Zach Tellman wrote: >> >> Hi Bob, >> >> Right now the API only allows for single puts, and fsyncing is >> all-or-nothing. However, this is just an artifact of my major use case for >> the library, which relies on upstream batching of tasks. I'm planning an >> 0.1.1 release which has an explicit `sync` method, and support for >> sync-intervals (i.e. sync twice a second) and sync-thresholds (i.e. sync >> every ten puts or takes). The use case you describe could be achieved by >> disabling automatic syncing, and doing a series of puts and takes followed >> by a call to `sync`. >> >> If you have thoughts or suggestions on how this can be more useful for >> you, please let me know. >> >> Zach >> >> >> On Fri, Feb 7, 2014 at 5:26 AM, Bob Hutchison <hutch...@recursive.ca>wrote: >> >>> >>> On Feb 6, 2014, at 6:45 PM, Zach Tellman <za...@factual.com> wrote: >>> >>> At Factual we get a lot of data thrown at us, and often don't have >>> control over the rate at which it comes in. As such, it's preferable that >>> our buffer isn't bounded by the process' memory, since a temporary blip in >>> throughput may cause GC pauses, OOM exceptions, and other things that will >>> only exacerbate the problem. It's also preferable that if the process >>> dies, we won't lose any data which hasn't yet escaped the process. A >>> disk-backed queue satisfies both of these requirements. >>> >>> As such, I'm happy to announce that we're open sourcing 'durable-queue': >>> https://github.com/Factual/durable-queue. It's a small, fast, >>> pure-Clojure implementation that in our production systems is responsible >>> for processing billions of entries daily. We believe it has broad >>> applications, and are excited to see how others will use it. >>> >>> >>> What excellent timing! I’ve been looking at ZeroMQ, RabbitMQ, and Kafka >>> for the last week or so. ZMQ is awfully attractive for what I’m trying to >>> do, but there are a few things it doesn’t do that I need done. I had begun >>> thinking of building something similar on top of Redis. >>> >>> You mention the idea of batching to reduce the impact of fsync. Is there >>> an API for batching puts? Is there a way to batch a complete! and put! new >>> tasks to the queue? >>> >>> One pattern that keeps coming up is: >>> - take a single task from the queue >>> - execute the task, which might generate a set of new tasks to be >>> queued on the same queue (and likely on other queues too) >>> - signal completion, and put the new tasks >>> >>> Cheers, >>> Bob >>> >>> >>> Zach >>> >>> P.S. If this sort of work is interesting to you, Factual is hiring: >>> https://groups.google.com/forum/#!searchin/clojure/factual/clojure/8bPIEnNpfyQ/lvv-9gkVozAJ >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clo...@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+u...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+u...@googlegroups.com. >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clo...@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+u...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Clojure" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/clojure/4tZFWdMKvjw/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> clojure+u...@googlegroups.com. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
