Hey Leif, When using :fsync-interval, the actual calls to .force() on the underlying ByteBuffers occur on another thread, making it effectively a background process. This is contrasted with :fsync-threshold, which will synchronously call fsync when the write threshold is hit. Note that if the fsync-interval is less than the time it takes to actually fsync, it will simply continuously fsync at whatever pace it can.
If you'd like to verify, I suggest using strace. Zach On Sunday, March 9, 2014 6:54:50 PM UTC-7, Leif wrote: > > Hi, Zach. > > I was trying to benchmark at different values of the :fysnc-* parameters, > and I noticed that it didn't matter what value of :fsync-interval I set, > the performance was constant, and about what it is with both :fsync-put? > and :fsync-take? disabled. > > Any suggestions on how to test if data is actually being synced to disk at > my specified interval? > > Please forgive my suspicious nature, > Leif > > On Friday, March 7, 2014 4:21:44 PM UTC-5, Zach Tellman wrote: >> >> I added the above-described features a few weeks back, but only got >> around to marking 0.1.1 today. Fsync batching is described at the end of >> the README, let me know if you have any questions. >> >> On Friday, February 7, 2014 11:52:11 AM UTC-8, Zach Tellman wrote: >>> >>> Hi Bob, >>> >>> Right now the API only allows for single puts, and fsyncing is >>> all-or-nothing. However, this is just an artifact of my major use case for >>> the library, which relies on upstream batching of tasks. I'm planning an >>> 0.1.1 release which has an explicit `sync` method, and support for >>> sync-intervals (i.e. sync twice a second) and sync-thresholds (i.e. sync >>> every ten puts or takes). The use case you describe could be achieved by >>> disabling automatic syncing, and doing a series of puts and takes followed >>> by a call to `sync`. >>> >>> If you have thoughts or suggestions on how this can be more useful for >>> you, please let me know. >>> >>> Zach >>> >>> >>> On Fri, Feb 7, 2014 at 5:26 AM, Bob Hutchison <hutch...@recursive.ca>wrote: >>> >>>> >>>> On Feb 6, 2014, at 6:45 PM, Zach Tellman <za...@factual.com> wrote: >>>> >>>> At Factual we get a lot of data thrown at us, and often don't have >>>> control over the rate at which it comes in. As such, it's preferable that >>>> our buffer isn't bounded by the process' memory, since a temporary blip in >>>> throughput may cause GC pauses, OOM exceptions, and other things that will >>>> only exacerbate the problem. It's also preferable that if the process >>>> dies, we won't lose any data which hasn't yet escaped the process. A >>>> disk-backed queue satisfies both of these requirements. >>>> >>>> As such, I'm happy to announce that we're open sourcing >>>> 'durable-queue': https://github.com/Factual/durable-queue. It's a >>>> small, fast, pure-Clojure implementation that in our production systems is >>>> responsible for processing billions of entries daily. We believe it has >>>> broad applications, and are excited to see how others will use it. >>>> >>>> >>>> What excellent timing! I’ve been looking at ZeroMQ, RabbitMQ, and Kafka >>>> for the last week or so. ZMQ is awfully attractive for what I’m trying to >>>> do, but there are a few things it doesn’t do that I need done. I had begun >>>> thinking of building something similar on top of Redis. >>>> >>>> You mention the idea of batching to reduce the impact of fsync. Is >>>> there an API for batching puts? Is there a way to batch a complete! and >>>> put! new tasks to the queue? >>>> >>>> One pattern that keeps coming up is: >>>> - take a single task from the queue >>>> - execute the task, which might generate a set of new tasks to be >>>> queued on the same queue (and likely on other queues too) >>>> - signal completion, and put the new tasks >>>> >>>> Cheers, >>>> Bob >>>> >>>> >>>> Zach >>>> >>>> P.S. If this sort of work is interesting to you, Factual is hiring: >>>> https://groups.google.com/forum/#!searchin/clojure/factual/clojure/8bPIEnNpfyQ/lvv-9gkVozAJ >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To post to this group, send email to clo...@googlegroups.com >>>> Note that posts from new members are moderated - please be patient with >>>> your first post. >>>> To unsubscribe from this group, send email to >>>> clojure+u...@googlegroups.com >>>> For more options, visit this group at >>>> http://groups.google.com/group/clojure?hl=en >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to clojure+u...@googlegroups.com. >>>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To post to this group, send email to clo...@googlegroups.com >>>> Note that posts from new members are moderated - please be patient with >>>> your first post. >>>> To unsubscribe from this group, send email to >>>> clojure+u...@googlegroups.com >>>> For more options, visit this group at >>>> http://groups.google.com/group/clojure?hl=en >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Clojure" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/clojure/4tZFWdMKvjw/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> clojure+u...@googlegroups.com. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
