On 05/27/2015 09:33 AM, Jan Schermer wrote:
Hi Nick,
responses inline, again ;-)
Thanks
Jan
On 27 May 2015, at 12:29, Nick Fisk <n...@fisk.me.uk> wrote:
Hi Jan,
Responses inline below
-----Original Message-----
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
Jan Schermer
Sent: 25 May 2015 21:14
To: Nick Fisk
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Synchronous writes - tuning and some thoughts
about them?
Hi Nick,
flashcache doesn’t support barriers, so I haven’t even considered it. I used a
few years ago to speed up some workloads out of curiosity and it worked
well, but I can’t use it to cache this kind of workload.
EnhanceIO passed my initial testing (although the documentation is very
sketchy and the project abandoned AFAIK), and is supposed to respect
barriers/flushes. I was only interested in a “volatile cache” scenario - create
a
ramdisk in the guest (for example 1GB) and use it to cache the virtual block
device (and of course flush and remove it before rebooting). All worked
pretty well during my testing with fio & stuff until I ran the actual workload -
in my case a DB2 9.7 database. It took just minutes for the kernel to panic (I
can share a screenshot if you’d like). So it was not a host failure but a guest
failure and it managed to fail on two fronts - stability and crash consistency -
at the same time. The filesystem was completely broken afterwards - while it
could be mounted “cleanly” (journal appeared consistent), there was
massive damage to the files. I expected the open files to be zeroed or
missing or damaged, but it did veryrandom damage all over the place
including binaries in /bin, manpage files and so on - things that nobody was
even touching. Scary.
I see, so just to confirm you don't want to use a caching solution with an SSD,
just a ram disk? I think that’s where are approaches differed and can
understand why you are probably having problems when the OS crashes or suffers
powerloss. I was going the SSD route, with something like:-
This actually proves that EnhanceIO doesn’t really respect barriers, at least
not when flushing blocks to the underlying device.
To be fair, maybe using a (mirrored!) SSD makes it crash-consistent, maybe it
has an internal journal and just replays whatever is in cache - I will not read
the source code to confirm that because to me that’s clearly not what I need.
FWIW, I think both dm-cache and bcache properly respect barriers, though
I haven't read through the source.
http://www.storagereview.com/hgst_ultrastar_ssd800mm_enterprise_ssd_review
On my iSCSI head nodes, but if you are exporting RBD's to lots of different
servers I guess this wouldn't work quite the same.
Exactly. If you want to maintain elasticity, want to be able to migrate
instances freely, then using any local storage is a no-go.
I don't really see a solution that could work for you without using SSD's for
the cache. You seem to be suffering from slow sync writes and want to cache
them in a volatile medium, but by their very nature sync writes are meant to be
safe once the write confirmation arrives. I guess in any caching solution
barriers go some length to help guard against data corruption but if properly
implemented they will probably also slow the speed down to what you can achieve
with just RBD caching. Much like Hardware Raid Controllers, they only enable
writeback cache if they can guarantee data security, either by a functioning
battery backup or flash device.
You are right. Sync writes and barriers are supposed to be flushed to physical
medium when returning (though in practice lots of RAID controllers and _all_
arrays will lie about that, slightly breaking the spec but still being safe if
you don’t let the battery die).
I don’t want to lose crash consistency, but I don’t need to have the latest
completed transaction flushed to the disk - I don’t care if power outage wipes
the last 1 minute of records from the database even though they were “commited”
by database and should thus be flushed to disks, and I don’t think too many
people care either as long as it’s fast.
Of course, this won’t work for everyone and in that respect the current rbd
cache behaviour is 100% correct.
And of course it won’t solve all problems - if you have an underlying device
that can do 200 IOPS but your workload needs 300 IOPS at all times, then
caching the writes is a bit futile - it may help for a few seconds and then it
gets back to 200 IOPS at best. It might, however help if you rewrite the same
blocks again and again, incrementing a counter or updating one set of thata -
there it will just update the dirty block in cache and flush it from time to
time. It can also turn some random-io into sequential-io, coalescing adjacent
blocks into one re/write or journaling it in some way (CEPH journal does
exactly this I think).
I don’t really understand your question about flashcache - do you run it in
writeback mode? It’s been years since I used it so I won’t be much help here
- I disregarded it as unsafe right away because of barriers and wouldn’t use it
in production.
What I mean is that for every IO that passed through flashcache, I see it write
to the SSD with no delay/buffering. So from a Kernel Panic/Powerloss situation,
as long as the SSD has powerloss caps and the flashcache device is assembled
correctly before mouting, I don't see a way for data to be lost. Although I
haven't done a lot of testing around this yet, so I could be missing something.
That would be flashcache in writethrough or writearound mode - in writeback
mode it will always end up in cache and only after some time get flushed to
disk (how fast it will get flushed can be tuned, but it will always by async to
the originating IO by nature).
I don’t think a persistent cache is something to do right now, it would be
overly complex to implement, it would limit migration, and it can be done on
the guest side with (for example) bcache if really needed - you can always
expose a local LVM volume to the guest and use it for caching (and that’s
something I might end up doing) with mostly the same effect.
For most people (and that’s my educated guess) the only needed features
are that it needs to be fast(-er) and it needs to come up again after a crash
without recovering for backup - that’s something that could be just a slight
modification to the existing RBD cache - just don’t flush it on every fsync()
but maintain ordering - and it’s done? I imagine some ordering is there
already, it must be flushed when the guest is migrated, and it’s production-
grade and not just some hackish attempt. It just doesn’t really cache the
stuff that matters most in my scenario…
My initial idea was just to be able to specify a block device to use for
writeback caching in librbd. This could either be a local block device (dual
port sas for failover/cluster) or an iSCSI device if it needs to be shared
around a larger cluster of hypervisors...etc
Ideally though this would all be managed through Ceph with some sort of
"OSD-lite" device which is optimized for sync writes but misses out on a lot of
the distributed functionality of a full fat OSD. This way you could create a writeback
pool and then just specify it in the librbd config.
That’s the same as giving a local device to the guest for caching, just one
level up. The elasticity/migration problem is still there, and you need to
invent another mechanism to flush the cache when something happens - all this
work is done in RBD cache already, it just isn’t non-volatile.
CEPH tiering will one day accomplish this to an extent, but even if you have a
super-fast local network with super-fast SSDs, the trip-time will be the
limiting factor here - writing over Fibre-Channel to a SAN cache is very very
fast compared to what the write latencies are right now, even for SSD-backed
journals. I certainly see this happening one day, but not anytime soon.
I’d love to hear some thoughts from CEPH developers on this.
I think that right now the only place where this can be cached comparably fast
is RBD cache (just needs crash-consistency and true writeback mode).
Another way would be to return the write as completed when the IO reaches all
OSD buffers (DRBD does it in one mode) - this is safe as long as you don’t lose
power on all OSDs holding the replicas at the same time, and in the worst case
you only lose the most recent transactions.
The optimal solution is to vastly decrease journal write latency, I guess this
would require most work.
I wonder if cache=unsafe does what I want, but it’s hard to test that
assumption unless something catastrophic happens like it did with EIO…
I think this option just doesn't act on flushes so will likely lose
data....probably not what you want
This is exactly what I want. But there’s a difference between not flushing (you
lose 1 minute of writes but everything is consistent) and reordering the IO
(you end up with filesystem journal oblivious to the changes that might/or
might have not really happened)
A couple of other things. Have you tried:
- disabling authentication
- disabling in-memory logging
Another area that we probably want to look at is CRC. you can only
disable this in the messenger, not the journal, and disabling it isn't a
good general solution, but it would be worth investigation.
disabling authentication for instance can definitely have an effect on
sync performance. See Page 6 and Page 9 here:
http://nhm.ceph.com/Ceph_SSD_OSD_Performance.pdf
Mark
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
