Sam and Oliver,
We've had tons of issues with Dumpling rbd volumes showing sporadic
periods of high latency for Windows guests doing lots of small writes.
We saw the issue occasionally with Cuttlefish, but it got significantly
worse with Dumpling. Initial results with wip-dumpling-perf2 appear
Oops!
-Sam
On Thu, Aug 29, 2013 at 2:00 PM, Oliver Daudey wrote:
> Hey Mark,
>
> Sam must have mistaken me for someone else, as I'm not currently using
> any non-standard hardware- or software-assisted caching on any of the
> Ceph-clusters I manage.
>
>
>Regards,
>
> Oliver
>
> On do, 2
Hey Mark,
Sam must have mistaken me for someone else, as I'm not currently using
any non-standard hardware- or software-assisted caching on any of the
Ceph-clusters I manage.
Regards,
Oliver
On do, 2013-08-29 at 13:56 -0500, Mark Nelson wrote:
> Excellent news!
>
> Btw, Sam mentioned
Excellent news!
Btw, Sam mentioned you are using flashcache. Would you mind talking a
little bit about what you are doing and what kind of performance you
see? This is an area I've been wanting to explore but haven't found the
time yet.
Mark
On 08/29/2013 01:52 PM, Oliver Daudey wrote:
H
Hey Mark and list,
FYI for you and the list: Samuel and I seem to have found and fixed the
remaining performance-problems. For those who can't wait, fixes are in
"wip-dumpling-perf2" and will probably be in the next point-release.
Regards,
Oliver
On 27-08-13 17:13, Mark Nelson wrote:
Something ala:
https://lwn.net/Articles/340010/
On 08/27/2013 12:06 PM, Ian Colle wrote:
Oliver,
Could you please have perf generate a call graph?
That would be very interesting as we look deeper into this.
Thanks!
Ian
On 8/27/13 8:55 AM, "Oliver Daudey" wrote:
Hey Samuel,
These:
Oliver,
Could you please have perf generate a call graph?
That would be very interesting as we look deeper into this.
Thanks!
Ian
On 8/27/13 8:55 AM, "Oliver Daudey" wrote:
>Hey Samuel,
>
>These:
>osd op threads = 8
>osd disk threads = 2
>filestore op threads = 8
>
>T
Oliver,
This patch isn't in dumpling head yet, you may want to wait on a
dumpling point release.
-Sam
On Tue, Aug 27, 2013 at 8:13 AM, Mark Nelson wrote:
> Ok, definitely let us know how it goes! For what it's worth, I'm testing
> Sam's wip-dumpling-perf branch with the wbthrottle code disabled
Ok, definitely let us know how it goes! For what it's worth, I'm
testing Sam's wip-dumpling-perf branch with the wbthrottle code disabled
now and comparing it both to that same branch with it enabled along with
0.67.1. Don't have any perf data, but quite a bit of other data to look
through, b
Hey Mark,
That will take a day or so for me to know with enough certainty. With
the low CPU-usage and preliminary results today, I'm confident enough to
upgrade all OSDs in production and test the cluster all-Dumpling
tomorrow. For now, I only upgraded a single OSD and measured CPU-usage
and wha
On 08/27/2013 09:55 AM, Oliver Daudey wrote:
Hey Samuel,
These:
osd op threads = 8
osd disk threads = 2
filestore op threads = 8
They increased performance on my test-cluster, but seemed to have the
opposite effect on the much heavier loaded production-environment.
Hey Samuel,
These:
osd op threads = 8
osd disk threads = 2
filestore op threads = 8
They increased performance on my test-cluster, but seemed to have the
opposite effect on the much heavier loaded production-environment.
Regards,
Oliver
On 27-08-13 16:37, Samu
What options were you using?
-Sam
On Tue, Aug 27, 2013 at 7:35 AM, Oliver Daudey wrote:
> Hey Ian, Samuel,
>
> FYI: I still had some attempted optimization-options in place on the
> production-cluster, which might have skewed my results a bit. OSD
> version 0.67.2-16-geeb1f86 seems to be a lot l
Hey Ian, Samuel,
FYI: I still had some attempted optimization-options in place on the
production-cluster, which might have skewed my results a bit. OSD
version 0.67.2-16-geeb1f86 seems to be a lot less hard on the CPU in the
configuration that I did all other tests in. I haven't yet verified
suf
On Aug 27, 2013, at 2:08, Oliver Daudey wrote:
> Hey Samuel,
>
> The "PGLog::check()" is now no longer visible in profiling, so it helped
> for that. Unfortunately, it doesn't seem to have helped to bring down
> the OSD's CPU-loading much. Leveldb still uses much more than in
> Cuttlefish. O
Hi Olver/Matthew,
Ignoring CPU usage, has speed remained slower as well?
Mark
On 08/27/2013 03:08 AM, Oliver Daudey wrote:
Hey Samuel,
The "PGLog::check()" is now no longer visible in profiling, so it helped
for that. Unfortunately, it doesn't seem to have helped to bring down
the OSD's CPU-
Hey Samuel,
The "PGLog::check()" is now no longer visible in profiling, so it helped
for that. Unfortunately, it doesn't seem to have helped to bring down
the OSD's CPU-loading much. Leveldb still uses much more than in
Cuttlefish. On my test-cluster, I didn't notice any difference in the
RBD b
Hi Sam,
It looks like that has dropped the CPU usage a fair bit. CPU usage
still seems a bit higher than Cuttlefish but that might just be due to
the levelDB changes.
Here's the updated perf report -
Events: 80K cycles
17.25% ceph-osd libc-2.15.so [.] 0x15d534
14.63% ceph-osd cep
I just pushed a patch to wip-dumpling-log-assert (based on current
dumpling head). I had disabled most of the code in PGLog::check() but
left an (I thought) innocuous assert. It seems that with (at least)
g++ 4.6.3, stl list::size() is linear in the size of the list, so that
assert actually trave
Hi Guys,
I'm having the same problem as Oliver with 0.67.2. CPU usage is around
double that of the 0.61.8 OSD's in the same cluster which appears to
be causing the performance decrease.
I did a perf comparison (not sure if I did it right but it seems ok).
Both hosts are the same spec running Ubun
Hey Samuel,
Nope, "PGLog::undirty()" doesn't use as much CPU as before, but I found
it curious that it still showed up, as I thought you disabled it. As
long as you can reproduce the abnormal leveldb CPU-usage. Let me know
if I can help with anything.
Regards,
Oliver
On ma, 2013-08-2
Saw your logs. I thought it might be enabling
filestore_xattr_use_omap, but it isn't. PGLog::undirty() doesn't seem
to be using very much cpu.
-Sam
On Mon, Aug 26, 2013 at 2:04 PM, Oliver Daudey wrote:
> Hey Samuel,
>
> I have been trying to get it reproduced on my test-cluster and seem to
> ha
Hey Samuel,
I have been trying to get it reproduced on my test-cluster and seem to
have found a way. Try: `rbd bench-write test --io-threads 80
--io-pattern=rand'. On my test-cluster, this closely replicates what I
see during profiling on my production-cluster, including the extra
CPU-usage by l
Can you attach a log from the startup of one of the dumpling osds on
your production machine (no need for logging, just need some of the
information dumped on every boot)?
libleveldb is leveldb. We've used leveldb for a few things since
bobtail. If anything, the load on leveldb should be lighter
Hey Samuel,
Unfortunately, disabling "wbthrottle" made almost no difference on my
production-cluster. OSD-load was still much higher on Dumpling.
I've mentioned this several times already, but when profiling with `perf
top' on my production-cluster, any time I'm running a Dumpling-OSD,
several "
Hey Samuel,
Before we go any further, I will try "wip-dumpling-perf" on my
production-platform with "wbthrottle" on and off, because this problem
seems very elusive and we might be chasing the wrong symptoms on my much
less busy test-platform. I'll get back to you. Many thanks for your
help so f
Hey Samuel,
Ok, here are the results.
wip-dumpling-perf, filestore_op_threads = 1, wbthrottle on:
# rbd bench-write test --io-pattern=rand
bench-write io_size 4096 io_threads 16 bytes 1073741824 pattern rand
SEC OPS OPS/SEC BYTES/SEC
1 666665.67 1948743.06
2 1
Ok, can you try setting filestore_op_threads to 1 on both cuttlefish
and wip-dumpling-perf (with and with wbthrottle, default wbthrottle
settings). I suspect I created contention in the filestore op threads
(FileStore::lfn_open specifically), and if so setting it to only use 1
thread should even o
Hey Samuel,
I commented the earlier settings out, so it was with defaults.
Regards,
Oliver
On vr, 2013-08-23 at 13:35 -0700, Samuel Just wrote:
> When you were running with the wbthrottle on, did you have the
> settings I gave you earlier set, or was it using the defaults?
> -Sam
>
>
When you were running with the wbthrottle on, did you have the
settings I gave you earlier set, or was it using the defaults?
-Sam
On Fri, Aug 23, 2013 at 12:48 PM, Oliver Daudey wrote:
> Hey Samuel,
>
> That changed something, for the better. :-)
>
> Your test-version, with wbthrottle off:
> # c
Hey Samuel,
That changed something, for the better. :-)
Your test-version, with wbthrottle off:
# ceph-osd --version
ceph version 0.67.1-18-g3fe3368
(3fe3368ac7178dcd312e89d264d8d81307e582d8)
# ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config show | grep
wbthrottle_enable
"filestore_wbt
I pushed a branch, wip-dumpling-perf. It does two things:
1) adds a config filestore_wbthrottle_enable (defaults to true) to
allow you to disable the wbthrottle altogether
2) causes the wbthrottle when enabled to fdatasync rather than fsync.
Can you rerun the random workload with that branch with
Hey Sage,
I'm all for it and will help testing.
Regards,
Oliver
On 22-08-13 17:23, Sage Weil wrote:
> We should perhaps hack the old (cuttlefish and earlier) flushing behavior
> into the new code so that we can confirm that it is really the writeback
> that is causing the problem an
Jumping in pretty late on this thread, but I can confirm much higher CPU
load on ceph-osd using 0.67.1 compared to 0.61.7 under a write-heavy RBD
workload. Under my workload, it seems like it might be 2x-5x higher CPU
load per process.
Thanks,
Mike Dawson
On 8/22/2013 4:41 AM, Oliver Daudey
For what it's worth, I was still seeing some small sequential write
degradation with kernel RBD with dumpling, though random writes were not
consistently slower in the testing I did. There was also some variation
in performance between 0.61.2 and 0.61.7 likely due to the workaround we
had to i
We should perhaps hack the old (cuttlefish and earlier) flushing behavior
into the new code so that we can confirm that it is really the writeback
that is causing the problem and not something else...
sage
On Thu, 22 Aug 2013, Oliver Daudey wrote:
> Hey Samuel,
>
> On wo, 2013-08-21 at 20:27
Hey Samuel,
On wo, 2013-08-21 at 20:27 -0700, Samuel Just wrote:
> I think the rbd cache one you'd need to run for a few minutes to get
> meaningful results. It should stabilize somewhere around the actual
> throughput of your hardware.
Ok, I now also ran this test on Cuttlefish as well as Dumpl
I think the rbd cache one you'd need to run for a few minutes to get
meaningful results. It should stabilize somewhere around the actual
throughput of your hardware.
Hmm, 10k ios I guess is only 10 rbd chunks. What replication level
are you using? Try setting them to 1000 (you only need to
Hey Samuel,
On wo, 2013-08-21 at 18:33 -0700, Samuel Just wrote:
> I am dumb. There *has* been a change in the osd which can account for
> this: the wbthrottle limits. We added some logic to force the kernel
> to start flushing writes out earlier, normally a good thing. In this
> case, it's pro
I am dumb. There *has* been a change in the osd which can account for
this: the wbthrottle limits. We added some logic to force the kernel
to start flushing writes out earlier, normally a good thing. In this
case, it's probably doing an fsync every 500 writes.
Can you run 3 tests?
1) rerun with
You'd need to unmount the fs to actually clear the cache. Did you see
a significant difference in load between the runs? To confirm, the
rbd client is dumpling the entire time?
-Sam
On Wed, Aug 21, 2013 at 2:28 PM, Oliver Daudey wrote:
> Hey Samuel,
>
> I repeated the same test several times be
Hey Samuel,
I repeated the same test several times before my post and just now 2
more times. It holds up and is also repeatable in reverse order, with
the same results. Remember, I restart all OSDs between tests, so any
caches should get destroyed and besides, I'm writing. That shouldn't
involv
Try it again in the reverse order, I strongly suspect caching effects.
-Sam
On Wed, Aug 21, 2013 at 1:34 PM, Oliver Daudey wrote:
> Hey Samuel,
>
> Finally got it reproduced on my test-cluster, which was otherwise
> unloaded at the time. First, with Dumpling:
>
> # rbd create --size 102400 test
Hey Samuel,
Finally got it reproduced on my test-cluster, which was otherwise
unloaded at the time. First, with Dumpling:
# rbd create --size 102400 test
# ceph-osd --version
ceph version 0.67.1-6-g0c4f2f3
(0c4f2f34b78b634efe7f4d56694e2edeeda5a130)
# rbd bench-write test
bench-write io_size 409
Hey Samuel,
CPU-usage still seems a bit higher, but not always equally on every OSD.
I profiled the node with the most CPU-usage on the OSD. Note the
libleveldb-related stuff right at the top. The Cuttlefish-OSD doesn't
show those at all. Could those be related to the problem?
OSD version 0.6
There haven't been any significant osd side changes that I can think
of. Is cpu usage still high? If so, can you post the profiler
results again?
-Sam
On Wed, Aug 21, 2013 at 12:02 PM, Oliver Daudey wrote:
> Hey Samuel,
>
> I had a good run on the production-cluster with it and unfortunately, i
Hey Samuel,
I had a good run on the production-cluster with it and unfortunately, it
still doesn't seem to have solved the problem. It seemed OK for a while
and individual OSD CPU-usage seemed quite low, but as the cluster's load
increased during the day, things got slower again. Write-performan
Good to hear.
-Sam
On Tue, Aug 20, 2013 at 2:55 PM, Oliver Daudey wrote:
> Hey Samuel,
>
> I picked up 0.67.1-10-g47c8949 from the GIT-builder and the osd from
> that seems to work great so far. I'll have to let it run for a while
> longer to really be sure it fixed the problem, but it looks pro
Hey Samuel,
I picked up 0.67.1-10-g47c8949 from the GIT-builder and the osd from
that seems to work great so far. I'll have to let it run for a while
longer to really be sure it fixed the problem, but it looks promising,
not taking any more CPU than the Cuttlefish-osds. Thanks! I'll get
back to
Hey Mark,
I'm working on a simple way to reproduce this if we fail to identify it
and `rados bench' indeed looks promising, as it can also simulate many
concurrent ops, which is probably what sets the 3 VM load on my
test-cluster apart from the +-80 VM load on my production-cluster. I'll
look int
Hi Oliver,
Have you ever tried using RADOS bench?
On one of the client nodes, you can do:
rados -p -b bench write 120 -t
It would be useful to know if that is also slower without any of the RBD
code involved. Btw, what was the dd command line you were using for
testing?
Thanks,
Mark
Can you try dumpling head without the option?
-Sam
On Tue, Aug 20, 2013 at 1:44 AM, Oliver Daudey wrote:
> Hey Mark,
>
> Sorry, but after some more tests I have to report that it only worked
> partly. The load seems lower with "wip-dumpling-pglog-undirty" in
> place, but the Cuttlefish-osd still
Hey Mark,
Sorry, but after some more tests I have to report that it only worked
partly. The load seems lower with "wip-dumpling-pglog-undirty" in
place, but the Cuttlefish-osd still seems significantly faster and even
with "wip-dumpling-pglog-undirty" in place, things slow down way too
much under
Hey Mark,
If I look at the "wip-dumpling-pglog-undirty"-version with regular top,
I see a slightly higher base-load on the osd, with significantly more
and higher spikes in it than the Dumpling-osds. Looking with `perf
top', "PGLog::undirty()" is still there, although pulling significantly
less C
Hi Oliver,
Glad that helped! How much more efficient do the cuttlefish OSDs seem
at this point (with wip-dumpling-pglog-undirty)? On modern Intel
platforms we were actually hoping to see CPU usage go down in many cases
due to the use of hardware CRC32 instructions.
Mark
On 08/19/2013 03:0
Hey Samuel,
Thanks! I installed your version, repeated the same tests on my
test-cluster and the extra CPU-loading seems to have disappeared. Then
I replaced one osd of my production-cluster with your modified version
and it's config-option and it seems to be a lot less CPU-hungry now.
Although
You're right, PGLog::undirty() looks suspicious. I just pushed a
branch wip-dumpling-pglog-undirty with a new config
(osd_debug_pg_log_writeout) which if set to false will disable some
strictly debugging checks which occur in PGLog::undirty(). We haven't
actually seen these checks causing excessi
Hey Mark,
On za, 2013-08-17 at 08:16 -0500, Mark Nelson wrote:
> On 08/17/2013 06:13 AM, Oliver Daudey wrote:
> > Hey all,
> >
> > This is a copy of Bug #6040 (http://tracker.ceph.com/issues/6040) I
> > created in the tracker. Thought I would pass it through the list as
> > well, to get an idea i
On 08/17/2013 06:13 AM, Oliver Daudey wrote:
Hey all,
This is a copy of Bug #6040 (http://tracker.ceph.com/issues/6040) I
created in the tracker. Thought I would pass it through the list as
well, to get an idea if anyone else is running into it. It may only
show under higher loads. More info
Hey all,
This is a copy of Bug #6040 (http://tracker.ceph.com/issues/6040) I
created in the tracker. Thought I would pass it through the list as
well, to get an idea if anyone else is running into it. It may only
show under higher loads. More info about my setup is in the bug-report
above. Her
60 matches
Mail list logo
