On 01/13/2014 10:39 PM, ?????? wrote:
Hi Josh
Thanks for your reply.
librbd version : 0.67.5-1
/etc/ceph/ceph.conf config file:
the ip and hostname was hiddened.
[global]
;open auth.
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
;global pid & log setting.
admin socket = /home/ceph/var/run/$cluster-$name.asok
[mon]
keyring = /home/ceph/var/lib/$type/$cluster-$id/keyring
mon data = /home/ceph/var/lib/$type/$cluster-$id
mon cluster log file = /home/ceph/log/$cluster.log
[mon.a]
host = cld-xx
mon addr = x.x.x.x:6789
user = ceph
[mon.b]
host = cld-xx
mon addr = x.x.x.x:6789
user = ceph
[mon.c]
host = cld-xx
mon addr = x.x.x.x:6789
user = ceph
[mon.d]
host = cld-xx
mon addr = x.x.x.x:6789
user = ceph
[mon.e]
host = cld-xx
mon addr = x.x.x.x:6789
user = ceph
[osd]
keyring = /home/ceph/var/lib/$type/$cluster-$id/keyring
osd data = /home/ceph/var/lib/$type/$cluster-$id
osd journal = /home/ceph/var/lib/$type/$cluster-$id/journal
osd journal size = 1000
osd mkfs type = xfs
osd mount options xfs = rw,noatime,inode64
[osd.0]
host = cld-xx
addr = x.x.x.x
user = ceph
devs = /dev/sdb1
[osd.1]
host = cld-xx
addr = x.x.x.x
user = ceph
devs = /dev/sda1
[osd.2]
host = cld-xx
addr = x.x.x.x
user = ceph
devs = /dev/sdb1
[osd.3]
host = cld-xx
addr = x.x.x.x
user = ceph
devs = /dev/sda1
[osd.4]
host = cld-xx
addr = x.x.x.x
user = ceph
devs = /dev/sdb1
[osd.5]
host = cld-xx
addr = x.x.x.x
user = ceph
devs = /dev/sda1
[client.libvirt]
rbd cache = true
Turning on rbd caching in ceph.conf, while telling QEMU it is not
writeback in libvirt's xml like:
<driver name='qemu' type='raw' cache='none'/>
is not safe since QEMU won't propagate flush requests from the guest
to librbd's cache. Since QEMU 1.3 setting the QEMU cache mode will
also set the librbd cache mode appropriately, so you don't need to
have the setting in your ceph.conf file at all.
Can you verify that your QEMU binary was built against librbd v0.60 or
later?
It would also help to see whether you can reproduce the problem when
QEMU is sending flushes (cache=writeback), and with the cache
disabled (cache=none, no 'rbd cache' setting in ceph.conf).
The next step would be gathering a log from librbd (debug rbd = 20,
debug objectcacher=20, debug objecter=20, debug ms = 1) when this
problem is occurring, and correlating it with a timestamped log of
the network latency.
Thanks,
Josh
auth support = cephx none
mon host =
x.x.x.x:6789;x.x.x.x:6789;x.x.x.x:6789;x.x.x.x:6789;x.x.x.x:6789
Regards
Alan ye
------------------
??????
Alan Ye
------------------ ???????? ------------------
*??????:* "Josh Durgin";<josh.dur...@inktank.com>;
*????????:* 2014??1??14??(??????) ????2:24
*??????:* "Stefan Hajnoczi"<stefa...@gmail.com>; "??????"
<yeshaoc...@foxmail.com>;
*????:* "qemu-devel"<qemu-devel@nongnu.org>;
*????:* Re: ?????? ?????? ?????? [Qemu-devel] qemu vm big network
latency when met heavy io
On 01/12/2014 06:39 PM, Stefan Hajnoczi wrote:
> On Fri, Jan 10, 2014 at 11:50 AM, ?????? <yeshaoc...@foxmail.com> wrote:
>
> Please use Reply-all to keep the CC list in tact. That way the
> conversation stays on the mailing list and others can participate.
>
>>> Is the sum of guests' RAM less than the total physical RAM on the host
>> The host run 3 vms, each vm use one vcpu core and 1G ram.
>> # free -m
>>
>> total used free shared buffers
cached
>>
>> Mem: 32242 4808 27434 0 278
2058
>>
>> -/+ buffers/cache: 2471 29771
>>
>> Swap: 4095 0 4095
>>
>>
>> The host has 8 cores.
>> # cat /proc/cpuinfo | grep processor
>> processor : 0
>> processor : 1
>> processor : 2
>> processor : 3
>> processor : 4
>> processor : 5
>> processor : 6
>> processor : 7
>>
>> so, both of two question's answer is 'yes'.
>> When I runing random write test, the host use 0 swap.
>
> Great. That means the host is not overcommitted.
>
> It's likely that the problem is a bug in QEMU's rbd driver or librados.
>
> Josh: Perhaps something you're interested in looking into?
Yes, thanks for bringing it to my attention. It does sound like a bug in
QEMU's rbd driver or ceph's userspace libraries.
Could you share what version of librbd you're using, and your
/etc/ceph/ceph.conf?
Thanks,
Josh