maybe this is debug perfcounter ?
----- Mail original ----- De: "Stefan Priebe - Profihost AG" <[email protected]> À: "Alexandre DERUMIER" <[email protected]> Cc: [email protected], [email protected] Envoyé: Lundi 12 Novembre 2012 16:33:48 Objet: Re: [pve-devel] less cores more iops / speed Adding this to ceph.conf on kvm host adds another 2000 iops (20.000 iop/s with one VM). I'm sure most of them are useless on a client kvm / rbd host but i don't know which one makes sense ;-) [global] debug ms = 0/0 debug rbd = 0/0 debug lockdep = 0/0 debug context = 0/0 debug crush = 0/0 debug buffer = 0/0 debug timer = 0/0 debug journaler = 0/0 debug osd = 0/0 debug optracker = 0/0 debug objclass = 0/0 debug filestore = 0/0 debug journal = 0/0 debug ms = 0/0 debug monc = 0/0 debug tp = 0/0 debug auth = 0/0 debug finisher = 0/0 debug heartbeatmap = 0/0 debug perfcounter = 0/0 debug asok = 0/0 debug throttle = 0/0 [client] debug ms = 0/0 debug rbd = 0/0 debug lockdep = 0/0 debug context = 0/0 debug crush = 0/0 debug buffer = 0/0 debug timer = 0/0 debug journaler = 0/0 debug osd = 0/0 debug optracker = 0/0 debug objclass = 0/0 debug filestore = 0/0 debug journal = 0/0 debug ms = 0/0 debug monc = 0/0 debug tp = 0/0 debug auth = 0/0 debug finisher = 0/0 debug heartbeatmap = 0/0 debug perfcounter = 0/0 debug asok = 0/0 debug throttle = 0/0 Stefan Am 12.11.2012 15:35, schrieb Alexandre DERUMIER: > Another idea, > > do you have tried to put >>>>> debug lockdep = 0/0 >>>>> debug context = 0/0 >>>>> debug crush = 0/0 >>>>> debug buffer = 0/0 >>>>> debug timer = 0/0 >>>>> debug journaler = 0/0 >>>>> debug osd = 0/0 >>>>> debug optracker = 0/0 >>>>> debug objclass = 0/0 >>>>> debug filestore = 0/0 >>>>> debug journal = 0/0 >>>>> debug ms = 0/0 >>>>> debug monc = 0/0 >>>>> debug tp = 0/0 >>>>> debug auth = 0/0 >>>>> debug finisher = 0/0 >>>>> debug heartbeatmap = 0/0 >>>>> debug perfcounter = 0/0 >>>>> debug asok = 0/0 >>>>> debug throttle = 0/0 > > in a ceph.conf on your kvm host ? > > > ----- Mail original ----- > > De: "Alexandre DERUMIER" <[email protected]> > À: "Stefan Priebe - Profihost AG" <[email protected]> > Cc: [email protected] > Envoyé: Lundi 12 Novembre 2012 15:26:36 > Objet: Re: [pve-devel] less cores more iops / speed > > Maybe some tracing on kvm process could give us clues to find where the cpu > is used ? > > Also another idea, can you try with "auth supported=none" ? maybe they are > some overhead with ceph authenfication ? > > > > > ----- Mail original ----- > > De: "Alexandre DERUMIER" <[email protected]> > À: "Stefan Priebe - Profihost AG" <[email protected]> > Cc: [email protected] > Envoyé: Lundi 12 Novembre 2012 15:20:07 > Objet: Re: [pve-devel] less cores more iops / speed > > Ok thanks. > > Seem to use a lot of cpu vs nfs,iscsi ... > > I hope that ceph dev will work on this soon ! > > > ----- Mail original ----- > > De: "Stefan Priebe - Profihost AG" <[email protected]> > À: "Alexandre DERUMIER" <[email protected]> > Cc: "eric" <[email protected]>, [email protected] > Envoyé: Lundi 12 Novembre 2012 15:05:08 > Objet: Re: [pve-devel] less cores more iops / speed > > Am 12.11.2012 13:49, schrieb Alexandre DERUMIER: >>>> One VM on one Host: 18.000 IOP/s >>>> Two VM on one Host: 2x11.000 IOP/s >>>> Three VM on one Host: 3x7.000 IOP/s >> >> And host cpu is 100% ? > > No. For three VMs yes. For one and two no. I think librbd / rbd > implementation in kvm is the bottleneck here. > > Stefan > >> ----- Mail original ----- >> >> De: "Stefan Priebe - Profihost AG" <[email protected]> >> À: "Alexandre DERUMIER" <[email protected]> >> Cc: "eric" <[email protected]>, [email protected] >> Envoyé: Lundi 12 Novembre 2012 12:58:35 >> Objet: Re: [pve-devel] less cores more iops / speed >> >> Am 12.11.2012 08:51, schrieb Alexandre DERUMIER: >>>>> Right now RBD in KVM is limited by CPU speed. >>> >>> Good to known, so it's seem lack of threading, or maybe somes locks. (so >>> faster cpu give more iops). >>> >>> If you lauch parallel fio on same host on different guest, do you get more >>> total iops ? (for me it's scale) >> >> One VM on one Host: 18.000 IOP/s >> Two VM on one Host: 2x11.000 IOP/s >> Three VM on one Host: 3x7.000 IOP/s >> >>> if you launch 2 parallel fio, on same guest (on differents disk), do you >>> get more iops ? (for me, it doesn't scale, so raid0 in guest doesn't help). >> No it doesn't scale. >> >> Stefan >> >>> ----- Mail original ----- >>> >>> De: "Stefan Priebe" <[email protected]> >>> À: "Alexandre DERUMIER" <[email protected]> >>> Cc: "eric" <[email protected]>, [email protected] >>> Envoyé: Dimanche 11 Novembre 2012 13:07:36 >>> Objet: Re: [pve-devel] less cores more iops / speed >>> >>> Am 11.11.2012 12:12, schrieb Alexandre DERUMIER: >>>> If I remember good, stefan can achieve 100.000 iops with iscsi with same >>>> kvm host. >>> >>> Correct but this was always with scsi-generic and I/O multipathing on >>> host. rbd does not support scsi-generic ;-( >>> >>>> I have checked ceph mailing, stefan seem to have resolved his problem with >>>> dual core with bios update ! >>> Correct. So speed on Dual Xeon is now 14.000 IOP/s and 18.000 IOP/s on >>> Single Xeon. But the difference is an issue of the CPU Speed. 3,6Ghz >>> Single Xeon vs. 2.5Ghz Dual Xeon. >>> >>> Right now RBD in KVM is limited by CPU speed. >>> >>> Greets, >>> Stefan >>> > _______________________________________________ > pve-devel mailing list > [email protected] > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > _______________________________________________ > pve-devel mailing list > [email protected] > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel >
_______________________________________________ pve-devel mailing list [email protected] http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
