Hi,
I'm looking to build a full osd ssd cluster, with this config:
6 nodes,
each node 10 osd/ ssd drives (dual 10gbit network). (1journal + datas on each
osd)
ssd drive will be entreprise grade,
maybe intel sc3500 800GB (well known ssd)
or new Samsung SSD PM853T 960GB (don't have too much
dredi 23 Mai 2014 04:57:51
Objet: Re: [ceph-users] full osd ssd cluster advise : replication 2x or 3x ?
Hello,
On Thu, 22 May 2014 18:00:56 +0200 (CEST) Alexandre DERUMIER wrote:
> Hi,
>
> I'm looking to build a full osd ssd cluster, with this config:
>
What is your main goa
oment)
- Mail original -
De: "Christian Balzer"
À: ceph-users@lists.ceph.com
Envoyé: Vendredi 23 Mai 2014 07:29:52
Objet: Re: [ceph-users] full osd ssd cluster advise : replication 2x or 3x ?
On Fri, 23 May 2014 07:02:15 +0200 (CEST) Alexandre DERUMIER wrote:
> >&g
)
my main concern, is to known if it's really needed to have replication x3
(mainly for cost price).
But I can wait to have lower ssd price next year, and go to 3x if necessary.
- Mail original -
De: "Alexandre DERUMIER"
À: "Christian Balzer"
Cc: ceph-use
>>https://github.com/rochaporto/collectd-ceph
>>
>>It has a set of collectd plugins pushing metrics which mostly map what
>>the ceph commands return. In the setup we have it pushes them to
>>graphite and the displays rely on grafana (check for a screenshot in
>>the link above).
Thanks for sh
Hi,
if you use debian,
try to use a recent kernel from backport (>3.10)
also check your libleveldb1 version, it should be 1.9.0-1~bpo70+1 (debian
wheezy version is too old)
I don't see it in ceph repo:
http://ceph.com/debian-firefly/pool/main/l/leveldb/
(only for squeeze ~bpo60+1)
but you c
De: "jan zeller"
À: aderum...@odiso.com
Cc: ceph-users@lists.ceph.com
Envoyé: Vendredi 23 Mai 2014 13:36:04
Objet: AW: [ceph-users] pgs incomplete; pgs stuck inactive; pgs stuck unclean
> -Ursprüngliche Nachricht-
> Von: Alexandre DERUMIER [mailto:aderum...@od
>> Are people using automation tools like puppet or ansible?
http://www.sebastien-han.fr/blog/2014/05/01/vagrant-up-install-ceph-in-one-command/
enjoy ;)
- Mail original -
De: "Don Talton (dotalton)"
À: ceph-users@lists.ceph.com
Envoyé: Mardi 27 Mai 2014 18:19:00
Objet: [ceph-use
Congratulations Eric !
- Mail original -
De: "Loic Dachary"
À: "ceph-users"
Envoyé: Jeudi 29 Mai 2014 12:28:56
Objet: [ceph-users] Ceph User Committee : welcome Eric Mourgaya
Hi Ceph,
Welcome Eric Mourgaya, head of the Ceph User Committee starting May 2014 until
the next elect
>>crush map has legacy tunables
you need t update tunables in the crush, to benefit from last optimisati
#ceph osd crush tunables optimal
http://ceph.com/docs/master/rados/operations/crush-map/
- Mail original -
De: "Fabio - NS3 srl"
À: "ceph-users"
Envoyé: Jeudi 29 Mai 2014 09
Hi,
I think you can check this wiki:
http://ceph.com/docs/master/start/os-recommendations/
currently, only ubuntu 12.04 is deeply tested with inktank (but I think it'll
be rhel7 soon ;)
the wiki don't have been updated yet for firefly.
I known that ceph enterprise users are using dumpling for
able to saturated
>>10Gbit network.
Thanks for your reply !
- Mail original -
De: "Łukasz Jagiełło"
À: "Alexandre DERUMIER"
Cc: ceph-users@lists.ceph.com
Envoyé: Mercredi 28 Mai 2014 01:25:40
Objet: Re: [ceph-users] full osd ssd cluster advise : replicati
hi!
>>See design here: http://adminlinux.com.br/cluster_design.txt
>>
>># dpkg -l |grep ceph
>>ii ceph 0.41-1ubuntu2.1
>>distributed storage
>>ii ceph-common0.41-1ubuntu2.1
>>common utilities to mount and
se it for ceph ? if yes, what about stability|performance ?
with comming rdma support in ceph, it seem to be the perfect solution (and
price is very good).
Regards,
Alexandre Derumier
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists
from
Ethernet, especially as it relates to network topology. It's cheap and
fast, but not always easy to setup if you've got a lot of existing
infrastructure.
Mark
On 06/02/2014 11:29 AM, Alexandre DERUMIER wrote:
> Hi,
> I'm looking for a fast and cheap 10gbe ethernet
.
Carlos M. Perez
CMP Consulting Services
305-669-1515
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Alexandre DERUMIER
> Sent: Monday, June 2, 2014 12:30 PM
> To: ceph-users
> Subject: [ceph-users] mellanox
I just found this:
http://www.mellanox.com/related-docs/whitepapers/WP_Deploying_Ceph_over_High_Performance_Networks.pdf
Good to see than ceph begin to be tested by hardware vendor :)
Whitepaper include radosbench and fio results
- Mail original -
De: "Alexandre DERUMIER
Hi,
>>My low-budget setup consists of two gigabit switches, capable of LACP,
>>but not stackable. For redundancy, I'd like to have my links spread
>>evenly over both switches.
If you want to do lacp with both switches, they need to be stackable.
(or use active-backup bonding)
>>My question wh
to use both links for public (client-osd). No
multipathing like iscsi for example.
- Mail original -
De: "Sven Budde"
À: "Alexandre DERUMIER"
Cc: ceph-users@lists.ceph.com
Envoyé: Jeudi 5 Juin 2014 18:27:32
Objet: AW: [ceph-users] Ceph networks, to bond or not
Hi everybody,
we are going to do our first french proxmox in Paris in september
http://www.meetup.com/Proxmox-VE-French-Meetup/
And of course, we'll talk about ceph integration in proxmox.
So if you are interested, feel free to join us !
Regards,
Alexandre
___
able. And it's too noisy to discuss anything.
Thanks fo the info Loic.
For the moment, I have a room in my company building in Paris (for 10-15
peoples), But good to known.
I'm waiting for the next ceph meetup too :)
- Mail original -
De: "Loic Dachary"
À: &q
>>Do for every read 1 Kb rbd will read 4MB from hdd? for write?
rados support partial read|write.
Note that with erasure code, write need to full rewrite object. (so 4MB)
I think that with key-value-store backend (like leveldb), read/write are full
too.
some interesting notes here :
http://eu
Hi,
I'm reading tiering doc here
http://ceph.com/docs/firefly/dev/cache-pool/
"
The hit_set_count and hit_set_period define how much time each HitSet should
cover, and how many such HitSets to store. Binning accesses over time allows
Ceph to independently determine whether an object was accesse
, because I have cold datas, but I have full backups jobs
running each week, reading all theses cold datas)
- Mail original -
De: "Gregory Farnum"
À: "Alexandre DERUMIER"
Cc: "ceph-users"
Envoyé: Mercredi 11 Juin 2014 21:56:29
Objet: Re: [ceph-users
Hi Eric,
timeframe is too short for me, I'm busy this day.
I'm interest too by slides if you have them.
See you soon,
Alexandre
- Mail original -
De: "eric mourgaya"
À: ceph-users@lists.ceph.com, "Loic Dachary"
Envoyé: Lundi 16 Juin 2014 08:04:14
Objet: [ceph-users] ceph meetup
Hi Greg,
>>So the only way to improve performance would be to not use O_DIRECT (as this
>>should bypass rbd cache as well, right?).
yes, indeed O_DIRECT bypass cache.
BTW, Do you need to use mysql with O_DIRECT ? default innodb_flush_method is
fdatasync, so it should work with cache.
(but yo
I don't known if it's related, but
"[Performance] Improvement on DB Performance"
http://www.spinics.net/lists/ceph-devel/msg19062.html
they are a patch here:
https://github.com/ceph/ceph/pull/1848
already pushed in master
- Mail original -
De: "Robert van Leeuwen"
À: ceph-users@l
Hi,
I would like to known if a centos7 respository will be available soon ?
Or can I use current rhel7 for the moment ?
http://ceph.com/rpm-firefly/rhel7/x86_64/
Cheers,
Alexandre
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.c
Hi,
sorry to spam the mailing list,
but they are a inktank mellanox webinar in 10minutes,
and I don't have receive access since I have been registered yesterday (same
for my co-worker).
and the webinar mellanox contact email (conta...@mellanox.com), does not
exist
Maybe somebody from In
Ok, sorry, we have finally receive the login a bit late.
Sorry again to have spam the mailing list
- Mail original -
De: "Alexandre DERUMIER"
À: "ceph-users"
Envoyé: Jeudi 10 Juillet 2014 16:55:22
Objet: [ceph-users] inktank-mellanox webinar access ?
Hi,
Hi,
>>But in reality (yum update or by using ceph-deploy install nodename) -
>>the package manager does restart ALL ceph services on that node by its own...
debian packages don't restart ceph services on package update, maybe it's a bug
in rpm packaging ?
- Mail original -
De: "
Same question here,
I'm contributor on proxmox, and we don't known if we can upgrade librbd safely,
for users with dumpling cluster.
Also, for ceph enterprise , s oes inktank support dumpling enterprise +
firefly librbd ?
- Mail original -
De: "Nigel Williams"
À: ceph-users@lis
Hi,
for RHEL5, I'm not sure
be barriers supported is maybe not implemented in virtio devices,lvm,dm raid
and some filesystem,
depend of the kernel version.
Not sure what is backported in rhel5 kernel
see
http://monolight.cc/2011/06/barriers-caches-filesystems/
- Mail original -
rivers-windows/issues/3
- Mail original -
De: "Yufang"
À: "Alexandre DERUMIER"
Cc: ceph-users@lists.ceph.com
Envoyé: Vendredi 22 Août 2014 18:05:32
Objet: Re: [ceph-users] Is it safe to enable rbd cache with qemu?
Thanks, Alexandre. But what about Windows? Does NT
>>Allegedly this model ssd (128G m550) can do 75K 4k random write IOPS
>>(running fio on the filesystem I've seen 70K IOPS so is reasonably
>>believable). So anyway we are not getting anywhere near the max IOPS
>>from our devices.
Hi,
Just check this:
http://www.anandtech.com/show/7864/crucia
Hi Sebastien,
>>I got 6340 IOPS on a single OSD SSD. (journal and data on the same
>>partition).
Shouldn't it better to have 2 partitions, 1 for journal and 1 for datas ?
(I'm thinking about filesystem write syncs)
- Mail original -
De: "Sebastien Han"
À: "Somnath Roy"
Cc: ce
Do you have same results, if you launch 2 fio benchs in parallel on 2
differents rbd volumes ?
- Mail original -
De: "Sebastien Han"
À: "Cédric Lemarchand"
Cc: "Alexandre DERUMIER" , ceph-users@lists.ceph.com
Envoyé: Mardi 2 Septembre 2014 13
De: "Sebastien Han"
À: "Alexandre DERUMIER"
Cc: ceph-users@lists.ceph.com, "Cédric Lemarchand"
Envoyé: Mardi 2 Septembre 2014 15:25:05
Objet: Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K
IOPS
Well the last time I ran two processes
I was waiting for the schedule, topics seem to be interesting.
I'm going to register now :)
BTW, are the speeches in french or english? (As I see loic,sebastian and yann
as speakers)
- Mail original -
De: "Patrick McGarry"
À: "Ceph Devel" , "Ceph-User"
, "Ceph User Committee"
Envoyé
>>Is there a way to resize the OSD without bringing the cluster down?
What is the HEALTH state of your cluster ?
If it's OK, simply replace the osd disk by a bigger one ?
- Mail original -
De: "JIten Shah"
À: ceph-us...@ceph.com
Envoyé: Samedi 6 Septembre 2014 00:31:01
Objet: [c
Hi Sebastien,
here my first results with crucial m550 (I'll send result with intel s3500
later):
- 3 nodes
- dell r620 without expander backplane
- sas controller : lsi LSI 9207 (no hardware raid or cache)
- 2 x E5-2603v2 1.8GHz (4cores)
- 32GB ram
- network : 2xgigabit link lacp + 2xgigabit lac
> Le 11/09/2014 08:20, Alexandre DERUMIER a écrit :
>> Hi Sebastien,
>>
>> here my first results with crucial m550 (I'll send result with intel s3500
>> later):
>>
>> - 3 nodes
>> - dell r620 without expander backplane
>> - sas cont
ename=/dev/sdb --direct=1 --rw=write --bs=4k --numjobs=2
--group_reporting --invalidate=0 --name=ab --sync=1
bw=177575KB/s, iops=44393
- Mail original -
De: "Alexandre DERUMIER"
À: "Cedric Lemarchand"
Cc: ceph-users@lists.ceph.com
Envoyé: Vendredi 12 Sept
/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await
w_await svctm %util
sdb 0,00 29,00 0,00 3075,00 0,00 36748,50 23,90 0,29 0,10 0,00 0,10 0,05 15,20
So, the write bottleneck seem to be in ceph.
I will send s3500 result today
- Mail original -
De: "Alexandre DER
it w_await svctm %util
sdb 0,00 1563,000,00 9880,00 0,00 75223,5015,23
2,090,210,000,21 0,07 80,00
- Mail original -----
De: "Alexandre DERUMIER"
À: "Cedric Lemarchand"
Cc: ceph-users@lists.ceph.com
Envoyé: Vendredi 12 Sept
Hi,
as ceph user, It could be wonderfull to have it for Giant,
optracker performance impact is really huge (See my ssd benchmark on ceph user
mailing)
Regards,
Alexandre Derumier
- Mail original -
De: "Somnath Roy"
À: "Samuel Just"
Cc: "Sage Weil"
Hi,
I would like to known with libleveldb should be us with firefly.
I'm using debian wheezy which provide really old libleveldb (I don't use it),
and in wheezy backport 1.17 is provided.
But in intank repositories , I see that 1.9 is provide for some distribs.
So, what is the best/tested ve
hursday, could be great if we could
share experience about ceph and ssd.
Alexandre.
- Mail original -
De: "Sebastien Han"
À: "Alexandre DERUMIER"
Cc: ceph-users@lists.ceph.com
Envoyé: Mardi 16 Septembre 2014 15:32:59
Objet: Re: [ceph-users] [Single OSD pe
my ssds model for my production
cluster (target 2015),I'll have a look for this optimus drives
- Mail original -
De: "Somnath Roy"
À: "Mark Kirkwood" , "Alexandre DERUMIER"
, "Sebastien Han"
Cc: ceph-users@lists.ceph.com
Envoyé: Mer
ly or Giant ?
I'll do benchs with 6 osd dc3500 tomorrow to compare firefly and giant.
- Mail original -
De: "Jian Zhang"
À: "Sebastien Han" , "Alexandre DERUMIER"
Cc: ceph-users@lists.ceph.com
Envoyé: Jeudi 18 Septembre 2014 08:12:32
Objet: RE:
_op_threads = 5
filestore_op_threads = 4
bw=62094KB/s, iops=15523
giant with same tuning
---
bw=247073KB/s, iops=61768 !
I think I could reach more, but my 2 gigabit link are satured.
- Mail original -----
De: "Alexandre DERUMIER"
À: "Ji
B/s, iops=29532
- Mail original -
De: "Jian Zhang"
À: "Alexandre DERUMIER"
Cc: ceph-users@lists.ceph.com
Envoyé: Vendredi 19 Septembre 2014 10:21:38
Objet: RE: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K
IOPS
Thanks for this gre
ing :)
Thanks Loic
- Mail original -
De: "Loic Dachary"
À: "aderumier" , "ceph-users"
Envoyé: Dimanche 1 Février 2015 18:42:51
Objet: Re: [ceph-users] erasure code : number of chunks for a small cluster ?
Hi Alexandre,
On 01/02/2015 18:15, Alexandre DERUMIER wr
y sense.
I start with 5 hosts and use k:3, m:2. In this case two hdds can fail or
one host can be down for maintenance.
Udo
PS: you also can't change k+m on a pool later...
On 01.02.2015 18:15, Alexandre DERUMIER wrote:
> Hi,
>
> I'm currently trying to understand how to se
Hi,
From my tests with giant, this was the cpu which limit the performance on osd.
I'm going to do some benchmark with 2x10 cores 3,1ghz for 6ssd next month.
I'll post results on the mailing list.
- Mail original -
De: "mad Engineer"
À: "Gregory Farnum"
Cc: "ceph-users"
Envoyé: Ma
debian deb packages update are not restarting services.
(So, I think it should be the same for ubuntu).
you need to restart daemons in this order:
-monitor
-osd
-mds
-rados gateway
http://ceph.com/docs/master/install/upgrading-ceph/
- Mail original -
De: "Stephen Jahl"
À: "Gregory Fa
Hi,
>>What I saw after enabling RBD cache it is working as expected, means
>>sequential write has better MBps than random write. can somebody explain this
>>behaviour ?
This is because rbd_cache merge coalesced ios in bigger ios, so it's working
only with sequential workload.
you'll do less i
Oh, I didn't thinked about this.
Thanks Hector !
- Mail original -
De: "Hector Martin"
À: "ceph-users"
Envoyé: Vendredi 6 Février 2015 09:06:29
Objet: Re: [ceph-users] erasure code : number of chunks for a small cluster ?
On 02/02/15 03:38, Udo Lembke wrote:
> With 3 hosts only you c
Hi,
I'm currently use crush tunables "optimal" value.
If I upgrade from firefly to hammer, does the optimal value will upgrade to
optimal values for hammer.
So, does my clients (qemu-librbd) need to be also upgraded to hammer to support
new hammer features ?
If yes,
I think to:
- change c
m firefly to
hammer behaviour ?
On Mon, 9 Feb 2015, Alexandre DERUMIER wrote:
> Hi,
>
> I'm currently use crush tunables "optimal" value.
>
> If I upgrade from firefly to hammer, does the optimal value will upgrade
> to optimal values for hammer.
The tunable
hours of benchmarking. So be careful.
crush optimal sets it to the current optimal value.
So you need to upgrade your clients first and then set it to optimal again.
Stefan
Excuse my typo s ent from my mobile phone.
Am 09.02.2015 um 06:32 schrieb Alexandre DERUMIER < aderum...@odiso.c
Hi,
you need to import foreign config from openmanage webui.
somewhere in storage controller
BTW, I'm currently testing new dell r630 with a perc h330 ( lsi 3008)
With this controller, it's possible to do hardware for some disks, and
passthrough for some others disks.
So, perfect for ceph :)
Hi,
I'm currently running a big mongodb cluster, around 2TB, (sharding +
replication).
And I have a lot of problems with mongo replication (out of syncs and need to
full replicate again and again datas between my mongo replicats).
So, I thinked to use rbd to replicate the storage and keep onl
the behavior of mongo when a shard is unavailable for some reason
(crash or network partition) ? If shard3 is on the wrong side of a network
partition and uses RBD, it will hang. Is it something that mongo will
gracefully handle ?
I have no experience in this but I'm curious about this
>>Hi,
>>Can you test with disabling rbd_cache ?
>>I remember of a bug detected in giant, not sure it's also the case for fireflt
This was this tracker:
http://tracker.ceph.com/issues/9513
But It has been solved and backported to firefly.
Also, can you test 0.80.6 and 0.80.7 ?
- Mai
Hi,
Do you have also tested 0.80.6 and 0.80.7 librbd ?
could be usefull to search commits in git.
(I'm not sure that all changes are in the release note)
- Mail original -
De: "杨万元"
À: "ceph-users"
Envoyé: Jeudi 12 Février 2015 04:14:15
Objet: [ceph-users] Upgrade 0.80.5 to 0.80.8 --t
>>To my surprise however these slow requests caused aborts from the block
>>device on the VM side, which ended up corrupting files
This is very strange, you shouldn't have corruption.
Do you use writeback ? if yes, do you have disable barrier on your filesystem ?
(What is the qemu version ? gue
>>Can this timeout be increased in some way? I've searched around and found the
>>/sys/block/sdx/device/timeout knob, which in my case is set to 30s.
yes, sure
echo 60 > /sys/block/sdx/device/timeout
for 60s for example
- Mail original -
De: "Krzysztof Nowicki"
À: "Andrey Korolyov"
Just buy it.
Nice book, I don't have read yet all the book, but it seem to cover all ceph
features.
Good job !
- Mail original -
De: "Karan Singh"
À: "Ceph Community" , "ceph-users"
, ceph-maintain...@ceph.com, "ceph-users"
, "ceph-devel"
Cc: "Sage Weil" , "don"
Envoyé: Jeudi 5 Fé
Nice Work Mark !
I don't see any tuning about sharding in the config file sample
(osd_op_num_threads_per_shard,osd_op_num_shards,...)
as you only use 1 ssd for the bench, I think it should improve results for
hammer ?
- Mail original -
De: "Mark Nelson"
À: "ceph-devel"
Cc: "ceph-us
ll. Have you done any testing here? It could
be an interesting follow-up paper.
Mark
On 02/18/2015 02:34 AM, Alexandre DERUMIER wrote:
> Nice Work Mark !
>
> I don't see any tuning about sharding in the config file sample
>
> (osd_op_num_threads_per_shard,osd_op_num_s
Hi,
Intel has just released new ssd s3610:
http://www.anandtech.com/show/8954/intel-launches-ssd-dc-s3610-s3710-enterprise-ssds
endurance is 10x bigger than 3500, for 10% cost addition.
Has somebody already tested them ?
Regards,
Alexandre
___
ceph-
his problem is cause from 0.80.8
2015-02-12 19:33 GMT+08:00 Alexandre DERUMIER < aderum...@odiso.com > :
>>Hi,
>>Can you test with disabling rbd_cache ?
>>I remember of a bug detected in giant, not sure it's also the case for
>>fireflt
This was this tra
Hi,
I known that Loic Dachary was currently working on backporting new feature on
giant,
I see that 0.87.1 has been tagged in git too:
here the difference:
https://github.com/ceph/ceph/compare/v0.87...v0.87.1
Loic, any annoucement/release note, yet ?
- Mail original -
De: "Lindsay M
I just upgraded my debian giant cluster,
1)on each node:
-
apt-get update
apt-get dist-upgrade
2)on each node:
-
/etc/init.d/ceph restart mon
#ceph -w ---> verify that HEALTH is ok before doing another node
3)on each node:
-
/etc/init.d/ceph restart osd
#ce
l -
De: "Mark Nelson"
À: "ceph-devel" , "ceph-users"
Envoyé: Jeudi 26 Février 2015 05:44:15
Objet: [ceph-users] Ceph Hammer OSD Shard Tuning Test Results
Hi Everyone,
In the Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison
thread, Alexandre D
Hi,
from qemu rbd.c
if (flags & BDRV_O_NOCACHE) {
rados_conf_set(s->cluster, "rbd_cache", "false");
} else {
rados_conf_set(s->cluster, "rbd_cache", "true");
}
and
block.c
int bdrv_parse_cache_flags(const char *mode, int *flags)
{
*flags &= ~BDRV_O_CACHE_MASK;
Hi,
First, test if your ssd can write fast with O_DSYNC
check this blog:
http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
Then, try with ceph Giant (or maybe wait for Hammer), because they are a lot of
optimisations for ssd for threads sh
and 9 OSD
with 3.16-3 kernel
> Am 28.02.2015 um 12:43 schrieb Alexandre DERUMIER :
>
> Hi,
>
> First, test if your ssd can write fast with O_DSYNC
> check this blog:
> http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-dev
As optimisation,
try to set ioscheduler to noop,
and also enable rbd_cache=true. (It's really helping for for sequential writes)
but your results seem quite low, 926kb/s with 4k, it's only 200io/s.
check if you don't have any big network latencies, or mtu fragementation
problem.
Maybe also t
compare
- Mail original -
De: "Mark Nelson"
À: "ceph-devel" , "ceph-users"
Envoyé: Jeudi 26 Février 2015 05:44:15
Objet: [ceph-users] Ceph Hammer OSD Shard Tuning Test Results
Hi Everyone,
In the Ceph Dumpling/Firefly/Hammer SSD/Memstore performa
Ceph Hammer OSD Shard Tuning Test Results
Can I ask what xio and simple messenger are and the differences?
Kind regards
Kevin Walker
+968 9765 1742
On 1 Mar 2015, at 18:38, Alexandre DERUMIER wrote:
Hi Mark,
I found an previous bench from Vu Pham (it's was about simplemessenger vs
Hi,
maybe this can help you:
http://www.sebastien-han.fr/blog/2013/11/22/map-slash-unmap-rbd-device-on-boot-slash-shutdown/
Regards,
Alexandre
- Mail original -
De: "Jesus Chavez (jeschave)"
À: "ceph-users"
Envoyé: Lundi 2 Mars 2015 11:14:49
Objet: [ceph-users] Permanente Mount RBD
tics match the
benchmark result numbers?
I also did some tests on fdcache, though just glancing at the results it
doesn't look like tweaking those parameters had much effect.
Mark
On 03/01/2015 08:38 AM, Alexandre DERUMIER wrote:
> Hi Mark,
>
> I found an previous bench from
I think this will be fixed in next firefly point release
tracker for firefly 0.80.8 speed decrease
http://tracker.ceph.com/issues/10956
Jason Dillaman link it to the famous object_cacher bug:
http://tracker.ceph.com/issues/9854
- Mail original -
De: "Gregory Farnum"
À: "Nathan O'Sulliv
Hi,
maybe this is related ?:
http://tracker.ceph.com/issues/9503
"Dumpling: removing many snapshots in a short time makes OSDs go berserk"
http://tracker.ceph.com/issues/9487
"dumpling: snaptrimmer causes slow requests while backfilling.
osd_snap_trim_sleep not helping"
http://lists.opennebula
HDD since firefly.
And yes, the problem hang the production at night during snap trimming.
I suppose there is a new OSD parameter which change behavior of the
journal, or something like that. But didn't find anything about that.
Olivier
Le mercredi 04 mars 2015 à 14:44 +0100, Alexandre D
Le mercredi 04 mars 2015 à 15:05 +0100, Alexandre DERUMIER a écrit :
> >>The load problem is permanent : I have twice IO/s on HDD since firefly.
>
> Oh, permanent, that's strange. (If you don't see more traffic coming from
> clients, I don't understand...)
&
Objet: Re: [ceph-users] Perf problem after upgrade from dumpling to firefly
Only writes ;)
Le mercredi 04 mars 2015 à 16:19 +0100, Alexandre DERUMIER a écrit :
> >>The change is only on OSD (and not on OSD journal).
>
> do you see twice iops for read and write ?
>
>
Hi for hardware, inktank have good guides here:
http://www.inktank.com/resource/inktank-hardware-selection-guide/
http://www.inktank.com/resource/inktank-hardware-configuration-guide/
ceph works well with multiple osd daemon (1 osd by disk),
so you should not use raid.
(xfs is the recommended fs
Hi, do you have tried with differents io schedulers to compare ?
- Mail original -
De: "Nick Fisk"
À: "ceph-users"
Envoyé: Jeudi 5 Mars 2015 18:17:27
Objet: [ceph-users] Strange krbd behaviour with queue depths
I’m seeing a strange queue depth behaviour with a kernel mapped RBD, libr
>>Is it possible all replicas of an object to be saved in the same node?
No. (until you don't wrongly modify the crushmap manually)
>>Is it possible to lose any?
with replicat x2, if you loose 2osd on 2differents nodes, with the same object
inside, you'll lost the object
>>Is there a mechanis
want to try that. But, I think it is available in the latest kernel
>> only.
>> Librbd is running with TCP_NODELAY = true by default, you may want to try
>> with ms_tcp_nodelay = false to simulate the similar behavior with librbd.
>>
>> Thanks & Regards
>> Somnath
>>And at this moment, some of the VM stored on this pool were stopped (on
>>some hosts, not all, it depends, no logic)
do you use librbd or krbd for theses vm ?
Is the guest os crashed ? or the qemu process killed?(which seem really
strange)
- Mail original -
De: "Florent Bautista"
>>That full system slows down, OK, but brutal stop...
This is strange, that could be:
- qemu crash, maybe a bug in rbd block storage (if you use librbd)
- oom-killer on you host (any logs ?)
what is your qemu version ?
- Mail original -
De: "Florent Bautista"
À: "ceph-users"
Envoyé:
ent Bautista"
À: "aderumier"
Cc: "ceph-users"
Envoyé: Lundi 16 Mars 2015 11:14:45
Objet: Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !
On 03/16/2015 11:03 AM, Alexandre DERUMIER wrote:
> This is strange, that could be:
>
> - qemu crash, maybe
derumier"
Cc: "ceph-users"
Envoyé: Lundi 16 Mars 2015 12:35:11
Objet: Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !
On 03/16/2015 12:23 PM, Alexandre DERUMIER wrote:
>>> We use Proxmox, so I think it uses librbd ?
> As It's me that I made t
Hi Josef,
I'm going to benchmark a 3nodes cluster with 6ssd each node (2x10 cores 3,1ghz).
From my previous bench, you need fast cpus if you need a lot of iops, and
writes are lot more expansive than reads.
Now i'm you are doing only small iops (big blocks / big throughput), you don't
need too
Hi,
I don't known how rbd read-ahead is working,
but with qemu virtio-scsi, you can have read merge request (for sequential
reads), so it's doing bigger ops to ceph cluster and improve throughput.
virtio-blk merge request will be supported in coming qemu 2.3.
(I'm not sure of virtio-win driver
’ve been done with it before I ordered :)
I will most likely order this week so I will see it when the stuff is being
assembled :o
Do you feel that there something in the setup that could be better if you would
decide on hardware as of today?
Also, will you try out replication set of 2 as well?
1 - 100 of 436 matches
Mail list logo
