Re: [ceph-users] Having problem to start Radosgw

Hi, Le 15/02/2015 08:37, B L a écrit : > this is the resulting output after adding “-n client.radosgw.gateway” : > https://gist.github.com/anonymous/f16701d6cacc8911620f > > > I can see one problem only in the above output: -1 Couldn't i

[ceph-users] Dedicated disks for monitor and mds?

Hi, I'm trying to plan the hardware for a little ceph cluster. We don't have a lot of financial means. In addition, we will have to pay attention to the electric consumption. At first, it will probably be a cluster with 3 physical servers and on each server will be osd node and monitor node (and m

Re: [ceph-users] Dedicated disks for monitor and mds?

Hello, Le 17/02/2015 05:55, Christian Balzer wrote : >> 1. I have read "10 GB per daemon for the monitor". But is >> I/O disk performance important for a monitor? Is it unreasonable >> to put the working directory of the monitor in the same partition >> of the root filesystem (ie /)? >> > Yes, mo

Re: [ceph-users] Dedicated disks for monitor and mds?

Hi, Le 17/02/2015 11:15, John Spray a écrit : > The MDS does not use local storage at all -- CephFS metadata is stored in > RADOS (i.e. the MDS stores data via the OSDs). Ah ok. So, consequently, I can put the working directory of the mds (ie /var/lib/ceph/mds/ceph-$id/) absolutely everywhere,

[ceph-users] Firefly, cephfs issues: different unix rights depending on the client and ls are slow

Hi, I'm trying cepfs and I have some problems. Here is the context: All the nodes (in cluster and the clients) are Ubuntu 14.04 with a 3.16 kernel (after apt-get install linux-generic-lts-utopic && reboot). The cluster: - one server with just one monitor daemon (RAM 2GB) - 2 servers (RAM 24GB) w

Re: [ceph-users] Firefly, cephfs issues: different unix rights depending on the client and ls are slow

t; environment and I have the same phenomena. Second, I have reinstalled my "virtualbox" environment but with the Hammer version of Ceph (ie the testing version 0.93-1trusty) and I have the same issues too. Le 04/03/2015 14:15, Francois Lafont wrote : [...] > ~#

Re: [ceph-users] Firefly, cephfs issues: different unix rights depending on the client and ls are slow

Hello, Thanks to Jcsp (John Spray I guess) that helps me on IRC. On 06/03/2015 04:04, Francois Lafont wrote: >> ~# mkdir /cephfs >> ~# mount -t ceph 10.0.2.150,10.0.2.151,10.0.2.152:/ /cephfs/ -o >> name=cephfs,secretfile=/etc/ceph/ceph.client.cephfs.secret >> >

Re: [ceph-users] ceph mds zombie

Hi, On 09/03/2015 04:06, kenmasida wrote : > I have resolved the problem，thank you very much。 When I use ceph-fuse to > mount the client，it work well. Good news but can you give the kernel version of your client cephfs OS? Like you, I had one problem with cephfs in the client side and it come

Re: [ceph-users] rados import error: short write

Hi, Le 10/03/2015 04:40, Leslie Teo a écrit : > we use `rados export poolA /opt/zs.rgw-buckets` export ceph cluster pool > named poolA into localdir /opt/ .and import the directroy > /opt/zs.rgw-buckets into another ceph cluster pool named hello , and > following the error :shell > rados

[ceph-users] PGs stuck unclean "active+remapped" after an osd marked out

Hi, I had a ceph cluster in "HEALTH_OK" state with Firefly 0.80.9. I just wanted to remove an OSD (which worked well). So after: ceph osd out 3 I waited for the rebalancing but I had "PGs stuck unclean": --- ~# ceph -s cluster

Re: [ceph-users] PGs stuck unclean "active+remapped" after an osd marked out

Le 11/03/2015 05:44, Francois Lafont a écrit : > PS: here is my conf. > [...] I have this too: ~# ceph osd crush show-tunables { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_d

Re: [ceph-users] PGs stuck unclean "active+remapped" after an osd marked out

Hi, I was always in the same situation: I couldn't remove an OSD without have some PGs definitely stuck to the "active+remapped" state. But I remembered I read on IRC that, before to mark out an OSD, it could be sometimes a good idea to reweight it to 0. So, instead of doing [1]: ceph osd ou

Re: [ceph-users] Firefly, cephfs issues: different unix rights depending on the client and ls are slow

Hi, Yan, Zheng wrote : >> http://tracker.ceph.com/issues/11059 >> > > It's a bug in ACL code, I have updated http://tracker.ceph.com/issues/11059 Ok, thanks. I have seen and I will answer quickly. ;) >> I'm still surprised by such times. For instance, It seems to me >> that, with a mounted nfs

Re: [ceph-users] PGs stuck unclean "active+remapped" after an osd marked out

Hi, Gregory Farnum a wrote : > If I remember/guess correctly, if you mark an OSD out it won't > necessarily change the weight of the bucket above it (ie, the host), > whereas if you change the weight of the OSD then the host bucket's > weight changes. I can just say that, indeed, I have noticed

Re: [ceph-users] SSD Hardware recommendation

Hi, Christian Balzer wrote : > Consider what you think your IO load (writes) generated by your client(s) > will be, multiply that by your replication factor, divide by the number of > OSDs, that will give you the base load per OSD. > Then multiply by 2 (journal on OSD) per OSD. > Finally based o

Re: [ceph-users] CephFS questions

Hi, Le 22/03/2015 09:29, Bogdan SOLGA a écrit : > I have a few questions related to the CephFS part of Ceph: > >- is it production ready? Currently, this is not considered "production ready". However, some people use Cephfs in production and I think they are satisfied. Personally, we are g

Re: [ceph-users] SSD Hardware recommendation

Hi, Sorry Christian for my late answer. I was a little busy. Christian Balzer a wrote: > You're asking the wrong person, as I'm neither a Ceph or kernel > developer. ^o^ No, no, the rest of the message proves to me that I talk to the right person. ;) > Back then Mark Nelson from the Ceph team

Re: [ceph-users] PG calculator queries

Hi, Sreenath BH wrote : > consider following values for a pool: > > Size = 3 > OSDs = 400 > %Data = 100 > Target PGs per OSD = 200 (This is default) > > The PG calculator generates number of PGs for this pool as : 32768. > > Questions: > > 1. The Ceph documentation recommends around 100 PGs/O

Re: [ceph-users] Is it possible to change the MDS node after its been created

Hi, Gregory Farnum wrote: > The MDS doesn't have any data tied to the machine you're running it > on. You can either create an entirely new one on a different machine, > or simply copy the config file and cephx keyring to the appropriate > directories. :) Sorry to enter in this post but how can

Re: [ceph-users] Is it possible to change the MDS node after its been created

Gregory Farnum wrote: >> Sorry to enter in this post but how can we *remove* a mds daemon of a >> ceph cluster? >> >> Are the commands below enough? >> >> stop the daemon >> rm -r /var/lib/ceph/mds/ceph-$id/ >> ceph auth del mds.$id >> >> Should we edit something in the mds map to remo

Re: [ceph-users] Cascading Failure of OSDs

Hi, Quentin Hartman wrote: > Since I have been in ceph-land today, it reminded me that I needed to close > the loop on this. I was finally able to isolate this problem down to a > faulty NIC on the ceph cluster network. It "worked", but it was > accumulating a huge number of Rx errors. My best gu

Re: [ceph-users] Slow performance during recovery operations

Hi, Lionel Bouton wrote : > Sorry this wasn't clear: I tried the ioprio settings before disabling > the deep scrubs and it didn't seem to make a difference when deep scrubs > occured. I have never tested these parameters (osd_disk_thread_ioprio_priority and osd_disk_thread_ioprio_class), but did

Re: [ceph-users] Slow performance during recovery operations

On 04/06/2015 02:54, Lionel Bouton wrote: >> I have never tested these parameters (osd_disk_thread_ioprio_priority and >> osd_disk_thread_ioprio_class), but did you check that the I/O scheduler of >> the disks is cfq? > > Yes I did. Ah ok. It was just in case. :) >> Because, if I understand we

Re: [ceph-users] Slow performance during recovery operations

Mark Nelson wrote: > We ran tests a while back looking at different IO elevators but they are > quite old now: > > http://ceph.com/community/ceph-bobtail-performance-io-scheduler-comparison/ It doesn't seem so interesting to switch from deadline to cfq with HDD. But in this case, I can't use so

Re: [ceph-users] What are you doing to locate performance issues in a Ceph cluster?

Hi, Chris Kitzmiller wrote: > I graph aggregate stats for `ceph --admin-daemon > /var/run/ceph/ceph-osd.$osdid.asok perf dump`. If the max latency strays too > far > outside of my mean latency I know to go look for the troublemaker. My graphs > look something like this: > > [...] Thanks Chri

[ceph-users] [a bit off-topic] Power usage estimation of hardware for Ceph

Hi, Sorry in advance for this thread not directly linked to Ceph. ;) We are thinking about buying servers to build a ceph cluster and we would like to have, if possible, a *approximative* power usage estimation of these servers (this parameter could be important in your choice): 1. the 12xbays su

Re: [ceph-users] Cascading Failure of OSDs

Hi, 01/04/2015 17:28, Quentin Hartman wrote: > Right now we're just scraping the output of ifconfig: > > ifconfig p2p1 | grep -e 'RX\|TX' | grep packets | awk '{print $3}' > > It clunky, but it works. I'm sure there's a cleaner way, but this was > expedient. > > QH Ok, thx for the information

Re: [ceph-users] What are you doing to locate performance issues in a Ceph cluster?

Chris Kitzmiller wrote: >> ~# ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok perf >> >> [...] >> >> "osd": { "opq": 0, >> "op_wip": 0, >> "op": 3566, >> "op_in_bytes": 208803635, >> "op_out_bytes": 146962506, >> "op_latency": { "avgcount": 3566, >> "sum": 100

Re: [ceph-users] What are you doing to locate performance issues in a Ceph cluster?

Hi, Dan Ryder (daryder) wrote: > Yes, the unit is in seconds for those latencies. The sum/avgcount is the > average since the daemon was (re)started. Thanks Dan for this confirmation. > If you're interested, I've co-authored a collectd plugin which captures data > from Ceph daemons - built in

Re: [ceph-users] [a bit off-topic] Power usage estimation of hardware for Ceph

Hi, Christian Balzer wrote: >> I'm not sure to well understand: the model that I indicated in the link >> above (page 2, model SSG-6027R-OSD040H in the table) already have hotswap >> bays in the back, for OS drives. >> > Yes, but that model is pre-configured: > 2x 2.5" 400GB SSDs, 10x 3.5" 4TB S

Re: [ceph-users] [a bit off-topic] Power usage estimation of hardware for Ceph

Chris Kitzmiller wrote: > Just as a single data point I can speak to my own nodes. I'm using SM 847A > [1] chassis. They're 4U, 36 x 3.5" hot swap bays with 2 internal 2.5" bays. > So: > > 30 x 7200 RPM SATA > 6 x SSD Journals > 2 x SSD OS / Mon > 2 x E5-2620 2.0GHz > >

[ceph-users] How to dispatch monitors in a multi-site cluster (ie in 2 datacenters)

Hi, To summarize, my principal question is: in a ceph cluster, is it possible to have, among the monitors, one monitor not necessarily very efficient and with potentially network access latencies and still avoid a negative effect on the cluster? I explain the context of my question because it's i

Re: [ceph-users] How to dispatch monitors in a multi-site cluster (ie in 2 datacenters)

Somnath Roy wrote: > Interesting scenario :-).. IMHO, I don't think cluster will be in healthy > state here if the connections between dc1 and dc2 is cut. The reason is the > following. > > 1. only osd.5 can talk to both data center OSDs and other 2 mons will not > be. So, they can't reach to

Re: [ceph-users] How to dispatch monitors in a multi-site cluster (ie in 2 datacenters)

Gregory Farnum wrote: >> If: (more clear with a schema in the head ;)) >> >> 1. mon.1 and mon.2 can talk together (in dc1) and can talk with mon.5 >> (via the VPN) >>but can't talk with mon.3 and mon.4 (in dc2) >> 2. mon.3 and mon.4 can talk together (in dc2) and can talk with mon

Re: [ceph-users] [a bit off-topic] Power usage estimation of hardware for Ceph

Christian Balzer wrote: > Simply put, a RAID1 of SSDs will require you to get twice as many SSDs as > otherwise needed. And most people don't want to spend that money. > In addition to that DC level SSDs tend to very reliable and your cluster > will have to be able to withstand losses like this an

[ceph-users] Radosgw: upgrade Firefly to Hammer, impossible to create bucket

Hi, On a testing cluster, I have a radosgw on Firefly and the other nodes, OSDs and monitors, are on Hammer. The nodes are installed with puppet in personal VM, so I can reproduce the problem. Generally, I use s3cmd to check the radosgw. While radosgw is on Firefly, I can create bucket, no problem

Re: [ceph-users] Radosgw: upgrade Firefly to Hammer, impossible to create bucket

Karan Singh wrote: > Things you can check > > * Is RGW node able to resolve bucket-2.ostore.athome.priv , try ping > bucket-2.ostore.athome.priv Yes, my DNS configuration is ok. In fact, I test s3cmd directly on my radosgw (its hostname is "ceph-radosgw1" but its fqdn is "ostore.athome.priv")

Re: [ceph-users] Purpose of the s3gw.fcgi script?

Hi, Yehuda Sadeh-Weinraub wrote: > You're not missing anything. The script was only needed when we used > the process manager of the fastcgi module, but it has been very long > since we stopped using it. Just to be sure, so if I understand well, these parts of the documentation: 1. http://

Re: [ceph-users] norecover and nobackfill

Hi, Robert LeBlanc wrote: > What I'm trying to achieve is minimal data movement when I have to service > a node to replace a failed drive. [...] I will perhaps say something stupid but it seems to me that it's the goal of the "noout" flag, isn't it? 1. ceph osd set noout 2. an old OSD disk fail

Re: [ceph-users] Radosgw: upgrade Firefly to Hammer, impossible to create bucket

Hi, Yehuda Sadeh-Weinraub wrote: > The 405 in this case usually means that rgw failed to translate the http > hostname header into > a bucket name. Do you have 'rgw dns name' set correctly? Ah, I have found and indeed it concerned "rgw dns name" as also Karan thought. ;) But it's a little cur

Re: [ceph-users] How to dispatch monitors in a multi-site cluster (ie in 2 datacenters)

Joao Eduardo wrote: > To be more precise, it's the lowest IP:PORT combination: > > 10.0.1.2:6789 = rank 0 > 10.0.1.2:6790 = rank 1 > 10.0.1.3:6789 = rank 3 > > and so on. Ok, so if there is 2 possible quorum, the quorum with the lowest IP:PORT will be chosen. But what happens if, in the 2 possi

Re: [ceph-users] norecover and nobackfill

Robert LeBlanc wrote: > HmmmI've been deleting the OSD (ceph osd rm X; ceph osd crush rm osd.X) > along with removing the auth key. This has caused data movement, Maybe but if the flag "noout" is set, removing an OSD of the cluster doesn't trigger at all data movement (I have tested with Fire

Re: [ceph-users] Upgrade from Firefly to Hammer

Hi, Garg, Pankaj wrote: > I have a small cluster of 7 machines. Can I just individually upgrade each of > them (using apt-get upgrade) from Firefly to Hammer release, or there more to > it than that? Not exactly, this is "individually" which is not correct. ;) You should indeed "apt-get upgrad

[ceph-users] Questions about an example of ceph infrastructure

Hi, We are thinking about a ceph infrastructure and I have questions. Here is the conceived (but not yet implemented) infrastructure: (please, be careful to read the schema with a monospace font ;)) +-+ | users | |(browser)|

[ceph-users] What is a "dirty" object

Hi, With my testing cluster (Hammer on Ubuntu 14.04), I have this: -- ~# ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 4073G 3897G 176G 4.33 23506 POOLS: NAME

Re: [ceph-users] Questions about an example of ceph infrastructure

Hi, Christian Balzer wrote: > For starters, make that 5 MONs. > It won't really help you with your problem of keeping a quorum when > loosing a DC, but being able to loose more than 1 monitor will come in > handy. > Note that MONs don't really need to be dedicated nodes, if you know what > you'r

Re: [ceph-users] What is a "dirty" object

Hi, John Spray wrote: > As far as I can see, this is only meaningful for cache pools, and object is > "dirty" in the sense of having been created or modified since their its last > flush. For a non-cache-tier pool, everything is logically dirty since it is > never flushed. > > I hadn't noti

Re: [ceph-users] Some more numbers - CPU/Memory suggestions for OSDs and Monitors

Hi, Christian Balzer wrote: >> thanks for the feedback regarding the network questions. Currently I try >> to solve the question of how much memory, cores and GHz for OSD nodes >> and Monitors. >> >> My research so far: >> >> OSD nodes: 2 GB RAM, 2 GHz, 1 Core (?) per OSD >> > RAM is enough, but

Re: [ceph-users] decrease pg number

Hi, Pavel V. Kaygorodov wrote: > I have updated my cluster to Hammer and got a warning "too many PGs > per OSD (2240 > max 300)". I know, that there is no way to decrease > number of page groups, so I want to re-create my pools with less pg > number, move all my data to them, delete old pools and

[ceph-users] Radosgw and mds hardware configuration

Hi Cephers, :) I would like to know if there are some rules to estimate (approximatively) the need of CPU and RAM for: 1. a radosgw server (for instance with Hammer and civetweb). 2. a mds server If I am not mistaken, for these 2 types of server, there is no need concerning the storage. For a

[ceph-users] Cephfs: proportion of data between data pool and metadata pool

Hi, When I want to have an estimation of the pg_num of a new pool, I use this very useful page: http://ceph.com/pgcalc/. In the table, I must give the %data of a pool. For instance, for a "rados gateway only" use case, I can see that, by default, the page gives: - .rgw.buckets => 96.90% of data -

Re: [ceph-users] Some more numbers - CPU/Memory suggestions for OSDs and Monitors

Mark Nelson wrote: > I'm not sure who came up with the 1GB for each 1TB of OSD daemons rule, but > frankly I don't think it scales well at the extremes. You can't get by with > 256MB of ram for OSDs backed by 256GB SSDs, nor do you need 6GB of ram per > OSD for 6TB spinning disks. > > 2-4GB o

Re: [ceph-users] Find out the location of OSD Journal

Hi, Patrik Plank wrote: > i cant remember on which drive I install which OSD journal :-|| > Is there any command to show this? It's probably not the answer you hope, but why don't use a simple: ls -l /var/lib/ceph/osd/ceph-$id/journal ? -- François Lafont

[ceph-users] Complete freeze of a cephfs client (unavoidable hard reboot)

Hi, I had a problem with a cephfs freeze in a client. Impossible to re-enable the mountpoint. A simple "ls /mnt" command totally blocked (of course impossible to umount-remount etc.) and I had to reboot the host. But even a "normal" reboot didn't work, the host didn't stop. I had to do a hard rebo

Re: [ceph-users] Complete freeze of a cephfs client (unavoidable hard reboot)

Hi, Sorry for my late answer. Gregory Farnum wrote: >> 1. Is this kind of freeze normal? Can I avoid these freezes with a >> more recent version of the kernel in the client? > > Yes, it's normal. Although you should have been able to do a lazy > and/or force umount. :) Ah, I haven't tried it.

Re: [ceph-users] Complete freeze of a cephfs client (unavoidable hard reboot)

John Spray wrote: > Greg's response is pretty comprehensive, but for completeness I'll add that > the specific case of shutdown blocking is http://tracker.ceph.com/issues/9477 Yes indeed, during the freeze, "INFO: task sync:3132 blocked for more than 120 seconds..." was exactly the message I ha

Re: [ceph-users] How to backup hundreds or thousands of TB

Hi, Wido den Hollander wrote: > Aren't snapshots something that should protect you against removal? IF > snapshots work properly in CephFS you could create a snapshot every hour. Are you talking about the .snap/ directory in a cephfs directory? If yes, does it work well? Because, with Hammer, i

Re: [ceph-users] Mount options nodcache and nofsc

Hi, Yan, Zheng wrote: > fsc means fs-cache. it's a kernel facility by which a network > filesystem can cache data locally, trading disk space to gain > performance improvements for access to slow networks and media. cephfs > does not use fs-cache by default. So enable this option can improve pe

Re: [ceph-users] Cephfs: one ceph account per directory?

Hi, Gregory Farnum wrote: >> 1. Can you confirm to me that currently it's impossible to restrict the read >> and write access of a ceph account to a specific directory of a cephfs? > > It's sadly impossible to restrict access to the filesystem hierarchy > at this time, yes. By making use of the

Re: [ceph-users] Complete freeze of a cephfs client (unavoidable hard reboot)

Hi, On 27/05/2015 22:34, Gregory Farnum wrote: > Sorry for the delay; I've been traveling. No problem, me too, I'm not really fast to answer. ;) >> Ok, I see. According to the online documentation, the way to close >> a cephfs client session is: >> >> ceph daemon mds.$id session ls

Re: [ceph-users] v0.94.2 Hammer released

Hi, On 11/06/2015 19:34, Sage Weil wrote: > Bug #11442 introduced a change that made rgw objects that start with > underscore incompatible with previous versions. The fix to that bug > reverts to the previous behavior. In order to be able to access objects > that start with an underscore and w

Re: [ceph-users] CephFS: 'ls -alR' performance terrible unless Linux cache flushed

Hi, On 16/06/2015 18:46, negillen negillen wrote: > Fixed! At least looks like fixed. That's cool for you. ;) > It seems that after migrating every node (both servers and clients) from > kernel 3.10.80-1 to 4.0.4-1 the issue disappeared. > Now I get decent speeds both for reading files and for

[ceph-users] Add osd to a Ceph cluster : how to choose the osd id?

Hi, I use Ceph firefly (0.80.6) on Ubuntu Trusty (14.04). When I add a new osd to a Ceph cluster, I run these commands : uuid=$(uuidgen) osd_id=$(ceph --cluster "my_cluster" osd create "$uuid") printf "The id of this osd will be $osd_id.\n" And the osd id is chosen automatically by t

Re: [ceph-users] Add osd to a Ceph cluster : how to choose the osd id?

Le 02/10/2014 00:53, Loic Dachary a écrit : > Hi François, Hello, > It's probably better to leave the OSD id to the Ceph cluster. Ah, ok. > Why do you need it ? It's just to have: srv1 172.31.10.1 --> osd-1 srv2 172.31.10.2 --> osd-2 srv3 172.31.10.3 --> osd-3 It's more friendly than: srv1

[ceph-users] Have 2 different public networks

Hi, Is it possible to have 2 different public networks in a Ceph cluster? I explain my question below. Currently, I have 3 identical nodes in my Ceph cluster. Each node has: - only 1 monitor; - n osds (we don't care about the value n here); - and 3 interfaces. One interface for the "cluster" ne

Re: [ceph-users] Have 2 different public networks

Le 19/12/2014 02:18, Craig Lewis a écrit : > The daemons bind to *, Yes but *only* for the OSD daemon. Am I wrong? Personally I must provide IP addresses for the monitors in the /etc/ceph/ceph.conf, like this: [global] mon host = 10.0.1.1, 10.0.1.2, 10.0.1.3 Or like this: [mon.1] mon addr = 1

Re: [ceph-users] 0.88

Hi, Le 19/12/2014 15:57, Loic Dachary a écrit : > The stable release have real names, that is what makes them different from > development releases (dumpling, emperor, firefly, giant, hammer). And I add that, from what I understand, one time in two the release is LTS (Long Time Support). Firefl

Re: [ceph-users] Have 2 different public networks

[Oh, sorry Craig for my mistake: I sent my response to your personal address instead of sending it to the list. Sorry for the duplicate. I send my message to list] Hello, Le 19/12/2014 19:17, Craig Lewis a écrit : > I'm not using mon addr lines, and my ceph-mon daemons are bound to 0.0.0.0:*. A

Re: [ceph-users] Have 2 different public networks

I react to this point. Le 20/12/2014 02:14, Francois Lafont a écrit : > when I create my cluster with the > first monitor, I have to generate a monitor map with this > command: > > monmaptool --create --add {hostname} {ip-address} --fsid {uuid}

Re: [ceph-users] Have 2 different public networks

Le 20/12/2014 02:18, Craig Lewis a écrit : >> And do you have several IP addresses on your server? >> Can you contact the *same* monitor process with different IP addresses? >> For instance: >> telnet -e ']' ip_addr1 6789 >> telnet -e ']' ip_addr2 6789 >> > > Oh. The second one fails, ev

[ceph-users] Installation of 2 radosgw, ceph username and instance

Hi, I have a Ceph cluster that works correctly (Firefly on Ubuntu Trusty servers). I would like to install a radosgw. In fact, I would like install 2 radosgw: radosgw-1 and radosgw-2 with a floating IP address to support failover etc. After reading the doc, I still have a point that is not clear

Re: [ceph-users] CephFS file to rados object mapping

Hi, On 08/10/2015 22:25, Gregory Farnum wrote: > So that means there's no automated way to guarantee the right copy of > an object when scrubbing. If you have 3+ copies I'd recommend checking > each of them and picking the one that's duplicated... It's curious because I have already tried with c

Re: [ceph-users] CephFS file to rados object mapping

Hi, Thanks for your answer Greg. On 09/10/2015 04:11, Gregory Farnum wrote: > The size of the on-disk file didn't match the OSD's record of the > object size, so it rejected it. This works for that kind of gross > change, but it won't catch stuff like a partial overwrite or loss of > data within

Re: [ceph-users] v9.1.0 Infernalis release candidate released

Hi and thanks at all for this good news, ;) On 13/10/2015 23:01, Sage Weil wrote: >#. Fix the data ownership during the upgrade. This is the preferred > option, > but is more work. The process for each host would be to: > > #. Upgrade the ceph package. This creates the ceph u

Re: [ceph-users] v9.1.0 Infernalis release candidate released

Sorry, another remark. On 13/10/2015 23:01, Sage Weil wrote: > The v9.1.0 packages are pushed to the development release repositories:: > > http://download.ceph.com/rpm-testing > http://download.ceph.com/debian-testing I don't see the 9.1.0 available for Ubuntu Trusty : http://downloa

Re: [ceph-users] CephFS file to rados object mapping

Hi, On 14/10/2015 06:45, Gregory Farnum wrote: >> Ok, however during my tests I had been careful to replace the correct >> file by a bad file with *exactly* the same size (the content of the >> file was just a little string and I have changed it by a string with >> exactly the same size). I had b

Re: [ceph-users] v0.94.4 Hammer released

Hi, On 20/10/2015 20:11, Stefan Eriksson wrote: > A change like this below, where we have to change ownership was not add to a > point release for hammer right? Right. ;) I have upgraded my ceph cluster from 0.94.3 to 0.94.4 today without any problem. The daemons used in 0.94.3 and currently u

Re: [ceph-users] v9.2.0 Infernalis released

Hi, I have just upgraded a cluster to 9.2.0 from 9.1.0. All seems to be well except I have this little error message : ~# ceph tell mon.* version --format plain mon.1: ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314e58) mon.2: ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314e58

Re: [ceph-users] v9.2.0 Infernalis released

On 09/11/2015 06:28, Francois Lafont wrote: > I have just upgraded a cluster to 9.2.0 from 9.1.0. > All seems to be well except I have this little error > message : > > ~# ceph tell mon.* version --format plain > mon.1: ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314

Re: [ceph-users] v9.2.0 Infernalis released

Oops, sorry Dan, I would like to send my message to the list. Sorry. > On Mon, Nov 9, 2015 at 11:55 AM, Francois Lafont >> >> 1. Ok, so, the rank of my monitors are 0, 1, 2 but the its ID are 1, 2, 3 >> (ID chosen by himself because the hosts are called ceph01, ceph02 and &

Re: [ceph-users] about PG_Number

Hi, On 13/11/2015 09:13, Vickie ch wrote: > If you have a large amount of OSDs but less pg number. You will find your > data write unevenly. > Some OSD have no change to write data. > In the other side, pg number too large but OSD number too small that have a > chance to cause data lost. Data lo

[ceph-users] Meaning of the "host" parameter in the section [client.radosgw.{instance-name}] in ceph.conf?

Hi, a) My first question is perfectly summarized in the title. ;) Indeed, here is a typical section [client.radosgw.{instance-name}] in the ceph.conf of a radosgw server "rgw-01": -- # The instance-name is "gateway" here. [client.radosgw.gateway

Re: [ceph-users] Meaning of the "host" parameter in the section [client.radosgw.{instance-name}] in ceph.conf?

Hi, On 26/05/2016 23:46, Francois Lafont wrote: > a) My first question is perfectly summarized in the title. ;) > Indeed, here is a typical section [client.radosgw.{instance-name}] in > the ceph.conf of a radosgw serve

[ceph-users] Infernalis => Jewel: ceph-fuse regression concerning the automatic mount at boot?

Hi, I have a Jewel Ceph cluster in OK state and I have a "ceph-fuse" Ubuntu Trusty client with ceph Infernalis. The cephfs is mounted automatically and perfectly during the boot via ceph-fuse and this line in /etc/fstab : ~# grep ceph /etc/fstab id=cephfs,keyring=/etc/ceph/ceph.client.cephfs.keyr

Re: [ceph-users] Infernalis => Jewel: ceph-fuse regression concerning the automatic mount at boot?

Hi, On 01/06/2016 23:16, Florent B wrote: > Don't have this problem on Debian migration from Infernalis to Jewel, > check all permissions... Ok, it's probably the reason (I hope) but currently I don't find the good unix rights. I have this (which doesn't work): ~# ll -d /etc/ceph drwxr-xr-x 2 r

Re: [ceph-users] Infernalis => Jewel: ceph-fuse regression concerning the automatic mount at boot?

Now, I have a explanation and it's _very_ strange, absolutely not related to a problem of Unix rights. For memory, my client node is an updated Ubuntu Trusty and I use ceph-fuse. Here is my fstab line: ~# grep ceph /etc/fstab id=cephfs,keyring=/etc/ceph/ceph.client.cephfs.keyring,client_mountpoint

Re: [ceph-users] Infernalis => Jewel: ceph-fuse regression concerning the automatic mount at boot?

Hi, On 02/06/2016 04:44, Francois Lafont wrote: > ~# grep ceph /etc/fstab > id=cephfs,keyring=/etc/ceph/ceph.client.cephfs.keyring,client_mountpoint=/ > /mnt/ fuse.ceph noatime,nonempty,defaults,_netdev 0 0 [...] > And I have rebooted. After the reboot, big surprise with this:

Re: [ceph-users] jewel upgrade and sortbitwise

Hi, On 03/06/2016 05:39, Samuel Just wrote: > Due to http://tracker.ceph.com/issues/16113, it would be best to avoid > setting the sortbitwise flag on jewel clusters upgraded from previous > versions until we get a point release out with a fix. > > The symptom is that setting the sortbitwise fl

Re: [ceph-users] jewel upgrade and sortbitwise

Hi, On 03/06/2016 16:29, Samuel Just wrote: > Sorry, I should have been more clear. The bug actually is due to a > difference in an on disk encoding from hammer. An infernalis cluster would > never had had such encodings and is fine. Ah ok, fine. ;) Thanks for the answer. Bye. -- François Lafo

[ceph-users] A radosgw keyring with the minimal rights, which pools have I to create?

Hi, In a from scratch Jewel cluster, I'm searching the exact list of pools I have to create and the minimal rights that I can set for the keyring used by the radosgw instance. This is for the default zone. I intend to just use the S3 API of the radosgw. a) I have read the doc here http://docs.ce

[ceph-users] ceph-fuse, fio largely better after migration Infernalis to Jewel, is my bench relevant?

Hi, I have a little Ceph cluster in production with 5 cluster nodes and 2 client nodes. The clients are using cephfs via fuse.ceph. Recently, I have upgraded my cluster from Infernalis to Jewel (servers _and_ clients). When the cluster was in Infernalis version the fio command below gave me appro

Re: [ceph-users] ceph-fuse, fio largely better after migration Infernalis to Jewel, is my bench relevant?

On 06/06/2016 18:41, Gregory Farnum wrote: > We had several metadata caching improvements in ceph-fuse recently which I > think went in after Infernalis. That could explain it. Ok, in this case, it could be good news. ;) I had doubts concerning my fio bench. I know that benchs can be tricky esp

Re: [ceph-users] ceph-mon memory issue jewel 10.2.5 kernel 4.4

Hi @all, On 02/08/2017 08:45 PM, Jim Kilborn wrote: > I have had two ceph monitor nodes generate swap space alerts this week. > Looking at the memory, I see ceph-mon using a lot of memory and most of the > swap space. My ceph nodes have 128GB mem, with 2GB swap (I know the > memory/swap ratio

Re: [ceph-users] ceph-fuse "Transport endpoint is not connected" on Jewel 10.2.2

Hi, I had exactly the same error in my production ceph client node with Jewel 10.2.1 in my case. In the client node : - Ubuntu 14.04 - kernel 3.13.0-92-generic - ceph 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269) - cephfs via _ceph-fuse_ In the cluster node : - Ubuntu 14.04 - kernel 3.13.0-9

Re: [ceph-users] ceph-fuse "Transport endpoint is not connected" on Jewel 10.2.2

On 08/27/2016 12:01 PM, Francois Lafont wrote: > I had exactly the same error in my production ceph client node with > Jewel 10.2.1 in my case. I have forgotten to say that the ceph cluster was perfectly HEALTH_OK before, during and after the error in the client side. R

Re: [ceph-users] ceph-fuse "Transport endpoint is not connected" on Jewel 10.2.2

Hi, On 08/29/2016 08:30 PM, Gregory Farnum wrote: > Ha, yep, that's one of the bugs Giancolo found: > > ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269) > 1: (()+0x299152) [0x7f91398dc152] > 2: (()+0x10330) [0x7f9138bbb330] > 3: (Client::get_root_ino()+0x10) [0x7f91397df6c0] >

[ceph-users] Keep previous versions of ceph in the APT repository

Hi @all, Ceph teaem, could it be possible to keep the previous versions of ceph* packages in the APT repository? Indeed, for instance for Ubuntu Trusty, currently we have: ~$ curl -s http://download.ceph.com/debian-jewel/dists/trusty/main/binary-amd64/Packages | grep -A 1 '^Package: ceph$'

Re: [ceph-users] 10.2.4 Jewel released

Hi, On 12/07/2016 01:21 PM, Abhishek L wrote: > This point release fixes several important bugs in RBD mirroring, RGW > multi-site, CephFS, and RADOS. > > We recommend that all v10.2.x users upgrade. Also note the following when > upgrading from hammer Well... little warning: after upgrade fro

Re: [ceph-users] 10.2.4 Jewel released

On 12/07/2016 11:16 PM, Steve Taylor wrote: > I'm seeing the same behavior with very similar perf top output. One server > with 32 OSDs has a load average approaching 800. No excessive memory usage > and no iowait at all. Exactly! And another interesting information (maybe). I have ceph-osd pro

Re: [ceph-users] 10.2.4 Jewel released

On 12/07/2016 11:33 PM, Ruben Kerkhof wrote: > Thanks, l'll check how long it takes for this to happen on my cluster. > > I did just pause scrub and deep-scrub. Are there scrubs running on > your cluster now by any chance? Yes but normally not currently because I have: osd scrub begin hour =

  1   2   >