[ceph-users] removed_snaps in ceph osd dump?

2015-06-15 Thread Jan Schermer
Hi, I have ~1800 removed_snaps listed in the output of “ceph osd dump”. Is that allright? Any way to get rid of those? What’s the significance? Thanks Jan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-use

Re: [ceph-users] CephFS client issue

2015-06-15 Thread John Spray
On 14/06/15 20:00, Matteo Dacrema wrote: Hi Lincoln, I'm using the kernel client. Kernel version is: 3.13.0-53-generic​ That's old by CephFS standards. It's likely that the issue you're seeing is one of the known bugs (which were actually the motivation for adding the warning message

[ceph-users] ec pool history objects

2015-06-15 Thread 池信泽
hi, all: when I use ec poll, I see there are some object history for object xx. [root@node3 2.1d6s2_head]# ll -R | grep xx -rw-r--r--. 1 root root 65536 Jun 15 17:41 xx__head_610951D6__2_fe1_2 -rw-r--r--. 1 root root 65536 Jun 15 17:41 xx__head_610951D6__2_fe2_2 -rw-r--r--.

[ceph-users] ec pool history objects

2015-06-15 Thread 池信泽
hi, all: when I use ec poll, I see there are some object history for object xx. Such as: xx__head_610951D6__2_fe1_2, xx__head_610951D6__2_fe2_2 xx__head_610951D6__2__2 I think this object is used for roll_back when not all shards have written object to

Re: [ceph-users] CephFS client issue

2015-06-15 Thread Matteo Dacrema
Ok, I'll update kernel to 3.16.3 version and let you know. Thanks, Matteo Da: John Spray Inviato: luned? 15 giugno 2015 10:51 A: Matteo Dacrema; Lincoln Bryant; ceph-users Oggetto: Re: [ceph-users] CephFS client issue On 14/06/15 20:00, Matteo Dacrema wrote:

[ceph-users] Rebalancing two nodes simultaneously

2015-06-15 Thread Lindsay Mathieson
If I have two nodes with identical drive/osd setups Drive 1 = 3TB Drive 2 = 1TB Drive 3 = 1TB All with equal weights of (1) I now decide to reweight Drive 1 to (3). Would it be best to do one node at a time, or do both nodes simultaneously? I would presume that all the data shuffling would be i

[ceph-users] cephfs unmounts itself from time to time

2015-06-15 Thread Roland Giesler
I have a small cluster of 4 machines and quite a few drives. After about 2 - 3 weeks cephfs fails. It's not properly mounted anymore in /mnt/cephfs, which of course causes the VM's running to fail too. In /var/log/syslog I have "/mnt/cephfs: File exists at /usr/share/perl5/PVE/Storage/DirPlugin.

Re: [ceph-users] cephfs unmounts itself from time to time

2015-06-15 Thread Gregory Farnum
On Mon, Jun 15, 2015 at 4:03 AM, Roland Giesler wrote: > I have a small cluster of 4 machines and quite a few drives. After about 2 > - 3 weeks cephfs fails. It's not properly mounted anymore in /mnt/cephfs, > which of course causes the VM's running to fail too. > > In /var/log/syslog I have "/m

[ceph-users] Rebalancing two nodes simultaneously

2015-06-15 Thread Lindsay Mathieson
If I have two nodes with identical drive/osd setups Drive 1 = 3TB Drive 2 = 1TB Drive 3 = 1TB All with equal weights of (1) I now decide to reweight Drive 1 to (3). Would it be best to do one node at a time, or do both nodes simultaneously? I would presume that all the data shuffling would be i

Re: [ceph-users] Ceph OSD with OCFS2

2015-06-15 Thread gjprabu
Hi The size differ issue is solved, This is related to ocfs2 format option and -C count should be 4K. (mkfs.ocfs2 /dev/mapper/mpatha -N 64 -b 4K -C 256K -T mail --fs-features=extended-slotmap --fs-feature-level=max-features -L ) Need to change like below. (mkfs.ocfs2 /dev/map

Re: [ceph-users] Rebalancing two nodes simultaneously

2015-06-15 Thread Lindsay Mathieson
On 15 June 2015 at 21:16, Lindsay Mathieson wrote: > p,s Is there a way to speed up the rebalance? the cluster is unused > overnight, so I can thrash the IO I bumped max_backfills to 20 and recovery max active to 30 using inject args. Nothing seems to be breaking yet :) I/O delay seems to be a

Re: [ceph-users] What is link and unlink options used for in radosgw-admin

2015-06-15 Thread WCMinor
Wenjun Huang writes: > > > Hello, everyone > I am now confused with the options of link & unlink in radosgw-admin utility. > > In my option, if I link the ownerA’s bucketA to ownerB through the command below: > > radosgw-admin bucket link —uid=ownerB —bucket=bucketA > > then, I think the own

Re: [ceph-users] anyone using CephFS for HPC?

2015-06-15 Thread Barclay Jameson
I am currently implementing Ceph into our HPC environment to handle SAS temp workspace. I am starting out with 3 OSD nodes with 1 MON/MDS node. 16 4TB HDDs per OSD node with 4 120GB SSD. Each node has 40Gb Mellanox interconnect between each other to a Mellanox switch. Each client node has 10Gb to s

Re: [ceph-users] NFS interaction with RBD

2015-06-15 Thread Simon Leinen
Trent Lloyd writes: > Jens-Christian Fischer writes: >> >> I think we (i.e. Christian) found the problem: >> We created a test VM with 9 mounted RBD volumes (no NFS server). As soon as > he hit all disks, we started to experience these 120 second timeouts. We > realized that the QEMU process on

Re: [ceph-users] NFS interaction with RBD

2015-06-15 Thread Simon Leinen
Christian Schnidrig writes: > Well that’s strange. I wonder why our systems behave so differently. One point about our cluster (I work with Christian, who's still on vacation, and Jens-Christian) is that it has 124 OSDs and 2048 PGs (I think) in the pool used for these RBD volumes. As a result, e

[ceph-users] need help

2015-06-15 Thread Ranjan, Jyoti
Hi, I have been trying to deploy ceph rados gateway on a single node but failing for a while. My ceph cluster is with three osd and looks fine. I could create gateway user but user is not able to create bucket. I am getting below error ... Traceback (most recent call last): File "s3test.py"

[ceph-users] unfound object(s)

2015-06-15 Thread GuangYang
Hello Cephers, On one of our production clusters, there is one *unfound* object reported which make the PG stuck at recovering. While trying to recover the object, I failed to find a way to tell which object is unfound. I tried:   1> PG query   2> Grep from monitor log Did I miss anything?   T

Re: [ceph-users] unfound object(s)

2015-06-15 Thread GuangYang
Thanks to Sam, we can use:   ceph pg list_missing to get the list of unfound objects. Thanks, Guang > From: yguan...@outlook.com > To: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com > Date: Mon, 15 Jun 2015 16:46:53 + > Subject: [ceph-users] u

[ceph-users] firefly to giant upgrade broke ceph-gw

2015-06-15 Thread Michael Kuriger
Hi all, I recently upgraded my 2 ceph clusters from firefly to giant. After the update, ceph gateway has some issues. I¹ve even gone so far as to completely remove all gateway related pools and recreated from scratch. I can write data into the gateway, and that seems to work (most of the time) b

[ceph-users] CephFS: 'ls -alR' performance terrible unless Linux cache flushed

2015-06-15 Thread negillen negillen
Hello everyone, something very strange is driving me crazy with CephFS (kernel driver). I copy a large directory on the CephFS from one node. If I try to perform a 'time ls -alR' on that directory it gets executed in less than one second. If I try to do the same 'time ls -alR' from another node it

[ceph-users] CephFS: 'ls -alR' performance terrible unless Linux cache flushed

2015-06-15 Thread negillen negillen
Hello everyone, something very strange is driving me crazy with CephFS (kernel driver). I copy a large directory on the CephFS from one node. If I try to perform a 'time ls -alR' on that directory it gets executed in less than one second. If I try to do the same 'time ls -alR' from another node it

[ceph-users] RADOS Bench

2015-06-15 Thread Garg, Pankaj
Hi, I have a few machines in my Ceph Cluster. I have another machine that I use to run RADOS Bench to get the performance. I am now seeing numbers around 1100 MB/Sec, which is quite close to saturation point of the 10Gbps link. I'd like to understand what does the total bandwidth number represen

Re: [ceph-users] RADOS Bench

2015-06-15 Thread Somnath Roy
Pankaj, It is the cumulative BW of ceph cluster but you will be limited by your single client BW always. To verify if you are single client 10Gb network limited or not, put another client and see if it is scaling or not. Thanks & Regards Somnath From: ceph-users [mailto:ceph-users-boun...@lists

Re: [ceph-users] RADOS Bench

2015-06-15 Thread Garg, Pankaj
Thanks Somnath. Do you mean that I should run Rados Bench in parallel on 2 different clients? Is there a way to run Rados Bench from 2 clients, so that they run in parallel, except launching them together manually? From: Somnath Roy [mailto:somnath@sandisk.com] Sent: Monday, June 15, 2015 1:

[ceph-users] Fwd: Too many PGs

2015-06-15 Thread Marek Dohojda
I hate to bug, but I truly hope someone has an answer to below. Thank you kindly! -- Forwarded message -- From: Marek Dohojda Date: Wed, Jun 10, 2015 at 7:49 AM Subject: Too many PGs To: ceph-users-requ...@lists.ceph.com Hello I am running “Hammer” Ceph and I am getting follow

Re: [ceph-users] RADOS Bench

2015-06-15 Thread Somnath Roy
No, you need to launch them manually... Here is my thought. 1. Say running 4 instances of rados clients from 4 different console you are getting 1100 MB/s as you said. 2. Now , say running 4 more instances from another client with 10 Gb, you are able to scale it more. This means you are limite

Re: [ceph-users] Fwd: Too many PGs

2015-06-15 Thread Somnath Roy
If you want to suppress the warning, do this in the conf file.. mon_pg_warn_max_per_osd = 0 or mon_pg_warn_max_per_osd = Thanks & Regards Somnath From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Marek Dohojda Sent: Monday, June 15, 2015 1:05 PM To: ceph-users@l

[ceph-users] help to new user

2015-06-15 Thread vida ahmadi
Dear all, I am new user in ceph and I would like to install ceph with minimum requirement.I read in some documents to put ceph components(OSDs,MON,MDSs) in different virtual machine. Is it good idea for starting? Please let me know about your suggestion and experience. Thank you in advance. -- Bes

Re: [ceph-users] help to new user

2015-06-15 Thread Shane Gibson
Vida - installing Ceph as hosted VMs is a great way to get experience "hands-on" with a Ceph cluster. It is NOT a good way to run Ceph for any real work load.NOTE that it's critical you structure your virtual disks and virtual network(s) to match how you'd like to run your Ceph work loads

Re: [ceph-users] CephFS client issue

2015-06-15 Thread Matteo Dacrema
With 3.16.3 kernel it seems to be stable but I've discovered one new issue. If I take down one of the two osd node all the client stop to respond. Here the output of ceph -s ceph -s cluster 2de7b17f-0a3e-4109-b878-c035dd2f7735 health HEALTH_WARN 256 pgs degraded

Re: [ceph-users] CephFS client issue

2015-06-15 Thread Christian Balzer
Hello, On Mon, 15 Jun 2015 23:11:07 + Matteo Dacrema wrote: > With 3.16.3 kernel it seems to be stable but I've discovered one new > issue. > > If I take down one of the two osd node all the client stop to respond. > How did you take the node down? What is your "osd_pool_default_min_size"

Re: [ceph-users] anyone using CephFS for HPC?

2015-06-15 Thread Shinobu Kinjo
Thanks for your info. I would like to know how large i/o that you mentioned, and what kind of app you used to do benchmarking? Sincerely, Kinjo On Tue, Jun 16, 2015 at 12:04 AM, Barclay Jameson wrote: > I am currently implementing Ceph into our HPC environment to handle > SAS temp workspace. >

Re: [ceph-users] Ceph OSD with OCFS2

2015-06-15 Thread gjprabu
Hi Somnath, Is there any fine tune for the blow issues. << Also please let us know the reason ( Extra 2-3 mins is taken for hg / git repository operation like clone , pull , checkout and update.) << Could you please explain a bit what you are trying to do here ? In ceph sha

Re: [ceph-users] Ceph OSD with OCFS2

2015-06-15 Thread Somnath Roy
Prabu, I am still not clear.. You are cloning git source repository on top of RBD + OCFS2 and that is taking extra time ? Thanks & Regards Somnath From: gjprabu [mailto:gjpr...@zohocorp.com] Sent: Monday, June 15, 2015 9:39 PM To: gjprabu Cc: Somnath Roy; Kamala Subramani; ceph-users@lis