[ceph-users] Fw: query about mapping of Swift/S3 APIs to Ceph cluster APIs

please somebody answer my queries. -RegardsPragya JainDepartment of Computer ScienceUniversity of DelhiDelhi, India On Saturday, 14 March 2015 3:34 PM, pragya jain wrote: Hello all! I am working on Ceph object storage architecture from last few months. I am unable to search  a

[ceph-users] query about region and zone creation while configuring RADOSGW

hello all! I am working on Ceph object storage architecture.I have some queries: In case of configuring federated system, we need to create regions containing one or more zones and the cluster must have a master region and each region must have a master zone. but in case of simple gateway configu

Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !

>>That full system slows down, OK, but brutal stop... This is strange, that could be: - qemu crash, maybe a bug in rbd block storage (if you use librbd) - oom-killer on you host (any logs ?) what is your qemu version ? - Mail original - De: "Florent Bautista" À: "ceph-users" Envoyé:

Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !

On 16/03/2015, at 11.14, Florent B wrote: > On 03/16/2015 11:03 AM, Alexandre DERUMIER wrote: >> This is strange, that could be: >> >> - qemu crash, maybe a bug in rbd block storage (if you use librbd) >> - oom-killer on you host (any logs ?) >> >> what is your qemu version ? >> > > Now, we

Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !

>>We use Proxmox, so I think it uses librbd ? As It's me that I made the proxmox rbd plugin, I can confirm that yes, it's librbd ;) Is the ceph cluster on dedicated nodes ? or vms are running on same nodes than osd daemons ? >>And I precise that not all VMs on that pool crashed, only some of

Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !

>>VMs are running on the same nodes than OSD Are you sure that you didn't some kind of out of memory. pg rebalance can be memory hungry. (depend how many osd you have). do you see oom-killer in your host logs ? - Mail original - De: "Florent Bautista" À: "aderumier" Cc: "ceph-users"

Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !

On 16/03/2015, at 12.23, Alexandre DERUMIER wrote: >>> We use Proxmox, so I think it uses librbd ? > > As It's me that I made the proxmox rbd plugin, I can confirm that yes, it's > librbd ;) > Is the ceph cluster on dedicated nodes ? or vms are running on same nodes > than osd daemons ? My c

[ceph-users] PHP Rados failed in read operation if object size is large (say more than 10 MB )

running on ubuntu with nginx + php-fpm ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !

May I know your ceph version.?. The latest version of firefly 80.9 has patches to avoid excessive data migrations during rewighting osds. You may need set a tunable inorder make this patch active. This is a bugfix release for firefly. It fixes a performance regression in librbd, an important CRUS

Re: [ceph-users] PHP Rados failed in read operation if object size is large (say more than 10 MB )

On 03/16/2015 01:55 PM, Gaurang Vyas wrote: > running on ubuntu with nginx + php-fpm > > $rados = rados_create('admin'); > > > rados_conf_read_file($rados, '/etc/ceph/ceph.conf'); > rados_conf_set($rados, 'keyring','/etc/ceph/ceph.client.admin.keyring'); > > $temp = rados_conf_get($rados, "rad

Re: [ceph-users] Ceph release timeline

Great work ! David Moreau Simard On 2015-03-15 06:29 PM, Loic Dachary wrote: > Hi Ceph, > > In an attempt to clarify what Ceph release is stable, LTS or development. a > new page was added to the documentation: > http://ceph.com/docs/master/releases/ It is a matrix where each cell is a > relea

Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !

I'm using the latest Giant and have the same issue. When i increase PG_num of a pool from 2048 to 2148, my VMs is still ok. When i increase from 2148 to 2400, some VMs die (Qemu-kvm process die). My physical servers (host VMs) running kernel 3.13 and use librbd. I think it's a bug in librbd with cr

Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !

I always keep my pg number a power of 2. So I’d go from 2048 to 4096. I’m not sure if this is the safest way, but it’s worked for me. [yp] Michael Kuriger Sr. Unix Systems Engineer • mk7...@yp.com |• 818-649-7235 From: Chu Duc Minh mailto:chu.ducm...@gmail.com>> Date

Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !

@Michael Kuriger: when ceph/librbd operate normally, i know that double the pg_num is the safe way. But when it has problem, i think double it can make many many VMs die (maybe >= 50%?) On Mon, Mar 16, 2015 at 9:53 PM, Michael Kuriger wrote: > I always keep my pg number a power of 2. So I’d

Re: [ceph-users] Calamari - Data

Sumit, You may have better luck on the ceph-calamari mailing list. Anyway - calamari uses graphite to handle metrics, and graphite does indeed write them to files. John On 11/03/2015 05:09, Sumit Gaur wrote: Hi I have a basic architecture related question. I know Calamari collect system u

Re: [ceph-users] CephFS: delayed objects deletion ?

On 14/03/2015 09:22, Florent B wrote: Hi, What do you call "old MDS" ? I'm on Giant release, it is not very old... With CephFS we have a special definition of "old" that is anything that doesn't have the very latest bug fixes ;-) There have definitely been fixes to stray file handling[1] betw

Re: [ceph-users] CephFS: authorizations ?

On 13/03/2015 11:51, Florent B wrote: Hi all, My question is about user management in CephFS. Is it possible to restrict a CephX user to access some subdirectories ? Not yet. The syntax for setting a "path=" part in the authorization caps for a cephx user exists, but the code for enforcing it

Re: [ceph-users] CephFS: delayed objects deletion ?

On 03/16/2015 05:14 PM, John Spray wrote: > With CephFS we have a special definition of "old" that is anything > that doesn't have the very latest bug fixes ;-) > > There have definitely been fixes to stray file handling[1] between > giant and hammer. Since with giant you're using a version that i

Re: [ceph-users] CephFS: delayed objects deletion ?

On 16/03/2015 16:30, Florent B wrote: Thank you John :) Hammer is not released yet, is it ? Is it 'safe' to upgrade a production cluster to 0.93 ? I keep forgetting that -- yes, I should have added "...when it's released" :-) John ___ ceph-users mai

Re: [ceph-users] rados duplicate object name

This is expected behavior - "put" uses write_full which is an object overwrite command. On Thu, Mar 12, 2015 at 4:17 PM Kapil Sharma wrote: > Hi Cephers, > > Has anyone tested the behavior of rados by adding an object to the > cluster with an object name which already exists in the cluster ? > wi

[ceph-users] OS file Cache, Ceph RBD cache and Network files systems

Hi Cephers, Our university could deploy ceph. The goal is to store datas for research laboratories (non-HPC) . To do this, we plan to use Ceph with RBD (mount block device) from a NFS ( or CIFS ) server (ceph client) to workstations in laboratories. According to our tests, the OS (ubuntu or ce

Re: [ceph-users] CephFS: stripe_unit=65536 + object_size=1310720 => pipe.fault, server, going to standby

On 11/03/2015 08:59, Florent B wrote: Hi all, I'm testing CephFS with Giant and I have a problem when I set these attrs : setfattr -n ceph.dir.layout.stripe_unit -v "65536" pool_cephfs01/ setfattr -n ceph.dir.layout.stripe_count -v "1" pool_cephfs01/ setfattr -n ceph.dir.layout.object_size -v

Re: [ceph-users] Mapping users to different rgw pools

Yes, the placement target feature is logically separate from multi-zone setups. Placement targets are configured in the region though, which somewhat muddies the issue. Placement targets are useful feature for multi-zone, so different zones in a cluster don't share the same disks. Federation set

Re: [ceph-users] osd laggy algorithm

On Wed, Mar 11, 2015 at 8:40 AM, Artem Savinov wrote: > hello. > ceph transfers osd node in the down status by default , after receiving 3 > reports about disabled nodes. Reports are sent per "osd heartbeat grace" > seconds, but the settings of "mon_osd_adjust_heartbeat_gratse = true, > mon_osd_

Re: [ceph-users] Cache Tier Flush = immediate base tier journal sync?

On Wed, Mar 11, 2015 at 2:25 PM, Nick Fisk wrote: > > I’m not sure if it’s something I’m doing wrong or just experiencing an > oddity, but when my cache tier flushes dirty blocks out to the base tier, the > writes seem to hit the OSD’s straight away instead of coalescing in the > journals, is t

Re: [ceph-users] PGs stuck unclean "active+remapped" after an osd marked out

On Wed, Mar 11, 2015 at 3:49 PM, Francois Lafont wrote: > Hi, > > I was always in the same situation: I couldn't remove an OSD without > have some PGs definitely stuck to the "active+remapped" state. > > But I remembered I read on IRC that, before to mark out an OSD, it > could be sometimes a good

Re: [ceph-users] client-ceph [can not connect from client][connect protocol feature mismatch]

Thanks a lot Stephane and Kamil, Your reply was really helpful. I needed a different version of ceph client on my client machine. Initially my java application using librados was throwing connection time out. Then I considered querying ceph from command line (ceph --id ...), which was giving the e

[ceph-users] RadosGW Direct Upload Limitation

Hi all! I have recently updated to CEPH version 0.80.9 (latest Firefly release) which presumably supports direct upload. I 've tried to upload a file using this functionality and it seems that is working for files up to 5GB. For files above 5GB there is an error. I believe that this is becau

Re: [ceph-users] PGs stuck unclean "active+remapped" after an osd marked out

Hi, Gregory Farnum a wrote : > If I remember/guess correctly, if you mark an OSD out it won't > necessarily change the weight of the bucket above it (ie, the host), > whereas if you change the weight of the OSD then the host bucket's > weight changes. I can just say that, indeed, I have noticed

Re: [ceph-users] RadosGW Direct Upload Limitation

On Mon, Mar 16, 2015 at 11:14 AM, Georgios Dimitrakakis wrote: > Hi all! > > I have recently updated to CEPH version 0.80.9 (latest Firefly release) > which presumably > supports direct upload. > > I 've tried to upload a file using this functionality and it seems that is > working > for files up

Re: [ceph-users] PGs stuck unclean "active+remapped" after an osd marked out

> > > If I remember/guess correctly, if you mark an OSD out it won't > necessarily change the weight of the bucket above it (ie, the host), > whereas if you change the weight of the OSD then the host bucket's > weight changes. > -Greg That sounds right. Marking an OSD out is a ceph osd reweight

Re: [ceph-users] RadosGW Direct Upload Limitation

> > > Maybe, but I'm not sure if Yehuda would want to take it upstream or > not. This limit is present because it's part of the S3 spec. For > larger objects you should use multi-part upload, which can get much > bigger. > -Greg > > Note that the multi-part upload has a lower limit of 4MiB per part

Re: [ceph-users] query about mapping of Swift/S3 APIs to Ceph cluster APIs

On Sat, Mar 14, 2015 at 3:04 AM, pragya jain wrote: > Hello all! > > I am working on Ceph object storage architecture from last few months. > > I am unable to search a document which can describe how Ceph object > storage APIs (Swift/S3 APIs) are mappedd with Ceph storage cluster APIs > (librado

Re: [ceph-users] Shadow files

Out of curiousity, what's the frequency of the peaks and troughs? RadosGW has configs on how long it should wait after deleting before garbage collecting, how long between GC runs, and how many objects it can GC in per run. The defaults are 2 hours, 1 hour, and 32 respectively. Search http://doc

Re: [ceph-users] Shadow files

On Mon, Mar 16, 2015 at 12:12 PM, Craig Lewis wrote: > Out of curiousity, what's the frequency of the peaks and troughs? > > RadosGW has configs on how long it should wait after deleting before garbage > collecting, how long between GC runs, and how many objects it can GC in per > run. > > The def

Re: [ceph-users] Firefly, cephfs issues: different unix rights depending on the client and ls are slow

On Sun, Mar 15, 2015 at 7:06 PM, Yan, Zheng wrote: > On Sat, Mar 14, 2015 at 7:03 AM, Scottix wrote: >> ... >> >> >>> The time variation is caused cache coherence. when client has valid >>> information >>> in its cache, 'stat' operation will be fast. Otherwise the client need to >>> send >>> requ

Re: [ceph-users] More than 50% osds down, CPUs still busy; will the cluster recover without help?

On Sat, Mar 14, 2015 at 1:56 AM, Chris Murray wrote: > Good evening all, > > Just had another quick look at this with some further logging on and thought > I'd post the results in case anyone can keep me moving in the right direction. > > Long story short, some OSDs just don't appear to come up a

Re: [ceph-users] RadosGW Direct Upload Limitation

- Original Message - > From: "Craig Lewis" > To: "Gregory Farnum" > Cc: ceph-users@lists.ceph.com > Sent: Monday, March 16, 2015 11:48:15 AM > Subject: Re: [ceph-users] RadosGW Direct Upload Limitation > > > > > Maybe, but I'm not sure if Yehuda would want to take it upstream or > n

[ceph-users] CephFS unexplained writes

Hi, I am getting relatively bad performance from cephfs. I use a replicated cache pool on ssd in front of an erasure coded pool on rotating media. When reading big files (streaming video), I see a lot of disk i/o, especially writes. I have no clue what could cause these writes. The writes are goi

Re: [ceph-users] CephFS unexplained writes

Hi, I forgot to mention: while I am seeing these writes in iotop and /proc/diskstats for the hdd's, I am -not- seeing any writes in "rados df" for the pool residing on these disks. There is only one pool active on the hdd's and according to rados df it is getting zero writes when I'm just reading

Re: [ceph-users] CephFS unexplained writes

The information you're giving sounds a little contradictory, but my guess is that you're seeing the impacts of object promotion and flushing. You can sample the operations the OSDs are doing at any given time by running ops_in_progress (or similar, I forget exact phrasing) command on the OSD admin

Re: [ceph-users] Shadow files

Thats the thing. The peaks and troughs are in USERS BUCKETS only. The actual cluster usage does not go up and down, it just goes up up up. I would expect to see peaks and troughs much the same as the user buckets peaks and troughs on the overall cluster disk usage. But this is not the case. We

Re: [ceph-users] Cache Tier Flush = immediate base tier journal sync?

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Gregory Farnum > Sent: 16 March 2015 17:33 > To: Nick Fisk > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Cache Tier Flush = immediate base tier journal > sync? > > On Wed, Ma

Re: [ceph-users] Cache Tier Flush = immediate base tier journal sync?

Nothing here particularly surprises me. I don't remember all the details of the filestore's rate limiting off the top of my head, but it goes to great lengths to try and avoid letting the journal get too far ahead of the backing store. Disabling the filestore flusher and increasing the sync interva

Re: [ceph-users] Cache Tier Flush = immediate base tier journal sync?

On Mon, 16 Mar 2015 16:09:12 -0700 Gregory Farnum wrote: > Nothing here particularly surprises me. I don't remember all the > details of the filestore's rate limiting off the top of my head, but > it goes to great lengths to try and avoid letting the journal get too > far ahead of the backing stor

Re: [ceph-users] Cache Tier Flush = immediate base tier journal sync?

On Mon, Mar 16, 2015 at 4:46 PM, Christian Balzer wrote: > On Mon, 16 Mar 2015 16:09:12 -0700 Gregory Farnum wrote: > >> Nothing here particularly surprises me. I don't remember all the >> details of the filestore's rate limiting off the top of my head, but >> it goes to great lengths to try and a

[ceph-users] ceph.conf

Hi all I have seen that new versions of CEPH with new OS like RHEL7 and Cento7 doesn’t need information like mon.node1 and osd.0 etc.. anymore, can anybody tell me if is that for real? or do I need still need to write config like this: [osd.0] host = sagitario addr = 192.168.1.67 [mo

Re: [ceph-users] CephFS: delayed objects deletion ?

On Mon, Mar 16, 2015 at 5:08 PM, Florent B wrote: > Since then I deleted the pool. > > But I now have another problem, in fact the "opposite" of the previous : > now I never deleted files in clients, data objects and metadata are > still in pools, but directory is empty for clients (it is another

[ceph-users] SUBSCRIBE

SUBSCRIBE ceph-users ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com