Re: [ceph-users] Storage

On 05/06/14 17:01, yalla.gnan.ku...@accenture.com wrote: Hi All, I have a ceph storage cluster with four nodes. I have created block storage using cinder in openstack and ceph as its storage backend. So, I see a volume is created in ceph in one of the pools. But how to get information like o

Re: [ceph-users] Run ceph from source code

I compile and run from the src build quite often. Here is my recipe: $ ./autogen.sh $ ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --with-radosgw $ time make $ sudo make install $ sudo cp src/init-ceph /etc/init.d/ceph $ sudo cp src/init-radosgw /etc/init.d/radosgw $ sudo ch

Re: [ceph-users] Poor performance on all SSD cluster

I can reproduce this in: ceph version 0.81-423-g1fb4574 on Ubuntu 14.04. I have a two osd cluster with data on two sata spinners (WD blacks) and journals on two ssd (Crucual m4's). I getting about 3.5 MB/s (kernel and librbd) using your dd command with direct on. Leaving off direct I'm seeing

Re: [ceph-users] Poor performance on all SSD cluster

On 22/06/14 14:09, Mark Kirkwood wrote: Upgrading the VM to 14.04 and restesting the case *without* direct I get: - 164 MB/s (librbd) - 115 MB/s (kernel 3.13) So managing to almost get native performance out of the librbd case. I tweaked both filestore max and min sync intervals (100 and 10

Re: [ceph-users] Poor performance on all SSD cluster

Good point, I had neglected to do that. So, amending my conf.conf [1]: [client] rbd cache = true rbd cache size = 2147483648 rbd cache max dirty = 1073741824 rbd cache max dirty age = 100 and also the VM's xml def to include cache to writeback:

Re: [ceph-users] Poor performance on all SSD cluster

On 23/06/14 18:51, Christian Balzer wrote: On Sunday, June 22, 2014, Mark Kirkwood rbd cache max dirty = 1073741824 rbd cache max dirty age = 100 Mark, you're giving it a 2GB cache. For a write test that's 1GB in size. "Aggressively set" is a bit of an understatement h

Re: [ceph-users] Poor performance on all SSD cluster

On 24/06/14 17:37, Alexandre DERUMIER wrote: Hi Greg, So the only way to improve performance would be to not use O_DIRECT (as this should bypass rbd cache as well, right?). yes, indeed O_DIRECT bypass cache. BTW, Do you need to use mysql with O_DIRECT ? default innodb_flush_method is fda

Re: [ceph-users] Poor performance on all SSD cluster

On 24/06/14 18:15, Robert van Leeuwen wrote: All of which means that Mysql performance (looking at you binlog) may still suffer due to lots of small block size sync writes. Which begs the question: Anyone running a reasonable busy Mysql server on Ceph backed storage? We tried and it did not pe

Re: [ceph-users] Poor performance on all SSD cluster

On 23/06/14 19:16, Mark Kirkwood wrote: For database types (and yes I'm one of those)...you want to know that your writes (particularly your commit writes) are actually making it to persistent storage (that ACID thing you know). Now I see RBD cache very like battery backed RAID cards -

Re: [ceph-users] Poor performance on all SSD cluster

On 24/06/14 23:39, Mark Nelson wrote: On 06/24/2014 03:45 AM, Mark Kirkwood wrote: On 24/06/14 18:15, Robert van Leeuwen wrote: All of which means that Mysql performance (looking at you binlog) may still suffer due to lots of small block size sync writes. Which begs the question: Anyone

Re: [ceph-users] Poor performance on all SSD cluster

On 26/06/14 03:15, Josef Johansson wrote: Hi, On 25/06/14 00:27, Mark Kirkwood wrote: Yes - same kind of findings, specifically: - random read and write (e.g index access) faster than local disk - sequential write (e.g batch inserts) similar or faster than local disk - sequential read (e.g

Re: [ceph-users] write performance per disk

On 04/07/14 02:32, VELARTIS Philipp Dürhammer wrote: Ceph.conf: rbd cache = true rbd cache size = 2147483648 rbd cache max dirty = 1073741824 Just a FYI - I posted a setting very like this in another thread, and remarked that it was "aggressive" - probably too much

Re: [ceph-users] [URGENT]. Can't connect to CEPH after upgrade from 0.72 to 0.80

On 13/07/14 17:07, Andrija Panic wrote: Hi, Sorry to bother, but I have urgent situation: upgraded CEPH from 0.72 to 0.80 (centos 6.5), and now all my CloudStack HOSTS can not connect. I did basic "yum update ceph" on the first MON leader, and all CEPH services on that HOST, have been restarted

Re: [ceph-users] [URGENT]. Can't connect to CEPH after upgrade from 0.72 to 0.80

On 13/07/14 18:38, Andrija Panic wrote: Hi Mark, actually, CEPH is running fine, and I have deployed NEW host (new compile libvirt with ceph 0.8 devel, and newer kernel) - and it works... so migrating some VMs to this new host... I have 3 physical hosts, that are both MON and 2x OSD per host, al

Re: [ceph-users] [URGENT]. Can't connect to CEPH after upgrade from 0.72 to 0.80

On 13/07/14 19:15, Mark Kirkwood wrote: On 13/07/14 18:38, Andrija Panic wrote: Any suggestion on need to recompile libvirt ? I got info from Wido, that libvirt does NOT need to be recompiled Thinking about this a bit more - Wido *may* have meant: - *libvirt* does not need to be

Re: [ceph-users] librbd tuning?

On 05/08/14 03:52, Tregaron Bayly wrote: Does anyone have any insight on how we can tune librbd to perform closer to the level of the rbd kernel module? In our lab we have a four node cluster with 1GbE public network and 10GbE cluster network. A client node connects to the public network with 1

Re: [ceph-users] librbd tuning?

On 05/08/14 23:44, Mark Nelson wrote: On 08/05/2014 02:48 AM, Mark Kirkwood wrote: On 05/08/14 03:52, Tregaron Bayly wrote: Does anyone have any insight on how we can tune librbd to perform closer to the level of the rbd kernel module? In our lab we have a four node cluster with 1GbE public

Re: [ceph-users] Using Crucial MX100 for journals or cache pool

It claims to have power loss protection, and reviews appear to back this up (http://www.anandtech.com/show/8066/crucial-mx100-256gb-512gb-review). I can't see a capacitor on the board... so I'm not sure of the mechanism Micron are using on these guys. The thing that requires attention would b

Re: [ceph-users] Using Crucial MX100 for journals or cache pool

A better picture here (http://img1.lesnumeriques.com/test/90/9096/crucial_mx100_512gb_pcb_hq.jpg). A row of small caps clearly visible on right of the left hand image... On 06/08/14 12:40, Mark Kirkwood wrote: It claims to have power loss protection, I can't see a capacitor on the board.

[ceph-users] Fresh deploy of ceph 0.83 has OSD down

Hi, I'm doing a fresh install of ceph 0.83 (src build) to an Ubuntu 14.04 VM using ceph-deploy 1.59. Everything goes well until the osd creation, which fails to start with a journal open error. The steps are shown below (ceph is the deploy target host): (ceph1) $ uname -a Linux ceph1 3.13.0

Re: [ceph-users] ceph-deploy activate actually didn't activate the OSD

On 08/08/14 07:07, German Anders wrote: Hi to all, I'm having some issues while trying to deploy a OSD: > ceph@cephmon01:~$ *sudo ceph osd tree* > # idweighttype nameup/downreweight > -12.73root default > -22.73host cephosd01 > 02.73o

Re: [ceph-users] Fresh deploy of ceph 0.83 has OSD down

On 07/08/14 11:06, Mark Kirkwood wrote: Hi, I'm doing a fresh install of ceph 0.83 (src build) to an Ubuntu 14.04 VM using ceph-deploy 1.59. Everything goes well until the osd creation, which fails to start with a journal open error. The steps are shown below (ceph is the deploy target

Re: [ceph-users] Fresh deploy of ceph 0.83 has OSD down

On 11/08/14 20:52, Mark Kirkwood wrote: On 07/08/14 11:06, Mark Kirkwood wrote: Hi, I'm doing a fresh install of ceph 0.83 (src build) to an Ubuntu 14.04 VM using ceph-deploy 1.59. Everything goes well until the osd creation, which fails to start with a journal open error. The steps are

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

On 29/08/14 04:11, Sebastien Han wrote: Hey all, See my fio template: [global] #logging #write_iops_log=write_iops_log #write_bw_log=write_bw_log #write_lat_log=write_lat_lo time_based runtime=60 ioengine=rbd clientname=admin pool=test rbdname=fio invalidate=0# mandatory #rw=randwrite r

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

On 29/08/14 14:06, Mark Kirkwood wrote: ... mounting (xfs) with nobarrier seems to get much better results. The run below is for a single osd on an xfs partition from an Intel 520. I'm using another 520 as a journal: ...and adding filestore_queue_max_ops = 2 improved IOPS a bit

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

On 29/08/14 22:17, Sebastien Han wrote: @Mark thanks trying this :) Unfortunately using nobarrier and another dedicated SSD for the journal (plus your ceph setting) didn’t bring much, now I can reach 3,5K IOPS. By any chance, would it be possible for you to test with a single OSD SSD? Funny

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

On 31/08/14 17:55, Mark Kirkwood wrote: On 29/08/14 22:17, Sebastien Han wrote: @Mark thanks trying this :) Unfortunately using nobarrier and another dedicated SSD for the journal (plus your ceph setting) didn’t bring much, now I can reach 3,5K IOPS. By any chance, would it be possible for

Re: [ceph-users] About IOPS num

Yes, as Jason suggests - 27 IOPS doing 4k blocks is: 27*4/1024 MB/s = 0.1 MB/s While the RBD volume is composed of 4MB objects - many of the (presumably) random IOs of 4k blocks can reside in the same 4MB object, so it is tricky to estimate how many 4MB objects are needing to be rewritten eac

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

On 01/09/14 12:36, Mark Kirkwood wrote: Allegedly this model ssd (128G m550) can do 75K 4k random write IOPS (running fio on the filesystem I've seen 70K IOPS so is reasonably believable). So anyway we are not getting anywhere near the max IOPS from our devices. We use the Intel S370

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

On 01/09/14 17:10, Alexandre DERUMIER wrote: Allegedly this model ssd (128G m550) can do 75K 4k random write IOPS (running fio on the filesystem I've seen 70K IOPS so is reasonably believable). So anyway we are not getting anywhere near the max IOPS >from our devices. Hi, Just check this: htt

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

On 02/09/14 19:38, Alexandre DERUMIER wrote: Hi Sebastien, I got 6340 IOPS on a single OSD SSD. (journal and data on the same partition). Shouldn't it better to have 2 partitions, 1 for journal and 1 for datas ? (I'm thinking about filesystem write syncs) Oddly enough, it does not seem to

Re: [ceph-users] SSD journal deployment experiences

On 05/09/14 10:05, Dan van der Ster wrote: That's good to know. I would plan similarly for the wear out. But I want to also prepare for catastrophic failures -- in the past we've had SSDs just disappear like a device unplug. Those were older OCZ's though... Yes - the Intel dc style drives s

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

On 17/09/14 08:39, Alexandre DERUMIER wrote: Hi, I’m just surprised that you’re only getting 5299 with 0.85 since I’ve been able to get 6,4K, well I was using the 200GB model Your model is DC S3700 mine is DC s3500 with lower writes, so that could explain the difference. Interesting - I

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

On 19/09/14 15:11, Aegeaner wrote: I noticed ceph added key/value store OSD backend feature in firefly, but i can hardly get any documentation about how to use it. At last I found that i can add a line in ceph.conf: osd objectstore = keyvaluestore-dev but got failed with ceph-deploy creating O

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

On 19/09/14 18:02, Mark Kirkwood wrote: On 19/09/14 15:11, Aegeaner wrote: I noticed ceph added key/value store OSD backend feature in firefly, but i can hardly get any documentation about how to use it. At last I found that i can add a line in ceph.conf: osd objectstore = keyvaluestore-dev

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

On 23/09/14 18:22, Aegeaner wrote: Now I use the following script to create key/value backended OSD, but the OSD is created down and never go up. ceph osd create umount /var/lib/ceph/osd/ceph-0 rm -rf /var/lib/ceph/osd/ceph-0 mkdir /var/lib/ceph/osd/ceph-0 ceph osd crush add

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

On 23/09/14 18:22, Aegeaner wrote: Now I use the following script to create key/value backended OSD, but the OSD is created down and never go up. ceph osd create umount /var/lib/ceph/osd/ceph-0 rm -rf /var/lib/ceph/osd/ceph-0 mkdir /var/lib/ceph/osd/ceph-0 ceph osd crush add

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

On 24/09/14 14:07, Aegeaner wrote: I turned on the debug option, and this is what I got: # ./kv.sh removed osd.0 removed item id 0 name 'osd.0' from crush map 0 umount: /var/lib/ceph/osd/ceph-0: not found updated add item id 0 name 'osd.0' weight 1 at location {host=

Re: [ceph-users] Repetitive builds for Ceph

On 03/02/15 01:28, Loic Dachary wrote: On 02/02/2015 13:27, Ritesh Raj Sarraf wrote: By the way, I'm trying to build Ceph from master, on Ubuntu Trusty. I hope that is supported ? Yes, that's also what I have. Same here - in the advent you need to rebuild the whole thing, using parallel

Re: [ceph-users] RGW region metadata sync prevents writes to non-master region

On 30/01/15 13:39, Mark Kirkwood wrote: On 30/01/15 12:34, Yehuda Sadeh wrote: On Thu, Jan 29, 2015 at 3:27 PM, Mark Kirkwood wrote: On 30/01/15 11:08, Yehuda Sadeh wrote: How does your regionmap look like? Is it updated correctly on all zones? Regionmap listed below - checking it on all

Re: [ceph-users] Ceph vs Hardware RAID: No battery backed cache

On 10/02/15 20:40, Thomas Güttler wrote: Hi, does the lack of a battery backed cache in Ceph introduce any disadvantages? We use PostgreSQL and our servers have UPS. But I want to survive a power outage, although it is unlikely. But "hope is not an option ..." You can certainly make use of

Re: [ceph-users] mongodb on top of rbd volumes (through krbd) ?

On 12/02/15 23:18, Alexandre DERUMIER wrote: What is the behavior of mongo when a shard is unavailable for some reason (crash or network partition) ? If shard3 is on the wrong side of a network partition and uses RBD, it will hang. Is it something that mongo will >>gracefully handle ? If one

Re: [ceph-users] Building Ceph

I think you want to do: $ dch $ dpkg-buildpackage You can muck about with what the package is gonna be called (versions, revisions etc) from dch, without changing the src. Cheers Mark On 03/04/15 10:17, Garg, Pankaj wrote: Hi, I am building Ceph Debian Packages off of the 0.80.9 (latest f

Re: [ceph-users] replace dead SSD journal

Yes, it sure is - my experience with 'consumer' SSD is that they die with obscure firmware bugs (wrong capacity, zero capacity, not detected in bios anymore) rather than flash wearout. It seems that the 'enterprise' tagged drives are less inclined to suffer this fate. Regards Mark On 18/04/1

Re: [ceph-users] ceph-deploy : systemd unit files not deployed to a centos7 nodes

I have just run into this after upgrading to Ubuntu 15.04 and trying to deploy ceph 0.94. Initially tried to get things going by changing relevant code for ceph-deploy and ceph-disk to use systemd for this release - however the unit files in ./systemd do not contain a ceph-create-keys step, so

Re: [ceph-users] Help with CEPH deployment

On 04/05/15 05:42, Venkateswara Rao Jujjuri wrote: Here is the output..I am still stuck at this step. :( (multiple times tried to by purging and restarting from scratch) vjujjuri@rgulistan-wsl10:~/ceph-cluster$ ceph-deploy mon create-initial [ceph_deploy.conf][DEBUG ] found configuration file at

Re: [ceph-users] Help with CEPH deployment

On 05/05/15 04:16, Venkateswara Rao Jujjuri wrote: Thanks Mark. I switched to completely different machine and started from scratch, things were much smoother this time. Cluster was up in 30 mins. I guess purgedata , droplets and and purge is Not enough to bring the machine back clean? What I was

Re: [ceph-users] Rados Gateway and keystone

On 07/05/15 20:21, ghislain.cheval...@orange.com wrote: HI all, After adding the nss and the keystone admin url parameters in ceph.conf and creating the openSSL certificates, all is working well. If I had followed the doc and processed by copy/paste, I wouldn't have encountered any problems.

Re: [ceph-users] OSD trashed by simple reboot (Debian Jessie, systemd?)

eyeball I think I might be seeing this: --- osd: fix journal direct-io shutdown (#9073 Mark Kirkwood, Ma Jianpeng, Somnath Roy) --- The details in the various related bug reports certainly make it look related. Funny that nobody involved in those bug reports noticed the similarity. Now I wouldn&#

Re: [ceph-users] OSD trashed by simple reboot (Debian Jessie, systemd?)

be best if this was on a test system)! Cheers Mark On 05/06/15 15:28, Christian Balzer wrote: Hello Mark, On Thu, 04 Jun 2015 20:34:55 +1200 Mark Kirkwood wrote: Sorry Christian, I did briefly wonder, then thought, oh yeah, that fix is already merged in...However - on reflection, perhaps *n

Re: [ceph-users] OSD trashed by simple reboot (Debian Jessie, systemd?)

15 16:49, Christian Balzer wrote: Hello, On Fri, 05 Jun 2015 16:33:46 +1200 Mark Kirkwood wrote: Well, whatever it is, I appear to not be the only one after all: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=773361 Looking quickly at the relevant code: FileJournal::stop_writer() in s

Re: [ceph-users] OSD trashed by simple reboot (Debian Jessie, systemd?)

orrow)! Cheers Mark On 06/06/15 18:04, Mark Kirkwood wrote: Righty - I'll see if I can replicate what you see if I setup an 0.80.9 cluster using the same workstation hardware (WD Raptors and Intel 520s) that showed up the issue previously at 0.83 (I wonder if I never tried a fresh install usin

Re: [ceph-users] OSD trashed by simple reboot (Debian Jessie, systemd?)

> On Mon, 08 Jun 2015 20:05:20 +1200 Mark Kirkwood wrote: > >> Trying out some tests on my pet VMs with 0.80.9 does not elicit any >> journal failures...However ISTR that running on the bare metal was the >> most reliable way to reproduce...(proceeding - currently can

Re: [ceph-users] Incomplete MON removal

On 09/07/15 00:03, Steve Thompson wrote: Ceph newbie here; ceph 0.94.2, CentOS 6.6 x86_64. Kernel 2.6.32. Initial test cluster of five OSD nodes, 3 MON, 1 MDS. Working well. I was testing the removal of two MONs, just to see how it works. The second MON was stopped and removed: no problems. The

Re: [ceph-users] ceph-deploy progress and CDS session

One thing that comes to mind is the ability to create (or activate) osd's with a custom crush specification from (say) a supplied file. Regards Mark On 03/08/13 06:02, Sage Weil wrote: There is a session at CDS scheduled to discuss ceph-deploy (4:40pm PDT on Monday). We'll be going over what

Re: [ceph-users] Usage pattern and design of Ceph

On 19/08/13 18:17, Guang Yang wrote: 3. Some industry research shows that one issue of file system is the metadata-to-data ratio, in terms of both access and storage, and some technic uses the mechanism to combine small files to large physical files to reduce the ratio (Haystack for example),

Re: [ceph-users] Usage pattern and design of Ceph

On 20/08/13 13:27, Guang Yang wrote: Thanks Mark. What is the design considerations to break large files into 4M chunk rather than storing the large file directly? Quoting Wolfgang from previous reply: => which is a good thing in terms of replication and OSD usage distribution ...which co

Re: [ceph-users] To put journals to SSD or not?

On 02/09/13 07:19, Fuchs, Andreas (SwissTXT) wrote: Reading through the documentation and talking to several peaople leads to the conclusion that it's a best practice to place the journal of an OSD instance to a separate SSD disk to speed writing up. But is this true? i have 3 new dell servers

[ceph-users] Radosgw S3 - can't authenticate user

I have a test setup for Radosgw on a single box. The Swift side of things works fine, but trying S3 (via boto) I am encountering the error: error reading user info, uid=('X5E5BXJHCZGGII3HAWBB',) can't authenticate Now the access key above is correct (see below), and I have copied the secret ke

Re: [ceph-users] Radosgw S3 - can't authenticate user

On 03/09/13 15:25, Yehuda Sadeh wrote: Boto prog: #!/usr/bin/python import boto import boto.s3.connection access_key = 'X5E5BXJHCZGGII3HAWBB', secret_key = '' # redacted conn = boto.connect_s3( aws_access_key_id = access_key, aws_secret_access_key = se

Re: [ceph-users] New to ceph, auth/permission error

On 06/09/13 11:07, Gary Mazzaferro wrote: Hi Installed the latest ceph and having an issue with permission and don't know where to start looking. My Config: (2) ods data nodes (1) monitor node (1) mds node (1) admin node (1) deploy node (1) client node (not configured) All on vmware I collect

Re: [ceph-users] ceph.conf changes and restarting ceph.

I think this is not quite right now: "Upstart does not require you to define daemon instances in the Ceph configuration file (*although, they are still required for sysvinit should you choose to use it*)." I find that simply doing: $ mv upstart sysvinit in the various mon/osd/mds etc dirs r

Re: [ceph-users] About the data movement in Ceph

You might find it easier to use the python implementation for this (I certainly did). See attached (I was only interested in number of bytes, but the other metrics are available too)! Cheers Mark On 28/09/13 14:48, Zh Chen wrote: And recently i have another questions as follows, 5. I fi

Re: [ceph-users] ...-all-starter documentation available?

Upstart itself could do with better docs :-( I'd recommend starting with 'man initctl', should help clarify things a bit! Cheers Mark On 10/10/13 17:50, John Wilkins wrote: Ceph deployed by ceph-deploy on Ubuntu uses upstart. On Wed, Oct 9, 2013 at 1:48 PM, Snider, Tim wrote: I was poking

Re: [ceph-users] Is there a way to query RBD usage

On 16/10/13 15:53, Wido den Hollander wrote: On 10/16/2013 03:15 AM, Blair Bethwaite wrote: I.e., can we see what the actual allocated/touched size of an RBD is in relation to its provisioned size? No, not an easy way. The only way would be to probe which RADOS objects exist, but that's a he

[ceph-users] Very unbalanced osd data placement with differing sized devices

I stumbled across this today: 4 osds on 4 hosts (names ceph1 -> ceph4). They are KVM guests (this is a play setup). - ceph1 and ceph2 each have a 5G volume for osd data (+ 2G vol for journal) - ceph3 and ceph4 each have a 10G volume for osd data (+ 2G vol for journal) I do a standard installa

Re: [ceph-users] Very unbalanced osd data placement with differing sized devices

Yeah - I'm *guessing* that my use of a 5G volume is essentially 'underflowing' the auto weight assignment. On 17/10/13 16:28, David Zafman wrote: I may be wrong, but I always thought that a weight of 0 means don't put anything there. All weights > 0 will be looked at proportionally. See http

Re: [ceph-users] ceph-deploy zap disk failure

I'd guess that your sudo config has a very limited path list. On the target hosts check the 'secure_path' entry in /etc/sudoers. E.g mine is (Ubuntu 13.10): $ sudo grep secure_path /etc/sudoers Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" On 19/10/13 03:19

Re: [ceph-users] SSD question

On 22/10/13 15:05, Martin Catudal wrote: Hi, I have purchase my hardware for my Ceph storage cluster but did not open any of my 960GB SSD drive box since I need to answer my question first. Here's my hardware. THREE server Dual 6 core Xeon 2U capable with 8 hotswap tray plus 2 SSD mount i

Re: [ceph-users] Radosgw and large files

I was looking at the same thing myself, and Boto seems to work ok (tested a 6G file - some sample code attached). Regards Mark On 27/10/13 11:46, Derek Yarnell wrote: Hi Shain, Yes we have tested and have working S3 Multipart support for files >5GB (RHEL64/0.67.4). However, crossftp unless

[ceph-users] Radosgw partial gc

I have a radosgw instance (ceph 0.71-299-g5cba838 src build), running on Ubuntu 13.10. I've been experimenting with multipart uploads (which are working fine). However while *most* objects (from radosgw perspective) have their storage space gc'd after a while post deletion, I'm seeing what look

Re: [ceph-users] Radosgw partial gc

On 29/10/13 17:46, Yehuda Sadeh wrote: The multipart abort operation is supposed to remove the objects (no gc needed for these). Were there any other issues during the run, e.g., restarted gateways, failed requests, etc.? Note that the objects here are from two different buckets (4902.1, 5001.2

Re: [ceph-users] radosgw-agent error

On 29/10/13 20:53, lixuehui wrote: Hi,list From the document that a radosgw-agent's right info should like this INFO:radosgw_agent.sync:Starting incremental sync INFO:radosgw_agent.worker:17910 is processing shard number 0 INFO:radosgw_agent.worker:shard 0 has 0 entries afte

Re: [ceph-users] radosgw-agent error

On 31/10/13 06:31, Josh Durgin wrote: Note that the wip in the url means it's a work-in-progress branch, so it's not totally ready yet either. If anything is confusing or missing, let us know. It's great people are interested in trying this early. It's very helpful to find issues sooner (like

Re: [ceph-users] Radosgw and large files

ck to cancel the upload if the program needs to abort - but it is still possible to get failed uploads for other reasons, so it probably still useful to have something to find any! Cheers Mark On 28/10/13 18:04, Mark Kirkwood wrote: I was looking at the same thing myself, and Boto seems to wo

Re: [ceph-users] Radosgw partial gc

On 29/10/13 18:08, Mark Kirkwood wrote: On 29/10/13 17:46, Yehuda Sadeh wrote: The multipart abort operation is supposed to remove the objects (no gc needed for these). Were there any other issues during the run, e.g., restarted gateways, failed requests, etc.? Note that the objects here are

Re: [ceph-users] Radosgw and large files

rsday, October 31, 2013 1:27 PM To: Mark Kirkwood; de...@umiacs.umd.edu; ceph-us...@ceph.com Subject: Re: [ceph-users] Radosgw and large files Mark, Thanks for the update. Just an FYI I ran into an issue using the script when it turned out that the last part of the file was exactly 0 bytes. in l

Re: [ceph-users] Radosgw and large files

your enjoyment). Cheers Mark On 01/11/13 09:51, Mark Kirkwood wrote: Blast -I must have some shoddy arithmetic around the bit where I work out the final pirce size. I'll experient... Cheers Mark On 01/11/13 06:35, Shain Miley wrote: PS...I tested the cancel script it worked like a

Re: [ceph-users] Very frustrated with Ceph!

On 05/11/13 06:37, Alfredo Deza wrote: On Mon, Nov 4, 2013 at 12:25 PM, Gruher, Joseph R wrote: Could these problems be caused by running a purgedata but not a purge? It could be, I am not clear on what the expectation was for just doing purgedata without a purge. Purgedata removes /etc/cep

Re: [ceph-users] Very frustrated with Ceph!

l mailto:s...@newdream.net>> wrote: Purgedata is only meant to be run *after* the package is uninstalled. We should make it do a check to enforce that. Otherwise we run into these problems... Mark Kirkwood mailto:mark.kirkw...@catalyst.net.nz>> wrote: On 05/11/13 06:

Re: [ceph-users] Very frustrated with Ceph!

... forgot to add: maybe 'uninstall' should be target for ceph-deploy that removes just the actual software daemons... On 06/11/13 14:16, Mark Kirkwood wrote: I think purge of several data containing packages will ask if you want to destroy that too (Mysql comes to mind - asks if y

Re: [ceph-users] Very frustrated with Ceph!

ake a bad choice. I'd much rather that users be annoyed with me that they have to go manually clean up old data vs users who can't get their data back without herculean efforts. Mark On 11/05/2013 07:19 PM, Mark Kirkwood wrote: ... forgot to add: maybe 'uninstall' should

Re: [ceph-users] USB pendrive as boot disk

On 07/11/13 13:54, Craig Lewis wrote: On 11/6/13 15:41 , Gandalf Corvotempesta wrote: With the suggested adapter why not using a standard 2.5'' sata disk? Sata for OS should be enough, no need for an ssd At the time, the smallest SSDs were about half the price of the smallest HDDs. My Ceph

Re: [ceph-users] USB pendrive as boot disk

On 07/11/13 20:22, ja...@peacon.co.uk wrote: On 2013-11-07 01:02, Mark Kirkwood wrote: The SSD failures I've seen have all been firmware bugs rather than flash wearout. This has the effect that a RAID1 pair are likley to fail at the same time! Very interesting... and good reason to us

Re: [ceph-users] ceph (deploy?) and drive paths / mounting / best practice.

On 19/11/13 18:56, Robert van Leeuwen wrote: Hi, Since the /dev/sdX device location could shuffle things up (and that would mess things up) I'd like to use a more-persistent device path. Since I'd like to be able to replace a disk without adjusting anything (e.g. just formatting the disk) the

Re: [ceph-users] How to replace a failed OSD

On 20/11/13 22:27, Robert van Leeuwen wrote: Hi, What is the easiest way to replace a failed disk / OSD. It looks like the documentation here is not really compatible with ceph_deploy: http://ceph.com/docs/master/rados/operations/add-or-rm-osds/ It is talking about adding stuff to the ceph.conf

Re: [ceph-users] radosgw-agent AccessDenied 403

On 13/11/13 21:16, lixuehui wrote: Hi ,list We've ever reflected that ,radosgw-agent sync data failed all the time ,before. We paste the concert log here to seek any help now . application/json; charset=UTF-8 Wed, 13 Nov 2013 07:24:45 GMT x-amz-copy-source:sss%2Frgwconf /sss/rgwconf 2013-11-13

Re: [ceph-users] PG state diagram

That's rather cool (very easy to change). However given that the current generated size is kinda a big thumbnail and too small to be actually read meaningfully, would it not make sense to generate a larger resolution version by default and make the current one a link to it? Cheers Mark On 26

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

On 24/09/14 14:29, Aegeaner wrote: I run ceph on Red Hat Enterprise Linux Server 6.4 Santiago, and when I run "service ceph start" i got: # service ceph start ERROR:ceph-disk:Failed to activate ceph-disk: Does not look like a Ceph OSD, or incompatible version: /var/lib/ceph/tmp/mnt.

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

On 24/09/14 16:21, Aegeaner wrote: I have got my ceph OSDs running with keyvalue store now! Thank Mark! I have been confused for a whole week. Pleased to hear it! Now you can actually start plying with key value store backend. There are quite a few parameters, not fully documented yet - se

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

On 25/09/14 01:03, Sage Weil wrote: On Wed, 24 Sep 2014, Mark Kirkwood wrote: On 24/09/14 14:29, Aegeaner wrote: I run ceph on Red Hat Enterprise Linux Server 6.4 Santiago, and when I run "service ceph start" i got: # service ceph start ERROR:ceph-disk:Failed to activate

Re: [ceph-users] How to avoid deep-scrubbing performance hit?

We are also becoming interested in understanding and taming the impact of deep scrubbing. We may start running something similar to the cron tasks mentioned. Looking at these fine examples of bash + awk I wondered if I could do the job using the python rados api. I have attached my initial (un

Re: [ceph-users] Openstack keystone with Radosgw

On 08/10/14 11:02, lakshmi k s wrote: I am trying to integrate OpenStack Keystone with Ceph Object Store using the link - http://ceph.com/docs/master/radosgw/keystone. Swift V1.0 (without keystone) works quite fine. But for some reason, Swift v2.0 ke

[ceph-users] Rados Gateway and Swift create containers/buckets that cannot be opened

I have a recent ceph (0.85-1109-g73d7be0) configured to use keystone for authentication: $ cat ceph.conf ... [client.radosgw.gateway] host = ceph4 keyring = /etc/ceph/ceph.rados.gateway.keyring rgw_socket_path = /var/run/ceph/$name.sock log_file = /var/log/ceph/radosgw.log rgw_data = /var/lib/ce

Re: [ceph-users] Rados Gateway and Swift create containers/buckets that cannot be opened

On 08/10/14 18:46, Mark Kirkwood wrote: I have a recent ceph (0.85-1109-g73d7be0) configured to use keystone for authentication: $ cat ceph.conf ... [client.radosgw.gateway] host = ceph4 keyring = /etc/ceph/ceph.rados.gateway.keyring rgw_socket_path = /var/run/ceph/$name.sock log_file = /var

Re: [ceph-users] Rados Gateway and Swift create containers/buckets that cannot be opened

Yes. I ran into that as well - I used WSGIChunkedRequest On in the virtualhost config for the *keystone* server [1] as indicated in issue 7796. Cheers Mark [1] i.e, not the rgw. On 08/10/14 22:58, Ashish Chandra wrote: Hi Mark, Good you got the solution. But since you have already done au

Re: [ceph-users] Openstack keystone with Radosgw

If you are using ceph + radosgw packages they should be built with the nss option (--with-nss), so nothing to do there. For the server running keystone you need to do: (root) $ mkdir /var/ceph/nss (root) $ openssl x509 -in /etc/keystone/ssl/certs/ca.pem -pubkey | \ certutil -d /var/ceph

Re: [ceph-users] Openstack keystone with Radosgw

Ok, so that is the thing to get sorted. I'd suggest posting the error(s) you are getting perhaps here (someone else might know), but definitely to one of the Debian specific lists. In the meantime perhaps try installing the packages with aptitude rather than apt-get - if there is some fancy fo

Re: [ceph-users] Openstack keystone with Radosgw

ls or certutil tool on debian/ubuntu? If so, how did you go about this problem. On Wednesday, October 8, 2014 7:01 PM, Mark Kirkwood wrote: Ok, so that is the thing to get sorted. I'd suggest posting the error(s) you are getting perhaps here (someone else might know), but definitely to one of

Re: [ceph-users] Openstack keystone with Radosgw

:~$ openssl x509 -in /home/gateway/ca.pem -pubkey | certutil -d /var/lib/ceph/nss -A -n ca -t "TCu,Cu,Tuw" certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. On Wednesday, October 8, 2014 7:55 PM, Mark Kirkwood wrote: As a

  1   2   3   >