Re: [ceph-users] 回复: half performace with keyvalue backend in 0.87

Haomai wang, Do you have proresss on this performance issue? 发件人： Haomai Wang 发送时间： 2014-10-31 10:05 收件人： 廖建锋 抄送： ceph-users; ceph-users 主题： Re: Re: [ceph-user

Re: [ceph-users] mds isn't working anymore after osd's running full

Hello Greg and John, Thanks for solving the bug. I will compile the patch and make new rpm packages and test it on the Ceph cluster. I will let you know what the results are. Kind regards, Jasper Van: Gregory Farnum [g...@gregs42.com] Verzonden: vrijdag

Re: [ceph-users] Cache Tier Statistics

Hi Jean-Charles, Thanks for your response, I have found the following using "ceph daemon osd.{id} perf dump". "tier_promote": 1425, "tier_flush": 0, "tier_flush_fail": 0, "tier_try_flush": 216, "tier_try_flush_fail": 21, "tier_evict": 1413, "tier_whiteou

Re: [ceph-users] 回复: half performace with keyvalue backend in 0.87

Yep, be patient. Need more time On Mon, Nov 10, 2014 at 9:33 AM, 廖建锋 wrote: > Haomai wang, >Do you have proresss on this performance issue? > > > > 发件人： Haomai Wang > 发送时间： 2014-10-31 10:05 > 收件人： 廖建锋 > 抄送： ceph-users; ceph-users > 主题： Re: Re: [ceph-users] 回复: half performace with keyvalu

Re: [ceph-users] Ceph on RHEL 7 using teuthology

Yes. I see similar package dependency when installing manually. ~Pras On Mon, Nov 10, 2014 at 3:00 PM, Loic Dachary wrote: > Hi, > > It looks like there are broken packages on the target machine even before > teuthology tries to install new packages. Do you see similar errors when > trying to i

Re: [ceph-users] emperor -> firefly 0.80.7 upgrade problem

Hi Craig and list, > > > If you create a real osd.20, you might want to leave it OUT until you > > > get things healthy again. I created a real osd.20 (and it turns out I needed an osd.21 also). ceph pg x.xx query no longer lists down osds for probing: "down_osds_we_would_probe": [], But I ca

Re: [ceph-users] Typical 10GbE latency

On 08-11-14 02:42, Gary M wrote: > Wido, > > Take the switch out of the path between nodes and remeasure.. ICMP-echo > requests are very low priority traffic for switches and network stacks. > I tried with a direct TwinAx and fiber cable. No difference. > If you really want to know, place a ne

Re: [ceph-users] mds isn't working anymore after osd's running full

Hello John and Greg, I used the new patch and now the undump succeeded and the mds is working fine and I can mount cephfs again! I still have one placement group which keeps deep scrubbing even after restarting the ceph cluster: dumped all in format plain 3.300 0 0 0

Re: [ceph-users] Installing CephFs via puppet

- Original Message - > From: "JIten Shah" > To: "Jean-Charles LOPEZ" > Cc: "ceph-users" > Sent: Friday, November 7, 2014 7:18:10 PM > Subject: Re: [ceph-users] Installing CephFs via puppet > > Thanks JC and Loic but we HAVE to use puppet. That’s how all of our > configuration and deplo

Re: [ceph-users] mds isn't working anymore after osd's running full

It's supposed to do that; deep scrubbing is an ongoing consistency-check mechanism. If you really want to disable it you can set an osdmap flag to prevent it, but you'll have to check the docs for exactly what that is as I can't recall. Glad things are working for you; sorry it took so long! -Greg

Re: [ceph-users] emperor -> firefly 0.80.7 upgrade problem

If all of your PGs now have an empty down_osds_we_would_probe, I'd run through this discussion again. The commands to tell Ceph to give up on lost data should have an effect now. That's my experience anyway. Nothing progressed until I took care of down_osds_we_would_probe. After that was empty,

Re: [ceph-users] Pg's stuck in inactive/unclean state + Association from PG-OSD does not seem to be happenning.

Folks, Now, we are running into an issue where the PG's(192) are stuck in creating state forever. I have experimented with various PG settings(osd_pool_default_pg_num from 50 to 400) for replicas and default and doesn't seem to help so far. Just to give you a brief overview, I have 8 osd's. I se

Re: [ceph-users] PG inconsistency

For #1, it depends what you mean by fast. I wouldn't worry about it taking 15 minutes. If you mark the old OSD out, ceph will start remapping data immediately, including a bunch of PGs on unrelated OSDs. Once you replace the disk, and put the same OSDID back in the same host, the CRUSH map will

Re: [ceph-users] OSD commits suicide

Have you tuned any of the recovery or backfill parameters? My ceph.conf has: [osd] osd max backfills = 1 osd recovery max active = 1 osd recovery op priority = 1 Still, if it's running for a few hours, then failing, it sounds like there might be something else at play. OSDs use a lot of RA

Re: [ceph-users] An OSD always crash few minutes after start

You're running 0.87-6. There were various fixes for this problem in Firefly. Were any of these snapshots created on early version of Firefly? So far, every fix for this issue has gotten developers involved. I'd see if you can talk to some devs on IRC, or post to the ceph-devel mailing list. M

Re: [ceph-users] How to remove hung object

Do you have any OSDs that are offline that you can bring back online? ceph pg query 6.9d8 should tell you. At the bottom, there is a section with down_osds_we_would_probe. Focus on getting those OSDs back up. On Sat, Nov 8, 2014 at 11:13 PM, Tuân Tạ Bá wrote: > > Hi all, > > I want to remov

Re: [ceph-users] Stuck in stale state

"nothing to send, going to standby" isn't necessarily bad, I see it from time to time. It shouldn't stay like that for long though. If it's been 5 minutes, and the cluster still isn't doing anything, I'd restart that osd. On Fri, Nov 7, 2014 at 1:55 PM, Jan Pekař wrote: > Hi, > > I was testing

Re: [ceph-users] emperor -> firefly 0.80.7 upgrade problem

Hi Craig, > If all of your PGs now have an empty down_osds_we_would_probe, I'd run > through this discussion again. Yep, looks to be true. So I ran: # ceph pg force_create_pg 2.5 and it has been creating for about 3 hours now. :/ # ceph health detail | grep creating pg 2.5 is stuck inactive

Re: [ceph-users] osd down

Craig, Thanks for the info. I ended up doing a zap and then a create via ceph-deploy. One question that I still have is surrounding adding the failed osd back into the pool. In this example...osd.70 was badwhen I added it back in via ceph-deploy...the disk was brought up as osd.108. Only

Re: [ceph-users] Pg's stuck in inactive/unclean state + Association from PG-OSD does not seem to be happenning.

It is simple. When you have this kind of problem (stuck), first look into crush map. And here you are: You have only one default ruleset 0 with "step take default" (so selecting osd's from default root subtree), but your root doesn't contain any osds. See below: rule replicated_ruleset {

Re: [ceph-users] Stuck in stale state

Thank you, sorry for bothering, I was new to ceph-users list and I couldn't cancel my message. I found out what happened few hours later. Main problem was, that I moved one OSD from "host hostname {}" crush map entry (I wanted to do so). Everything was OK, but restart of OSD caused automatic o

[ceph-users] Node down question

I have searched the list archives, and have seen a couple of references to this question, but no real solution, unfortunately... We are running multiple ceph clusters, pretty much as media appliances. As such, the number of nodes is variable, and all of the nodes are symmetric (i.e. same CPU powe

Re: [ceph-users] Node down question

On Mon, Nov 10, 2014 at 2:21 PM, Jason wrote: > I have searched the list archives, and have seen a couple of references > to this question, but no real solution, unfortunately... > > We are running multiple ceph clusters, pretty much as media appliances. > As such, the number of nodes is variable,

Re: [ceph-users] osd down

Yes, removing an OSD before re-creating it will give you the same OSD ID. That's my preferred method, because it keeps the crushmap the same. Only PGs that existed on the replaced disk need to be backfilled. I don't know if adding the replacement to the same host then removing the old OSD gives y

[ceph-users] Trying to figure out usable space on erasure coded pools

Hi, It's easy to calculate the amount of "raw" storage vs "actual" storage on replicated pools. Example with 4x 2TB disks: - 8TB "raw" - 4TB "usable" (when using 2 replicas) I understand how erasure coded pools reduces the overhead of storage required for data redundancy and resiliency and how

Re: [ceph-users] emperor -> firefly 0.80.7 upgrade problem

I had the same experience with force_create_pg too. I ran it, and the PGs sat there in creating state. I left the cluster overnight, and sometime in the middle of the night, they created. The actual transition from creating to active+clean happened during the recovery after a single OSD was kick

Re: [ceph-users] Trying to figure out usable space on erasure coded pools

On Mon, 10 Nov 2014, David Moreau Simard wrote: > Hi, > > It's easy to calculate the amount of "raw" storage vs "actual" storage on > replicated pools. > Example with 4x 2TB disks: > - 8TB "raw" > - 4TB "usable" (when using 2 replicas) > > I understand how erasure coded pools reduces the overhea

Re: [ceph-users] Trying to figure out usable space on erasure coded pools

Oh, that's interesting - I didn't know that. Thanks. -- David Moreau Simard > On Nov 10, 2014, at 6:06 PM, Sage Weil wrote: > > On Mon, 10 Nov 2014, David Moreau Simard wrote: >> Hi, >> >> It's easy to calculate the amount of "raw" storage vs "actual" storage on >> replicated pools. >> Examp

[ceph-users] PG's incomplete after OSD failure

Hi All, We've had a string of very unfortunate failures and need a hand fixing the incomplete PG's that we're now left with. We're configured with 3 replicas over different hosts with 5 in total. The timeline goes - -1 week :: A full server goes offline with a failed backplane. Still not working

Re: [ceph-users] osds fails to start with mismatch in id

Hi Greg, Thanks for the pointer. I think you are right. The full story is like this. After installation, everything works fine until I reboot. I do observe udevadm getting triggered in logs, but the devices do not come up after reboot. Exact issue as http://tracker.ceph.com/issues/5194. But t

Re: [ceph-users] PG's incomplete after OSD failure

Just an update, it appears that no data actually exists for those PG's on osd.117 and osd.111 but it's showing as incomplete anyway. So for the 8.ca PG, osd.111 has only an empty directory but osd 190 is filled with data. For 8.6ae, osd.117 has no data in the pg directory and osd.190 is filled wit

[ceph-users] does anyone know what xfsaild and kworker are?they make osd disk busy. produce 100-200iops per osd disk?

ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipie

Re: [ceph-users] PG's incomplete after OSD failure

On Tue, 11 Nov 2014, Matthew Anderson wrote: > Just an update, it appears that no data actually exists for those PG's > on osd.117 and osd.111 but it's showing as incomplete anyway. > > So for the 8.ca PG, osd.111 has only an empty directory but osd 190 is > filled with data. > For 8.6ae, osd.117

Re: [ceph-users] osds fails to start with mismatch in id

Hi, Ramakrishna. I think you understand what the problem is: [ceph@ceph05 ~]$ cat /var/lib/ceph/osd/ceph-56/whoami 56 [ceph@ceph05 ~]$ cat /var/lib/ceph/osd/ceph-57/whoami 57 Tue Nov 11 2014 at 6:01:40, Ramakrishna Nishtala (rnishtal) < rnish...@cisco.com>: > Hi Greg, > > Thanks for the pointer

Re: [ceph-users] osds fails to start with mismatch in id

Hi Ramakrishna, we use the phy. path (containing the serial number) to a disk to prevent complexity and wrong mapping... This path will never change: /etc/ceph/ceph.conf [osd.16] devs = /dev/disk/by-id/scsi-SATA_ST4000NM0033-9Z_Z1Z0SDCY-part1 osd_jo

Re: [ceph-users] Triggering shallow scrub on OSD where scrub is already in progress

On Sun, Nov 9, 2014 at 9:29 PM, Mallikarjun Biradar wrote: > Hi all, > > Triggering shallow scrub on OSD where scrub is already in progress, restarts > scrub from beginning on that OSD. > > > Steps: > Triggered shallow scrub on an OSD (Cluster is running heavy IO) > While scrub is in progress, tri

[ceph-users] Deep scrub, cache pools, replica 1

Hello, One of my clusters has become busy enough (I'm looking at you, evil Window VMs that I shall banish elsewhere soon) to experience client noticeable performance impacts during deep scrub. Before this I instructed all OSDs to deep scrub in parallel at Saturday night and that finished before