Hi,
i made a 'stop ceph-all' on my ceph-admin-host and then a kernel-upgrade
from 3.9 to 3.11 on all of my 3 nodes.
Ubuntu 13.04, ceph 0,68
The kernel-upgrade required a reboot.
Now after rebooting i get the following errors:
/root@bd-a:~# ceph -s//
//cluster e0dbf70d-af59-42a5-b834-7ad739a
Hi,
I have a space problem on a production cluster, like if there is unused
data not freed : "ceph df" and "rados df" reports 613GB of data, and
disk usage is 2640GB (with 3 replica). It should be near 1839GB.
I have 5 hosts, 3 with SAS storage and 2 with SSD storage. I use crush
rules to put po
OSD created only if I use single disk for data and journal.
Situation with separate disks:
1.
ceph-deploy disk zap ceph001:sdaa ceph001:sda1 [ceph_deploy.osd][DEBUG ]
zapping /dev/sdaa on ceph001 [ceph_deploy.osd][DEBUG ] zapping /dev/sda1 on
ceph001
2.
Wiped file system on ceph001
wipefs /dev/s
On Tue, Sep 10, 2013 at 3:03 AM, Josh Durgin wrote:
> On 09/09/2013 04:57 AM, Andrey Korolyov wrote:
>>
>> May I also suggest the same for export/import mechanism? Say, if image
>> was created by fallocate we may also want to leave holes upon upload
>> and vice-versa for export.
>
>
> Import and e
Some additionnal informations : if I look on one PG only, for example
the 6.31f. "ceph pg dump" report a size of 616GB :
# ceph pg dump | grep ^6\\. | awk '{ SUM+=($6/1024/1024) } END { print SUM }'
631717
But on disk, on the 3 replica I have :
# du -sh /var/lib/ceph/osd/ceph-50/current/6.31f_he
I also checked that all files in that PG still are on that PG :
for IMG in `find . -type f -printf '%f\n' | awk -F '__' '{ print $1 }' |
sort --unique` ; do echo -n "$IMG "; ceph osd map ssd3copies $IMG | grep
-v 6\\.31f ; echo ; done
And all objects are referenced in rados (compared with "rados
Hi All,
tl;dr - does glance/rbd and cinder/rbd play together nicely in grizzly?
I'm currently testing a ceph/rados back end with an openstack installation.
I have the following things working OK:
1. cinder configured to create volumes in RBD
2. nova configured to boot from RBD backed cinder vol
Hi all,
recently i read the source code and paper, and i have some questions about the
data movement:
1. when OSD's add or removal, how Ceph do this data migration and rebalance the
crush map? is it the rados modify the crush map or cluster map, and the primary
OSD does the data movement accor
On Tue, 10 Sep 2013, Pavel Timoschenkov wrote:
> OSD created only if I use single disk for data and journal.
>
> Situation with separate disks:
> 1.
> ceph-deploy disk zap ceph001:sdaa ceph001:sda1
> [ceph_deploy.osd][DEBUG ] zapping /dev/sdaa on ceph001
> [ceph_deploy.osd][DEBUG ] zapping /dev/sd
On 09/10/2013 12:59 AM, Gaylord Holder wrote:
There are a lot of numbers ceph status prints.
Is there any documentation on what they are?
I'm particulary curious about what seems a total data.
ceph status says I have 314TB, when I calculate I have 24TB.
It also says:
10615 GB used, 8005 GB /
On Tue, 10 Sep 2013, atrmat wrote:
> Hi all,
> recently i read the source code and paper, and i have some questions about
> the data movement:
> 1. when OSD's add or removal, how Ceph do this data migration and rebalance
> the crush map? is it the rados modify the crush map or cluster map, and the
This point release fixes a few important performance regressions with the
OSD (both with CPU and disk utilization), as well as several other
important but less common problems. We recommend that all production users
upgrade.
Notable changes since v0.67.2 include:
* ceph-disk: partprobe after
Can you post the rest of you crush map?
-Sam
On Tue, Sep 10, 2013 at 5:52 AM, Olivier Bonvalet wrote:
> I also checked that all files in that PG still are on that PG :
>
> for IMG in `find . -type f -printf '%f\n' | awk -F '__' '{ print $1 }' |
> sort --unique` ; do echo -n "$IMG "; ceph osd map
On Tue, Sep 10, 2013 at 10:54 AM, Oliver Daudey wrote:
> Hey list,
>
> I just upgraded to Ceph 0.67.3. What I did on every node of my 3-node
> cluster was:
> - Unmount CephFS everywhere.
> - Upgrade the Ceph-packages.
> - Restart MON.
> - Restart OSD.
> - Restart MDS.
>
> As soon as I got to the
Yes I am able to do that.
On Fri, Sep 6, 2013 at 8:19 AM, Alfredo Deza wrote:
> On Fri, Sep 6, 2013 at 11:05 AM, sriram wrote:
> > sudo su -c 'rpm --import
> > "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
> > error: https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/re
Hey list,
I just upgraded to Ceph 0.67.3. What I did on every node of my 3-node
cluster was:
- Unmount CephFS everywhere.
- Upgrade the Ceph-packages.
- Restart MON.
- Restart OSD.
- Restart MDS.
As soon as I got to the second node, the MDS crashed right after startup.
Part of the logs (more on
Le mardi 10 septembre 2013 à 11:19 -0700, Samuel Just a écrit :
> Can you post the rest of you crush map?
> -Sam
>
Yes :
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
devi
Darren,
I can confirm Copy on Write (show_image_direct_url = True) does work in
Grizzly.
It sounds like you are close. To check permissions, run 'ceph auth
list', and reply with "client.images" and "client.volumes" (or whatever
keys you use in Glance and Cinder).
Cheers,
Mike Dawson
On 9
and for more visibility (I hope :D), the osd tree :
# idweight type name up/down reweight
-8 11.65 root SSDroot
-33 5.8 datacenter SSDrbx1
-32 5.8 room SSDs01
-31 5.8 net SSD188-165-15
-30 5.8
I removed some garbage about hosts faude / rurkh / murmillia (they was
temporarily added because cluster was full). So the "clean" CRUSH map :
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
# devices
device 0 device0
device 1 d
Hi Mike,
Thanks - glad to hear it definitely works as expected! Here's my
client.glance and client.volumes from 'ceph auth list':
client.glance
key: AQAWFi9SOKzAABAAPV1ZrpWkx72tmJ5E7nOi3A==
caps: [mon] allow r
caps: [osd] allow rwx pool=images, allow class-read object_prefix
rbd_children
client.
Hi Sriram,
this should help: http://ceph.com/docs/master/install/rpm/
Regards,
Tamil
On Tue, Sep 10, 2013 at 12:55 PM, sriram wrote:
> Can someone tell me the equivalent steps in RHEL for the steps below -
>
> wget -q -O-
> 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' |
Can someone tell me the equivalent steps in RHEL for the steps below -
wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'
| sudo apt-key add -
echo deb http://ceph.com/debian-dumpling/ $(lsb_release -sc) main |
sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt-get upd
On 9/10/2013 4:50 PM, Darren Birkett wrote:
Hi Mike,
That led me to realise what the issue was. My cinder (volumes) client
did not have the correct perms on the images pool. I ran the following
to update the perms for that client:
ceph auth caps client.volumes mon 'allow r' osd 'allow class-
It's not an upgrade issue. There's an MDS object that is somehow
missing. If it exists, then on restart you'll be fine.
Oliver, what is your general cluster config? What filesystem are your
OSDs running on? What version of Ceph were you upgrading from? There's
really no way for this file to not ex
On 09/10/2013 01:50 PM, Darren Birkett wrote:
One last question: I presume the fact that the 'volume_image_metadata'
field is not populated when cloning a glance image into a cinder volume
is a bug? It means that the cinder client doesn't show the volume as
bootable, though I'm not sure what oth
Also, can you scrub the PG which contains the "mds_anchortable" object
and see if anything comes up? You should be able to find the key from
the logs (in the osd_op line that contains "mds_anchortable") and
convert that into the PG. Or you can just scrub all of osd 2.
-Greg
Software Engineer #42 @
This is scary. Should I hold on upgrade?
On 9/10/13 11:33 AM, "Oliver Daudey" wrote:
>Hey Gregory,
>
>On 10-09-13 20:21, Gregory Farnum wrote:
>> On Tue, Sep 10, 2013 at 10:54 AM, Oliver Daudey
>>wrote:
>>> Hey list,
>>>
>>> I just upgraded to Ceph 0.67.3. What I did on every node of my 3-node
Hi Mike,
That led me to realise what the issue was. My cinder (volumes) client did
not have the correct perms on the images pool. I ran the following to
update the perms for that client:
ceph auth caps client.volumes mon 'allow r' osd 'allow class-read
object_prefix rbd_children, allow rwx pool
On Tue, Sep 10, 2013 at 4:33 PM, sriram wrote:
> I had followed that to install ceph-deploy but then I am not sure if there
> is a difference between -
>
> INSTALL-CEPH-DEPLOY described in
> http://ceph.com/docs/master/start/quick-start-preflight/
The preflight seems to cover installation of ceph
If the problem is somewhere in RADOS/xfs/whatever, then there's a good
chance that the "mds_anchortable" object exists in its replica OSDs,
but when listing objects those aren't queried, so they won't show up
in a listing. You can use the osdmaptool to map from an object name to
the PG it would sho
Hey Gregory,
The only objects containing "table" I can find at all, are in the
"metadata"-pool:
# rados --pool=metadata ls | grep -i table
mds0_inotable
Looking at another cluster where I use CephFS, there is indeed an object
named "mds_anchortable", but the broken cluster is missing it. I don't
On Tue, Sep 10, 2013 at 2:36 PM, Oliver Daudey wrote:
> Hey Gregory,
>
> My cluster consists of 3 nodes, each running 1 mon, 1 osd and 1 mds. I
> upgraded from 0.67, but was still running 0.61.7 OSDs at the time of the
> upgrade, because of performance-issues that have just recently been
> fixed.
On 09/10/2013 01:51 AM, Andrey Korolyov wrote:
On Tue, Sep 10, 2013 at 3:03 AM, Josh Durgin wrote:
On 09/09/2013 04:57 AM, Andrey Korolyov wrote:
May I also suggest the same for export/import mechanism? Say, if image
was created by fallocate we may also want to leave holes upon upload
and vic
Hey Gregory,
My cluster consists of 3 nodes, each running 1 mon, 1 osd and 1 mds. I
upgraded from 0.67, but was still running 0.61.7 OSDs at the time of the
upgrade, because of performance-issues that have just recently been
fixed. These have now been upgraded to 0.67.3, along with the rest of
C
Hey Gregory,
On 10-09-13 20:21, Gregory Farnum wrote:
> On Tue, Sep 10, 2013 at 10:54 AM, Oliver Daudey wrote:
>> Hey list,
>>
>> I just upgraded to Ceph 0.67.3. What I did on every node of my 3-node
>> cluster was:
>> - Unmount CephFS everywhere.
>> - Upgrade the Ceph-packages.
>> - Restart MON
Nope, a repair won't change anything if scrub doesn't detect any
inconsistencies. There must be something else going on, but I can't
fathom what...I'll try and look through it a bit more tomorrow. :/
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
On Tue, Sep 10, 2013 at 3:49 P
Hey Gregory,
Thanks for your explanation. Turns out to be 1.a7 and it seems to scrub
OK.
# ceph osd getmap -o osdmap
# osdmaptool --test-map-object mds_anchortable --pool 1 osdmap
osdmaptool: osdmap file 'osdmap'
object 'mds_anchortable' -> 1.a7 -> [2,0]
# ceph pg scrub 1.a7
osd.2 logs:
2013-0
I installed xfsprogs from
http://rpm.pbone.net/index.php3/stat/26/dist/74/size/1400502/name/xfsprogs-3.1.1-4.el6.src.rpm
.
I then ran "sudo yum install ceph" and I still get the same error. Any
ideas?
On Wed, Aug 28, 2013 at 3:47 PM, sriram wrote:
> Can anyone point me to which xfsprogs RPM to
Hey Gregory,
Ok, thanks for all your help! It's weird, as if the object gets deleted
somewhere along the way, but the problem only becomes visible once you
restart the MDSs, which probably have it in memory and then fail to load
it after restart. I'll answer the questions you had about my test-s
Hey Gregory,
On di, 2013-09-10 at 14:48 -0700, Gregory Farnum wrote:
> On Tue, Sep 10, 2013 at 2:36 PM, Oliver Daudey wrote:
> > Hey Gregory,
> >
> > My cluster consists of 3 nodes, each running 1 mon, 1 osd and 1 mds. I
> > upgraded from 0.67, but was still running 0.61.7 OSDs at the time of th
Hello,
Got so-famous error on 0.61.8, just for little disk overload on OSD
daemon start. I currently have very large metadata per osd (about
20G), this may be an issue.
#0 0x7f2f46adeb7b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00860469 in reraise_fatal (signum=6)
Any help here is appreciated. I am pretty much stuck in trying to install
ceph on my local box.
On Tue, Sep 10, 2013 at 11:02 AM, sriram wrote:
> Yes I am able to do that.
>
>
> On Fri, Sep 6, 2013 at 8:19 AM, Alfredo Deza wrote:
>
>> On Fri, Sep 6, 2013 at 11:05 AM, sriram wrote:
>> > sudo su
I had followed that to install ceph-deploy but then I am not sure if there
is a difference between -
INSTALL-CEPH-DEPLOY described in
http://ceph.com/docs/master/start/quick-start-preflight/
and
INSTALLING CEPH DEPLOY described in http://ceph.com/docs/master/install/rpm/
The reason "ceph-deplo
I am new to ceph.I am trying to follow the official document to install
ceph on the machine .All things work fine but whenever I try to install
ceph using ceph-deploy install ceph-server I get the following error
http://pastebin.com/HxzP5Npi . I am behind a proxy server .Initially I
thought the pro
Hi,
I believe you need to tell apt about your proxy server:
cat /etc/apt/apt.conf
Acquire::http::Proxy "http://my.proxy.server:3142";;
wogri
On 09/11/2013 08:28 AM, kumar rishabh wrote:
> I am new to ceph.I am trying to follow the official document to install
> ceph on the machine .All things
Does noone have an idea ?
I can't mount the cluster anymore.
Thank you,
Markus
Am 10.09.2013 09:43, schrieb Markus Goldberg:
Hi,
i made a 'stop ceph-all' on my ceph-admin-host and then a
kernel-upgrade from 3.9 to 3.11 on all of my 3 nodes.
Ubuntu 13.04, ceph 0,68
The kernel-upgrade require
47 matches
Mail list logo
