Re: [ceph-users] adding osd node best practice

[Anthony D'Atri]

> We have application cluster and ceph as storage solution, cluster consists of 
> six servers, so we've installed
> monitor on every one of them, to have ceph cluster sane (quorum) if server or 
> two of them goes down. 

You want an odd number for sure, to avoid the classic split-brain problem:

http://ceph.com/docs/master/rados/operations/add-or-rm-mons/

I think the bit re diminishing returns with >5 mons was told to me by a 
consultant, but I don’t have a reference.  The more you have the more traffic 
they have to exchange among themselves, I’m thinking that’s probably not a huge 
deal until N gets a lot bigger.

>  or is it not necessary/recommended to have mon on node with osds?

I’ve read multiple documents recommending against an AIO config, IIRC e.g.. so 
that heavy backfilling or client operations to the OSD’s don’t starve the mons. 
Best to Google around a bit, the size/density/number/workload of your OSD’s is 
likely a significant factor.  On a small cluster I can see the appeal of an AIO 
strategy, unless you perhaps have hypervisors on the appropriate network and 
might consider running mons as VM’s with resource reservations.


—aad



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Stuck PGs blocked_by non-existent OSDs

[joel.merr...@gmail.com]

Sure thing, n.b. I increased pg count to see if it would help. Alas not. :)

Thanks again!

health_detail
https://gist.github.com/199bab6d3a9fe30fbcae

osd_dump
https://gist.github.com/499178c542fa08cc33bb

osd_tree
https://gist.github.com/02b62b2501cbd684f9b2

Random selected queries:
queries/0.19.query
https://gist.github.com/f45fea7c85d6e665edf8
queries/1.a1.query
https://gist.github.com/dd68fbd5e862f94eb3be
queries/7.100.query
https://gist.github.com/d4fd1fb030c6f2b5e678
queries/7.467.query
https://gist.github.com/05dbcdc9ee089bd52d0c

On Tue, Mar 10, 2015 at 2:49 PM, Samuel Just  wrote:
> Yeah, get a ceph pg query on one of the stuck ones.
> -Sam
>
> On Tue, 2015-03-10 at 14:41 +, joel.merr...@gmail.com wrote:
>> Stuck unclean and stuck inactive. I can fire up a full query and
>> health dump somewhere useful if you want (full pg query info on ones
>> listed in health detail, tree, osd dump etc). There were blocked_by
>> operations that no longer exist after doing the OSD addition.
>>
>> Side note, spent some time yesterday writing some bash to do this
>> programatically (might be useful to others, will throw on github)
>>
>> On Tue, Mar 10, 2015 at 1:41 PM, Samuel Just  wrote:
>> > What do you mean by "unblocked" but still "stuck"?
>> > -Sam
>> >
>> > On Mon, 2015-03-09 at 22:54 +, joel.merr...@gmail.com wrote:
>> >> On Mon, Mar 9, 2015 at 2:28 PM, Samuel Just  wrote:
>> >> > You'll probably have to recreate osds with the same ids (empty ones),
>> >> > let them boot, stop them, and mark them lost.  There is a feature in the
>> >> > tracker to improve this behavior: http://tracker.ceph.com/issues/10976
>> >> > -Sam
>> >>
>> >> Thanks Sam, I've readded the OSDs, they became unblocked but there are
>> >> still the same number of pgs stuck. I looked at them in some more
>> >> detail and it seems they all have num_bytes='0'. Tried a repair too,
>> >> for good measure. Still nothing I'm afraid.
>> >>
>> >> Does this mean some underlying catastrophe has happened and they are
>> >> never going to recover? Following on, would that cause data loss.
>> >> There are no missing objects and I'm hoping there's appropriate
>> >> checksumming / replicas to balance that out, but now I'm not so sure.
>> >>
>> >> Thanks again,
>> >> Joel
>> >
>> >
>>
>>
>>
>
>



-- 
$ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

[Malcolm Haak]

I ran ceph-osd via the command line...

It's not really given me much more to go off...  Well except that it's hitting 
an early end of buffer for some reason.

Also I've hit another issue... 

The /etc/init.d/ceph script is not seeing my new mon (I decided to add more 
mon's to see if it would help since the mon map looks like it is the issue)

The script starts the mon fine. And the new mon (on the same host as this 
problem osd) appears to be good. 

The issue is when you do /etc/init.d/ceph status 

It tells you the mon.b is dead.. It seems to be one of the greps that is failing
Specifically 
grep -qwe -i.$daemon_id /proc/\$pid/cmdline
returns 1

What's odd is the same grep works on the other node for mon.a it just doesn't 
work on this node for mon.b

I'm wondering if there is something odd happening. 

Anyway here is the output of the manual start of ceph-osd


# /usr/bin/ceph-osd -i 3 --pid-file /var/run/ceph/osd.3.pid -c 
/etc/ceph/ceph.conf --cluster ceph -f
starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 
/var/lib/ceph/osd/ceph-3/journal
2015-03-11 20:38:56.401205 7f04221e6880 -1 journal FileJournal::_open: 
disabling aio for non-block journal.  Use journal_force_aio t   

o force use of aio anyway
2015-03-11 20:38:56.418747 7f04221e6880 -1 osd.3 2757 log_to_monitors 
{default=true}
terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
*** Caught signal (Aborted) **
 in thread 7f041192a700
 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
 1: /usr/bin/ceph-osd() [0xac7cea]
 2: (()+0x10050) [0x7f04210f1050]
 3: (gsignal()+0x37) [0x7f041f5c40f7]
 4: (abort()+0x13a) [0x7f041f5c54ca]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f041fea9fe5]
 6: (()+0x63186) [0x7f041fea8186]
 7: (()+0x631b3) [0x7f041fea81b3]
 8: (()+0x633d2) [0x7f041fea83d2]
 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
 16: (DispatchQueue::entry()+0x649) [0xc6fe09]
 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
 18: (()+0x83a4) [0x7f04210e93a4]
 19: (clone()+0x6d) [0x7f041f673a4d]
2015-03-11 20:38:56.471624 7f041192a700 -1 *** Caught signal (Aborted) **
 in thread 7f041192a700

 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
 1: /usr/bin/ceph-osd() [0xac7cea]
 2: (()+0x10050) [0x7f04210f1050]
 3: (gsignal()+0x37) [0x7f041f5c40f7]
 4: (abort()+0x13a) [0x7f041f5c54ca]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f041fea9fe5]
 6: (()+0x63186) [0x7f041fea8186]
 7: (()+0x631b3) [0x7f041fea81b3]
 8: (()+0x633d2) [0x7f041fea83d2]
 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
 16: (DispatchQueue::entry()+0x649) [0xc6fe09]
 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
 18: (()+0x83a4) [0x7f04210e93a4]
 19: (clone()+0x6d) [0x7f041f673a4d]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.

  -308> 2015-03-11 20:38:56.401205 7f04221e6880 -1 journal FileJournal::_open: 
disabling aio for non-block journal.  Use journal_for   

ce_aio to force use of aio anyway
   -77> 2015-03-11 20:38:56.418747 7f04221e6880 -1 osd.3 2757 log_to_monitors 
{default=true}
 0> 2015-03-11 20:38:56.471624 7f041192a700 -1 *** Caught signal (Aborted) 
**
 in thread 7f041192a700

 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
 1: /usr/bin/ceph-osd() [0xac7cea]
 2: (()+0x10050) [0x7f04210f1050]
 3: (gsignal()+0x37) [0x7f041f5c40f7]
 4: (abort()+0x13a) [0x7f041f5c54ca]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f041fea9fe5]
 6: (()+0x63186) [0x7f041fea8186]
 7: (()+0x631b3) [0x7f041fea81b3]
 8: (()+0x633d2) [0x7f041fea83d2]
 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
 13: (OSD::handle_osd_map(MOS

Re: [ceph-users] Firefly Tiering

[Nick Fisk]

> Hi Nick,
> 
> Am 11.03.2015 um 10:52 schrieb Nick Fisk:
> > Hi Stefan,
> >
> > If the majority of your hot data fits on the cache tier you will see
> > quite a marked improvement in read performance
> I don't have writes ;-) just around 5%. 95% are writes.
> 
> > and similar write performance
> > (assuming you would have had your hdds backed by SSD journals).
> 
> similar write performance of SSD cache tier or HDD "backend" tier?
> 
> I'm mainly interested in a writeback mode.

Writes on Cache tiering are the same speed as a non cache tiering solution
(with SSD journals), if the blocks are in the cache. 


> 
> > However for data that is not in the cache tier you will get 10-20%
> > less read performance and anything up to 10x less write performance.
> > This is because a cache write miss has to read the entire object from
> > the backing store into the cache and then modify it.
> >
> > The read performance degradation will probably be fixed in Hammer with
> > proxy reads, but writes will most likely still be an issue.
> 
> Why is writing to the HOT part so slow?
> 

If the object is in the cache tier or currently doesn't exist, then writes
are fast as it just has to write directly to the cache tier SSD's. However
if the object is in the slow tier and you write to it, then its very slow.
This is because it has to read it off the slow tier (~12ms), write it on to
the cache tier(~.5ms) and then update it (~.5ms).

With a non caching solution, you would have just written straight to the
journal (~.5ms)

> Stefan
> 
> > Nick
> >
> >
> >> -Original Message-
> >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf
> >> Of Stefan Priebe - Profihost AG
> >> Sent: 11 March 2015 07:27
> >> To: ceph-users@lists.ceph.com
> >> Subject: [ceph-users] Firefly Tiering
> >>
> >> Hi,
> >>
> >> has anybody successfully tested tiering while using firefly? How much
> >> does
> > it
> >> impact performance vs. a normal pool? I mean is there any difference
> >> between a full SSD pool und a tiering SSD pool with SATA Backend?
> >>
> >> Greets,
> >> Stefan
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] hang osd --zap-disk

[Loic Dachary]

Hi,

It means the monitor 192.168.4.35:6789 cannot be accessed from the tauro 
machine.

Cheers

On 12/03/2015 00:55, Jesus Chavez (jeschave) wrote:
> 
> I don’t know what is going on =( the system hangs with the message below 
> after commaand "ceph-deploy osd --zap-disk create tauro:sdb” 
> 
> [tauro][WARNING] No data was received after 300 seconds, disconnecting...
> [ceph_deploy.osd][DEBUG ] Host tauro is now ready for osd use.
> [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
> [ceph_deploy.cli][INFO  ] Invoked (1.5.22): /usr/bin/ceph-deploy osd activate 
> tauro:sdb1
> [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks tauro:/dev/sdb1:
> [tauro][DEBUG ] connection detected need for sudo
> [tauro][DEBUG ] connected to host: tauro
> [tauro][DEBUG ] detect platform information from remote host
> [tauro][DEBUG ] detect machine type
> [ceph_deploy.osd][INFO  ] Distro info: Red Hat Enterprise Linux Server 7.1 
> Maipo
> [ceph_deploy.osd][DEBUG ] activating host tauro disk /dev/sdb1
> [ceph_deploy.osd][DEBUG ] will use init type: sysvinit
> [tauro][INFO  ] Running command: sudo ceph-disk -v activate --mark-init 
> sysvinit --mount /dev/sdb1
> [tauro][WARNING] INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE 
> -ovalue -- /dev/sdb1
> [tauro][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-conf 
> --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
> [tauro][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-conf 
> --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
> [tauro][WARNING] DEBUG:ceph-disk:Mounting /dev/sdb1 on 
> /var/lib/ceph/tmp/mnt.lNpFro with options noatime,inode64
> [tauro][WARNING] INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o 
> noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.lNpFro
> [tauro][WARNING] DEBUG:ceph-disk:Cluster uuid is 
> fc72a252-15be-40e9-9de1-34593be5668a
> [tauro][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-osd 
> --cluster=ceph --show-config-value=fsid
> [tauro][WARNING] DEBUG:ceph-disk:Cluster name is ceph
> [tauro][WARNING] DEBUG:ceph-disk:OSD uuid is 
> bf192166-86e9-4c68-9bff-7ced1c9ba8ee
> [tauro][WARNING] DEBUG:ceph-disk:Allocating OSD id...
> [tauro][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph 
> --name client.bootstrap-osd --keyring 
> /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise 
> bf192166-86e9-4c68-9bff-7ced1c9ba8ee
> [tauro][WARNING] 2015-03-11 17:49:31.782184 7f9cf05a8700  0 -- :/1015927 >> 
> 192.168.4.35:6789/0 pipe(0x7f9cec0253f0 sd=4 :0 s=1 pgs=0 cs=0 l=1 
> c=0x7f9cec025680).fault
> [tauro][WARNING] 2015-03-11 17:49:35.782524 7f9cf04a7700  0 -- :/1015927 >> 
> 192.168.4.35:6789/0 pipe(0x7f9cec00 sd=4 :0 s=1 pgs=0 cs=0 l=1 
> c=0x7f9cee90).fault
> [tauro][WARNING] 2015-03-11 17:49:37.781846 7f9cf05a8700  0 -- :/1015927 >> 
> 192.168.4.35:6789/0 pipe(0x7f9ce00030e0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
> c=0x7f9ce0003370).fault
> [tauro][WARNING] 2015-03-11 17:49:41.782566 7f9cf04a7700  0 -- :/1015927 >> 
> 192.168.4.35:6789/0 pipe(0x7f9cec00 sd=5 :0 s=1 pgs=0 cs=0 l=1 
> c=0x7f9cee90).fault
> [tauro][WARNING] 2015-03-11 17:49:43.782303 7f9cf05a8700  0 -- :/1015927 >> 
> 192.168.4.35:6789/0 pipe(0x7f9ce00031b0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
> c=0x7f9ce00025d0).fault
> [tauro][WARNING] 2015-03-11 17:49:47.784627 7f9cf04a7700  0 -- :/1015927 >> 
> 192.168.4.35:6789/0 pipe(0x7f9cec00 sd=5 :0 s=1 pgs=0 cs=0 l=1 
> c=0x7f9cee90).fault
> [tauro][WARNING] 2015-03-11 17:49:49.782712 7f9cf05a8700  0 -- :/1015927 >> 
> 192.168.4.35:6789/0 pipe(0x7f9ce00031b0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
> c=0x7f9ce0002c60).fault
> [tauro][WARNING] 2015-03-11 17:49:53.784690 7f9cf04a7700  0 -- :/1015927 >> 
> 192.168.4.35:6789/0 pipe(0x7f9ce0003fb0 sd=4 :0 s=1 pgs=0 cs=0 l=1 
> c=0x7f9ce0004240).fault
> [tauro][WARNING] 2015-03-11 17:49:55.783248 7f9cf05a8700  0 -- :/1015927 >> 
> 192.168.4.35:6789/0 pipe(0x7f9ce0004930 sd=4 :0 s=1 pgs=0 cs=0 l=1 
> c=0x7f9ce0004bc0)
> 
> *
> *Jesus Chavez**
> SYSTEMS ENGINEER-C.SALES
> 
> jesch...@cisco.com 
> Phone: *+52 55 5267 3146*
> Mobile: *+51 1 5538883255*
> 
> CCIE - 44433
> 
>   
> 
> 
> Cisco.com 
> 
> 
> 
> 
> 
> 
>   Think before you print.
> 
> This email may contain confidential and privileged material for the sole use 
> of the intended recipient. Any review, use, distribution or disclosure by 
> others is strictly prohibited. If you are not the intended recipient (or 
> authorized to receive for the recipient), please contact the sender by reply 
> email and delete all copies of this message.
> 
> Please click here 
>  for 
> Company Registration Information.
> 
> 
> 
> 
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Loïc Dachary, 

[ceph-users] [ERROR ] Some monitors have still not reached quorum

[Jesus Chavez (jeschave)]

Hi all


Does anybody know why am I having this error:

[ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying
[ceph_deploy.mon][ERROR ] Some monitors have still not reached quorum:
[ceph_deploy.mon][ERROR ] capricornio
[ceph_deploy.mon][ERROR ] tauro
[ceph_deploy.mon][ERROR ] aries



the installation sequence was:


ceph-deploy new aries capricornio tauro

vi /my-cluster/ceph.conf

[global]
fsid = 66239891-8129-46e3-83cd-845f27af189b
mon_initial_members = aries, capricornio, tauro
mon_host = 192.168.4.35,192.168.4.44,192.168.4.36
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd pool default size = 2
public network = 192.168.4.0/25


ceph-deploy install aries capricornio tauro
ceph-deploy mon create-initial


thanks


[cid:image005.png@01D00809.A6D502D0]


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433


Cisco.com





[cid:image006.gif@01D00809.A6D502D0]



  Think before you print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.

Please click 
here for 
Company Registration Information.





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Steffen W Sørensen]

On 12/03/2015, at 03.08, Jesus Chavez (jeschave)  wrote:
> Thanks Steffen I have followed everything not sure what is going on, the mon 
> keyring and client admin are individual? Per mon host? Or do I need to copy 
> from the first initial mon node?
I'm no expert, but I would assume keyring could be both as long as it got the 
right permissions.
I followed the manually route once with success

/Steffen



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Doesn't Support Qcow2 Disk images

[Azad Aliyar]

Community please explain the 2nd warning on this page:

http://ceph.com/docs/master/rbd/rbd-openstack/

"Important Ceph doesn’t support QCOW2 for hosting a virtual machine disk.
Thus if you want to boot virtual machines in Ceph (ephemeral backend or
boot from volume), the Glance image format must be RAW."


-- 
   Warm Regards,  Azad Aliyar
 Linux Server Engineer
 *Email* :  azad.ali...@sparksupport.com   *|*   *Skype* :   spark.azad
 

  
3rd Floor, Leela Infopark, Phase -2,Kakanad, Kochi-30, Kerala, India
*Phone*:+91 484 6561696 , *Mobile*:91-8129270421.   *Confidentiality
Notice:* Information in this e-mail is proprietary to SparkSupport. and is
intended for use only by the addressed, and may contain information that is
privileged, confidential or exempt from disclosure. If you are not the
intended recipient, you are notified that any use of this information in
any manner is strictly prohibited. Please delete this mail & notify us
immediately at i...@sparksupport.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

[Malcolm Haak]

Hi all,

So the init script issue is sorted.. my grep binary is not working correctly.  
I've replaced it and everything seems to be fine. 

Which now has me wondering if the binaries I generated are any good... the bad 
grep might have caused issues with the build...

I'm going to recompile after some more sanity testing..

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Malcolm Haak
Sent: Wednesday, 11 March 2015 8:56 PM
To: Samuel Just; jl...@redhat.com
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

I ran ceph-osd via the command line...

It's not really given me much more to go off...  Well except that it's hitting 
an early end of buffer for some reason.

Also I've hit another issue... 

The /etc/init.d/ceph script is not seeing my new mon (I decided to add more 
mon's to see if it would help since the mon map looks like it is the issue)

The script starts the mon fine. And the new mon (on the same host as this 
problem osd) appears to be good. 

The issue is when you do /etc/init.d/ceph status 

It tells you the mon.b is dead.. It seems to be one of the greps that is failing
Specifically 
grep -qwe -i.$daemon_id /proc/\$pid/cmdline
returns 1

What's odd is the same grep works on the other node for mon.a it just doesn't 
work on this node for mon.b

I'm wondering if there is something odd happening. 

Anyway here is the output of the manual start of ceph-osd


# /usr/bin/ceph-osd -i 3 --pid-file /var/run/ceph/osd.3.pid -c 
/etc/ceph/ceph.conf --cluster ceph -f
starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 
/var/lib/ceph/osd/ceph-3/journal
2015-03-11 20:38:56.401205 7f04221e6880 -1 journal FileJournal::_open: 
disabling aio for non-block journal.  Use journal_force_aio t   

o force use of aio anyway
2015-03-11 20:38:56.418747 7f04221e6880 -1 osd.3 2757 log_to_monitors 
{default=true}
terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
*** Caught signal (Aborted) **
 in thread 7f041192a700
 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
 1: /usr/bin/ceph-osd() [0xac7cea]
 2: (()+0x10050) [0x7f04210f1050]
 3: (gsignal()+0x37) [0x7f041f5c40f7]
 4: (abort()+0x13a) [0x7f041f5c54ca]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f041fea9fe5]
 6: (()+0x63186) [0x7f041fea8186]
 7: (()+0x631b3) [0x7f041fea81b3]
 8: (()+0x633d2) [0x7f041fea83d2]
 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
 16: (DispatchQueue::entry()+0x649) [0xc6fe09]
 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
 18: (()+0x83a4) [0x7f04210e93a4]
 19: (clone()+0x6d) [0x7f041f673a4d]
2015-03-11 20:38:56.471624 7f041192a700 -1 *** Caught signal (Aborted) **
 in thread 7f041192a700

 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
 1: /usr/bin/ceph-osd() [0xac7cea]
 2: (()+0x10050) [0x7f04210f1050]
 3: (gsignal()+0x37) [0x7f041f5c40f7]
 4: (abort()+0x13a) [0x7f041f5c54ca]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f041fea9fe5]
 6: (()+0x63186) [0x7f041fea8186]
 7: (()+0x631b3) [0x7f041fea81b3]
 8: (()+0x633d2) [0x7f041fea83d2]
 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
 16: (DispatchQueue::entry()+0x649) [0xc6fe09]
 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
 18: (()+0x83a4) [0x7f04210e93a4]
 19: (clone()+0x6d) [0x7f041f673a4d]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.

  -308> 2015-03-11 20:38:56.401205 7f04221e6880 -1 journal FileJournal::_open: 
disabling aio for non-block journal.  Use journal_for   

ce_aio to force use of aio anyway
   -77> 2015-03-11 20:38:56.418747 7f04221e6880 -1 osd.3 2757 log_to_monitors 
{default=true}
 0> 2015-03-11 20:38:56.471624 7f041192a700 -1 *** Caught signal (Aborted) 
**
 in thread 7f041192a700

 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
 1: /

Re: [ceph-users] Doesn't Support Qcow2 Disk images

[Andrija Panic]

ceph is RAW format - should be all fine...so VM will be using that RAW
format

On 12 March 2015 at 09:03, Azad Aliyar  wrote:

> Community please explain the 2nd warning on this page:
>
> http://ceph.com/docs/master/rbd/rbd-openstack/
>
> "Important Ceph doesn’t support QCOW2 for hosting a virtual machine disk.
> Thus if you want to boot virtual machines in Ceph (ephemeral backend or
> boot from volume), the Glance image format must be RAW."
>
>
> --
>Warm Regards,  Azad Aliyar
>  Linux Server Engineer
>  *Email* :  azad.ali...@sparksupport.com   *|*   *Skype* :   spark.azad
>  
> 
> 
> 3rd Floor, Leela Infopark, Phase
> -2,Kakanad, Kochi-30, Kerala, India  *Phone*:+91 484 6561696 , 
> *Mobile*:91-8129270421.
>   *Confidentiality Notice:* Information in this e-mail is proprietary to
> SparkSupport. and is intended for use only by the addressed, and may
> contain information that is privileged, confidential or exempt from
> disclosure. If you are not the intended recipient, you are notified that
> any use of this information in any manner is strictly prohibited. Please
> delete this mail & notify us immediately at i...@sparksupport.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 

Andrija Panić
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Duplication name Container

[Jimmy Goffaux]

Hello All,

I use Ceph in production for several months. but i have an errors with 
Ceph Rados Gateway for multiple users.


I am faced with the following error:

Error trying to create container 'xs02': 409 Conflict: 
BucketAlreadyExists


Which corresponds to the documentation : 
http://ceph.com/docs/master/radosgw/s3/bucketops/


By which means I can avoid this kind of problem?

Here are my versions used:

radosgw-agent  => 1.2-1precise
ceph   => 0.87-1precise

Thank you for your help

--

Jimmy Goffaux
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Sparse RBD instance snapshots in OpenStack

[Charles 'Boyo]

Hello all.

The current behavior of snapshotting instances RBD-backed in OpenStack involves 
uploading the snapshot into Glance.

The resulting Glance image is fully allocated, causing an explosion of 
originally sparse RAW images. Is there a way to preserve the sparseness? Else I 
can use qemu-img convert (or rbd export/import) to manually sparsify it?

On a related note, my Glance is also backed by the same Ceph cluster, in 
another pool and I was wondering if Ceph snapshots would not be a better way to 
do this. Any ideas?

Regards,

Charles

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Monitor stay in synchronizing state for over 24hour

[Luke Kao]

Hello everyone,
I am currently trying to recover a ceph cluster from the disaster, now I have 
enough osd (171 up and in/195) and have 2 incomplete pgs at the end.

However the question now is not the incomplete pgs, is about one mon services 
fail to start due to a strange, wrong monmap is used.  After inject monmap 
exported from cluster, it's up and enter synchronizing and unable to be back 
after several hours.  I originally guess it's common for the fact the whole 
cluster is still busy in recovering and backfilling, however it's over 24hour 
now and no hint when sync can be done or if it's still in healthy status.

The log tells it is still doing synchronizing and I can see the file under 
store.db keep being updated.


a small piece of log for the reference:
2015-03-12 03:20:15.025048 7f3cb6c48700 10 
mon.NVMBD1CIF290D00@0(synchronizing).data_health(0) service_tick
2015-03-12 03:20:15.025075 7f3cb6c48700  0 
mon.NVMBD1CIF290D00@0(synchronizing).data_health(0) update_stats avail 71% 
total 103080888 used 24281956 avail 73539668
2015-03-12 03:20:30.460672 7f3cb4b43700 10 -- 10.137.36.30:6789/0 >> 
10.137.36.31:6789/0 pipe(0x3528280 sd=9 :57111 s=2 pgs=30630 cs=15 l=0 
c=0x34b1760).aborted = 0
2015-03-12 03:20:30.460923 7f3cb4b43700 10 -- 10.137.36.30:6789/0 >> 
10.137.36.31:6789/0 pipe(0x3528280 sd=9 :57111 s=2 pgs=30630 cs=15 l=0 
c=0x34b1760).reader got message 1466470577 0x45b3c80 mon_sync(chunk cookie 
37950063980 lc 12343379 bl 791970 bytes last_key logm,full_5120265) v2
2015-03-12 03:20:30.460963 7f3cbc783700 10 -- 10.137.36.30:6789/0 >> 
10.137.36.31:6789/0 pipe(0x3528280 sd=9 :57111 s=2 pgs=30630 cs=15 l=0 
c=0x34b1760).writer: state = open policy.server=0
2015-03-12 03:20:30.460988 7f3cbc783700 10 -- 10.137.36.30:6789/0 >> 
10.137.36.31:6789/0 pipe(0x3528280 sd=9 :57111 s=2 pgs=30630 cs=15 l=0 
c=0x34b1760).write_ack 1466470577
2015-03-12 03:20:30.461011 7f3cbc783700 10 -- 10.137.36.30:6789/0 >> 
10.137.36.31:6789/0 pipe(0x3528280 sd=9 :57111 s=2 pgs=30630 cs=15 l=0 
c=0x34b1760).writer: state = open policy.server=0
2015-03-12 03:20:30.461030 7f3cb6447700  1 -- 10.137.36.30:6789/0 <== mon.1 
10.137.36.31:6789/0 1466470577  mon_sync(chunk cookie 37950063980 lc 
12343379 bl 791970 bytes last_key logm,full_5120265) v2  792163+0+0 
(2147002791 0 0) 0x45b3c80 con 0x34b1760
2015-03-12 03:20:30.461048 7f3cb6447700 10 mon.NVMBD1CIF290D00@0(synchronizing) 
e1 handle_sync mon_sync(chunk cookie 37950063980 lc 12343379 bl 791970 bytes 
last_key logm,full_5120265) v2
2015-03-12 03:20:30.461052 7f3cb6447700 10 mon.NVMBD1CIF290D00@0(synchronizing) 
e1 handle_sync_chunk mon_sync(chunk cookie 37950063980 lc 12343379 bl 791970 
bytes last_key logm,full_5120265) v2
2015-03-12 03:20:30.463832 7f3cb6447700 10 mon.NVMBD1CIF290D00@0(synchronizing) 
e1 sync_reset_timeout


I am also wondering some osd are fail to join cluster due to this.  Some osd 
processes are up without error, but after load pgs, it cannot keep moving to 
boot and status is still down and out.

Please advise, thanks



Luke Kao

MYCOM OSI




This electronic message contains information from Mycom which may be privileged 
or confidential. The information is intended to be for the use of the 
individual(s) or entity named above. If you are not the intended recipient, be 
aware that any disclosure, copying, distribution or any other use of the 
contents of this information is prohibited. If you have received this 
electronic message in error, please notify us by post or telephone (to the 
numbers or correspondence address above) or by email (at the email address 
above) immediately.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Stuck PGs blocked_by non-existent OSDs

[joel.merr...@gmail.com]

Thanks Sam, I'll take a look. Seems sensible enough and worth a shot.

We'll probably call it a day after this and flatten in, but I'm
wondering if it's possible some rbd devices may miss these pg's and
could be exportable? Will have a tinker!



On Wed, Mar 11, 2015 at 7:06 PM, Samuel Just  wrote:
> For each of those pgs, you'll need to identify the pg copy you want to be
> the winner and either
> 1) Remove all of the other ones using ceph-objectstore-tool and hopefully
> the winner you left alone will allow the pg to recover and go active.
> 2) Export the winner using ceph-objectstore-tool, use ceph-objectstore-tool
> to delete *all* copies of the pg, use force_create_pg to recreate the pg
> empty, use ceph-objectstore-tool to do a rados import on the exported pg
> copy.
>
> Also, the pgs which are still down still have replicas which need to be
> brought back or marked lost.
> -Sam
>
>
> On 03/11/2015 07:29 AM, joel.merr...@gmail.com wrote:
>>
>> I'd like to not have to null them if possible, there's nothing
>> outlandishly valuable, its more the time to reprovision (users have
>> stuff on there, mainly testing but I have a nasty feeling some users
>> won't have backed up their test instances). When you say complicated
>> and fragile, could you expand?
>>
>> Thanks again!
>> Joel
>>
>> On Wed, Mar 11, 2015 at 1:21 PM, Samuel Just  wrote:
>>>
>>> Ok, you lost all copies from an interval where the pgs went active. The
>>> recovery from this is going to be complicated and fragile.  Are the pools
>>> valuable?
>>> -Sam
>>>
>>>
>>> On 03/11/2015 03:35 AM, joel.merr...@gmail.com wrote:

 For clarity too, I've tried to drop the min_size before as suggested,
 doesn't make a difference unfortunately

 On Wed, Mar 11, 2015 at 9:50 AM, joel.merr...@gmail.com
  wrote:
>
> Sure thing, n.b. I increased pg count to see if it would help. Alas
> not.
> :)
>
> Thanks again!
>
> health_detail
> https://gist.github.com/199bab6d3a9fe30fbcae
>
> osd_dump
> https://gist.github.com/499178c542fa08cc33bb
>
> osd_tree
> https://gist.github.com/02b62b2501cbd684f9b2
>
> Random selected queries:
> queries/0.19.query
> https://gist.github.com/f45fea7c85d6e665edf8
> queries/1.a1.query
> https://gist.github.com/dd68fbd5e862f94eb3be
> queries/7.100.query
> https://gist.github.com/d4fd1fb030c6f2b5e678
> queries/7.467.query
> https://gist.github.com/05dbcdc9ee089bd52d0c
>
> On Tue, Mar 10, 2015 at 2:49 PM, Samuel Just  wrote:
>>
>> Yeah, get a ceph pg query on one of the stuck ones.
>> -Sam
>>
>> On Tue, 2015-03-10 at 14:41 +, joel.merr...@gmail.com wrote:
>>>
>>> Stuck unclean and stuck inactive. I can fire up a full query and
>>> health dump somewhere useful if you want (full pg query info on ones
>>> listed in health detail, tree, osd dump etc). There were blocked_by
>>> operations that no longer exist after doing the OSD addition.
>>>
>>> Side note, spent some time yesterday writing some bash to do this
>>> programatically (might be useful to others, will throw on github)
>>>
>>> On Tue, Mar 10, 2015 at 1:41 PM, Samuel Just 
>>> wrote:

 What do you mean by "unblocked" but still "stuck"?
 -Sam

 On Mon, 2015-03-09 at 22:54 +, joel.merr...@gmail.com wrote:
>
> On Mon, Mar 9, 2015 at 2:28 PM, Samuel Just 
> wrote:
>>
>> You'll probably have to recreate osds with the same ids (empty
>> ones),
>> let them boot, stop them, and mark them lost.  There is a feature
>> in
>> the
>> tracker to improve this behavior:
>> http://tracker.ceph.com/issues/10976
>> -Sam
>
> Thanks Sam, I've readded the OSDs, they became unblocked but there
> are
> still the same number of pgs stuck. I looked at them in some more
> detail and it seems they all have num_bytes='0'. Tried a repair
> too,
> for good measure. Still nothing I'm afraid.
>
> Does this mean some underlying catastrophe has happened and they
> are
> never going to recover? Following on, would that cause data loss.
> There are no missing objects and I'm hoping there's appropriate
> checksumming / replicas to balance that out, but now I'm not so
> sure.
>
> Thanks again,
> Joel


>>>
>
> --
> $ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'



>>
>>
>



-- 
$ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Doesn't Support Qcow2 Disk images

[Lindsay Mathieson]

On Thu, 12 Mar 2015 09:27:43 AM Andrija Panic wrote:
> ceph is RAW format - should be all fine...so VM will be using that RAW
> format


If you use cephfs you can use qcow2.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Doesn't Support Qcow2 Disk images

[Vieresjoki, Juha]

But there's really no point, block storage is the only viable option for
virtual machines performance-wise. With images you're dealing with multiple
filesystem layers on top of the actual block devices, plus Ceph as block
storage supports pretty much everything that qcow2 as a format does.

On 12 March 2015 at 12:18, Lindsay Mathieson 
wrote:

> On Thu, 12 Mar 2015 09:27:43 AM Andrija Panic wrote:
> > ceph is RAW format - should be all fine...so VM will be using that RAW
> > format
>
>
> If you use cephfs you can use qcow2.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Doesn't Support Qcow2 Disk images

[Thomas Foster]

http://docs.openstack.org/image-guide/content/ch_converting.html
On Mar 12, 2015 6:50 AM, "Vieresjoki, Juha"  wrote:

> But there's really no point, block storage is the only viable option for
> virtual machines performance-wise. With images you're dealing with multiple
> filesystem layers on top of the actual block devices, plus Ceph as block
> storage supports pretty much everything that qcow2 as a format does.
>
> On 12 March 2015 at 12:18, Lindsay Mathieson 
> wrote:
>
>> On Thu, 12 Mar 2015 09:27:43 AM Andrija Panic wrote:
>> > ceph is RAW format - should be all fine...so VM will be using that RAW
>> > format
>>
>>
>> If you use cephfs you can use qcow2.
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Doesn't Support Qcow2 Disk images

[Lindsay Mathieson]

On Thu, 12 Mar 2015 12:49:51 PM Vieresjoki, Juha wrote:
> But there's really no point, block storage is the only viable option for
> virtual machines performance-wise. With images you're dealing with multiple
> filesystem layers on top of the actual block devices, plus Ceph as block
> storage supports pretty much everything that qcow2 as a format does.


Not much difference that I've noticed on the small system I run, in fact cepfs 
seems to do some caching which speeds up things considerably in benchmarks. 
Not a huge difference in actual app performance that I've noticed.

Delete/Restore of snapshots is a lot quicker with qcow2 and for somereasons 
saving memory state is orders of magnitude quicker with qcow2.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] problem with rbd map

[Marc Boisis]

I’m trying to create my first ceph disk from a client named bjorn :

[ceph@bjorn ~]$ rbd create foo --size 512000 -m helga -k 
/etc/ceph/ceph.client.admin.keyring
[ceph@bjorn ~]$ sudo rbd map foo --pool pool_ulr_1 --name client.admin -m 
helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring
rbd: sysfs write failed
rbd: map failed: (2) No such file or directory

Can you help me to find the problem ?




[ceph@bjorn ~]$ ceph -v
ceph version 0.87.1 (283c2e7cfa2457799f534744d7d549f83ea1335e)

[ceph@bjorn ~]$ ceph -s
cluster cd7dd0a4-075c-4317-8aed-0758085ea9d2
 health HEALTH_OK
 monmap e5: 5 mons at 
{borg=10.10.10.58:6789/0,floki=10.10.10.57:6789/0,helga=10.10.10.64:6789/0,horik=10.10.10.60:6789/0,siggy=10.10.10.59:6789/0},
 election epoch 88, quorum 0,1,2,3,4 floki,borg,siggy,horik,helga
 osdmap e732: 60 osds: 60 up, 60 in
  pgmap v2352: 4160 pgs, 2 pools, 131 bytes data, 2 objects
4145 MB used, 218 TB / 218 TB avail
4160 active+clean

[ceph@bjorn ~]$ lsmod | grep rbd
rbd73133  0 
libceph   235953  1 rbd


with strace: 

open("/sys/bus/rbd/add_single_major", O_WRONLY) = 4
write(4, "10.10.10.64:6789 name=admin,key="..., 61) = -1 ENOENT (No such file 
or directory)
close(4)= 0
write(2, "rbd: sysfs write failed", 23rbd: sysfs write failed) = 23
write(2, "\n", 1
)   = 1
close(3)= 0
write(2, "rbd: map failed: ", 17rbd: map failed: )   = 17
write(2, "(2) No such file or directory", 29(2) No such file or directory) = 29
write(2, "\n", 1
)   = 1
exit_group(2)   = ?
+++ exited with 2 +++



[ceph@bjorn ~]$ ll /sys/bus/rbd/add_single_major
--w--- 1 root root 4096 Mar 12 12:01 /sys/bus/rbd/add_single_major

thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] problem with rbd map

[Ilya Dryomov]

On Thu, Mar 12, 2015 at 3:33 PM, Marc Boisis  wrote:
> I’m trying to create my first ceph disk from a client named bjorn :
>
> [ceph@bjorn ~]$ rbd create foo --size 512000 -m helga -k 
> /etc/ceph/ceph.client.admin.keyring
> [ceph@bjorn ~]$ sudo rbd map foo --pool pool_ulr_1 --name client.admin -m 
> helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring
> rbd: sysfs write failed
> rbd: map failed: (2) No such file or directory
>
> Can you help me to find the problem ?

Which kernel is this?  Is there anything in dmesg?

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] problem with rbd map

[Marc Boisis]

In dmesg:
[ 5981.113104] libceph: client14929 fsid cd7dd0a4-075c-4317-8aed-0758085ea9d2
[ 5981.115853] libceph: mon0 10.10.10.64:6789 session established

My systems are RHEL 7 with 3.10.0-229.el7.x86_64 kernel



> 
> On Thu, Mar 12, 2015 at 3:33 PM, Marc Boisis  wrote:
>> I’m trying to create my first ceph disk from a client named bjorn :
>> 
>> [ceph@bjorn ~]$ rbd create foo --size 512000 -m helga -k 
>> /etc/ceph/ceph.client.admin.keyring
>> [ceph@bjorn ~]$ sudo rbd map foo --pool pool_ulr_1 --name client.admin -m 
>> helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring
>> rbd: sysfs write failed
>> rbd: map failed: (2) No such file or directory
>> 
>> Can you help me to find the problem ?
> 
> Which kernel is this?  Is there anything in dmesg?
> 
> Thanks,
> 
>Ilya

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] problem with rbd map

[Ilya Dryomov]

On Thu, Mar 12, 2015 at 3:33 PM, Marc Boisis  wrote:
> I’m trying to create my first ceph disk from a client named bjorn :
>
> [ceph@bjorn ~]$ rbd create foo --size 512000 -m helga -k 
> /etc/ceph/ceph.client.admin.keyring
> [ceph@bjorn ~]$ sudo rbd map foo --pool pool_ulr_1 --name client.admin -m 
> helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring
> rbd: sysfs write failed
> rbd: map failed: (2) No such file or directory
>
> Can you help me to find the problem ?

Ah, you are creating an image an a standard pool ("rbd"), but trying to
map from a custom pool ("pool_ulr_1") - hence the -ENOENT.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly, cephfs issues: different unix rights depending on the client and ls are slow

[Yan, Zheng]

On Sun, Mar 8, 2015 at 9:21 AM, Francois Lafont  wrote:
> Hello,
>
> Thanks to Jcsp (John Spray I guess) that helps me on IRC.
>
> On 06/03/2015 04:04, Francois Lafont wrote:
>
>>> ~# mkdir /cephfs
>>> ~# mount -t ceph 10.0.2.150,10.0.2.151,10.0.2.152:/ /cephfs/ -o 
>>> name=cephfs,secretfile=/etc/ceph/ceph.client.cephfs.secret
>>>
>>> Then in ceph-testfs, I do:
>>>
>>> root@test-cephfs:~# mkdir /cephfs/d1
>>> root@test-cephfs:~# ll /cephfs/
>>> total 4
>>> drwxr-xr-x  1 root root0 Mar  4 11:45 ./
>>> drwxr-xr-x 24 root root 4096 Mar  4 11:42 ../
>>> drwxr-xr-x  1 root root0 Mar  4 11:45 d1/
>>>
>>> After, in test-cephfs2, I do:
>>>
>>> root@test-cephfs2:~# ll /cephfs/
>>> total 4
>>> drwxr-xr-x  1 root root0 Mar  4 11:45 ./
>>> drwxr-xr-x 24 root root 4096 Mar  4 11:42 ../
>>> drwxrwxrwx  1 root root0 Mar  4 11:45 d1/
>>>
>>> 1) Why the unix rights of d1/ are different when I'm in test-cephfs
>>> and when I'm in test-cephfs2? It should be the same, isn't it?
>
> In fact, this problem is (maybe) a bug in the Linux kernel ceph
> client version "3.16". Indeed, if I mount the cephfs with ceph-fuse
> on the 2 client nodes, the problem doesn't happen and if I mount the
> cephfs with "mount.ceph" but with the Linux kernel 3.13, the problem
> doesn't happen too. I have made a bug report here :
>
> http://tracker.ceph.com/issues/11059
>

It's a bug in ACL code, I have updated http://tracker.ceph.com/issues/11059

>>> 2) If I create 100 files in /cephfs/d1/ with test-cephfs:
>>>
>>> for i in $(seq 100)
>>> do
>>> echo "$(date +%s.%N)" >/cephfs/d1/f_$i
>>> done
>>>
>>> sometimes, in test-cephfs2, when I do a simple:
>>>
>>> root@test-cephfs2:~# time \ls -la /cephfs
>>
>> Sorry error of copy and paste, of course it was:
>>
>> root@test-cephfs2:~# time \ls -la /cephfs/d1/
>>
>>> the command can take 2 or 3 seconds which seems to me very long
>>> for a directory with just 100 files. Generally, if I repeat the
>>> command on test-cephfs2 just after, it's immediate but not always.
>>> I can not reproduce the problem in a determinist way. Sometimes,
>>> to reproduce the problem, I must remove all the files in /cephfs/
>>> on test-cepfs and recreate them. It's very strange. Sometimes and
>>> randomly, something seems to be stalled but I don't know what. I
>>> suspect a problem of mds tuning but, In fact, I don't know what
>>> to do.
>>
>> I have the same problem with hammer too.
>> But someone can confirm me that 3s (not always) for "ls -la" in
>> a cephfs directory which contains 100 file it's pathological? After
>> all, maybe is it normal? I don't have much experience with cephfs.
>
> In fact, according to what I was told on IRC, a such time for the
> "ls -la" command could be normal because the client node requests
> a "stat" for each file in the directory and each "stat" can take
> a little time. But I'm still a little puzzled. Indeed the first
> "ls -la" can take 2 or 3 seconds and the next "ls -la" is usually
> faster but not always. Sometimes (it's very random), the second
> "ls -la", even the third "ls -la" etc. can be very slow. I admit
> that after a number of attempts, "ls -la" becomes faster but not
> always from the second attempt.
>
> I'm still surprised by such times. For instance, It seems to me
> that, with a mounted nfs share, commands like "ls -la" are very
> fast in comparison (with a directory which contains the same number
> of files). Can anyone explain to me why there is a such difference
> between the nfs case and the cephfs case? This is absolutely not a
> criticism but it's just to understand the concepts that come into
> play. In the case of "ls -al" ie just reading (it is assumed that
> there is no writing on the directory), the nfs and the cephfs cases
> seem to me very similar: the client just requests a stat on each file
> in the directory. Am I wrong?

NFS has no cache coherence mechanism. It can't guarantee one client always
see other client's change.

The time variation is caused cache coherence. when client has valid information
in its cache, 'stat' operation will be fast. Otherwise the client need
to send request
to MDS and wait for reply, which will be slow.


>
> --
> François Lafont
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] problem with rbd map

[Marc Boisis]

Thanks a lot it’s good

ROOT:bjorn:/root > rbd create foo --pool pool_ulr_1 --size 512000 -m 
helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring
ROOT:bjorn:/root > rbd map foo --pool pool_ulr_1 --name client.admin -m 
helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring
/dev/rbd0
ROOT:bjorn:/root > 


> Le 12 mars 2015 à 13:42, Ilya Dryomov  a écrit :
> 
> On Thu, Mar 12, 2015 at 3:33 PM, Marc Boisis  wrote:
>> I’m trying to create my first ceph disk from a client named bjorn :
>> 
>> [ceph@bjorn ~]$ rbd create foo --size 512000 -m helga -k 
>> /etc/ceph/ceph.client.admin.keyring
>> [ceph@bjorn ~]$ sudo rbd map foo --pool pool_ulr_1 --name client.admin -m 
>> helga.univ-lr.fr -k /etc/ceph/ceph.client.admin.keyring
>> rbd: sysfs write failed
>> rbd: map failed: (2) No such file or directory
>> 
>> Can you help me to find the problem ?
> 
> Ah, you are creating an image an a standard pool ("rbd"), but trying to
> map from a custom pool ("pool_ulr_1") - hence the -ENOENT.
> 
> Thanks,
> 
>Ilya

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Doesn't Support Qcow2 Disk images

[Wido den Hollander]

On 12-03-15 13:00, Lindsay Mathieson wrote:
> On Thu, 12 Mar 2015 12:49:51 PM Vieresjoki, Juha wrote:
>> But there's really no point, block storage is the only viable option for
>> virtual machines performance-wise. With images you're dealing with multiple
>> filesystem layers on top of the actual block devices, plus Ceph as block
>> storage supports pretty much everything that qcow2 as a format does.
> 
> 
> Not much difference that I've noticed on the small system I run, in fact 
> cepfs 
> seems to do some caching which speeds up things considerably in benchmarks. 
> Not a huge difference in actual app performance that I've noticed.
> 
> Delete/Restore of snapshots is a lot quicker with qcow2 and for somereasons 
> saving memory state is orders of magnitude quicker with qcow2.

Don't, simply don't run your VMs with QCOW2 over CephFS. You bring in so
much more complexity by adding a Posix Filesystem and the Ceph MDS.

Although CephFS might look faster due to the kernel caching, it's not safer!

Use RBD with Qemu, it keeps it all in userspace, a lot better then going
through all the kernelspace things.

Wido

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

[Malcolm Haak]

I've no idea if this helps. But I was looking in the meta file of osd.3 to see 
if things there made any sense.  I'm very much out of my depth.

To me this looks like a bug. Quite possibly a corner case, but bug none the 
less.

Anyway I've included my crush map and what look like the osdmap files out of 
the osd that wont start.

Cracking them open it appears that the new osd.3 is not in the map at all.. 
which might be correct, but I would have expected to see it in the layout. 

I've also added the current osdmap dump as well... 


If I'm asking in the wrong place, please let me know. I don't want to be 
wasting peoples time. 

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Malcolm Haak
Sent: Thursday, 12 March 2015 4:16 PM
To: Samuel Just; jl...@redhat.com
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

Sorry about all the unrelated grep issues..

So I've rebuilt and reinstalled and it's still broken. 

On the working node, even with the new packages, everything works.
On the new broken node, I've added a mon and it works. But I still cannot start 
an OSD on the new node.

What else do you need from me? I'll get logs run any number of tests.

I've got data in this cluster already, and it's full so I need to expand it, 
I've already got hardware.

Thanks in advance for even having a look


-Original Message-
From: Samuel Just [mailto:sj...@redhat.com] 
Sent: Wednesday, 11 March 2015 1:41 AM
To: Malcolm Haak; jl...@redhat.com
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

Joao, it looks like map 2759 is causing trouble, how would he get the
full and incremental maps for that out of the mons?
-Sam

On Tue, 2015-03-10 at 14:12 +, Malcolm Haak wrote:
> Hi Samuel,
> 
> The sha1? I'm going to admit ignorance as to what you are looking for. They 
> are all running the same release if that is what you are asking. 
> Same tarball built into rpms using rpmbuild on both nodes... 
> Only difference being that the other node has been upgraded and the problem 
> node is fresh.
> 
> added the requested config here is the command line output
> 
> microserver-1:/etc # /etc/init.d/ceph start osd.3
> === osd.3 === 
> Mounting xfs on microserver-1:/var/lib/ceph/osd/ceph-3
> 2015-03-11 01:00:13.492279 7f05b2f72700  1 -- :/0 messenger.start
> 2015-03-11 01:00:13.492823 7f05b2f72700  1 -- :/1002795 --> 
> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 
> 0x7f05ac0290b0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.510814 7f05b07ef700  1 -- 192.168.0.250:0/1002795 learned 
> my addr 192.168.0.250:0/1002795
> 2015-03-11 01:00:13.527653 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== 
> mon.0 192.168.0.10:6789/0 1  mon_map magic: 0 v1  191+0+0 (1112175541 
> 0 0) 0x7f05aab0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.527899 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== 
> mon.0 192.168.0.10:6789/0 2  auth_reply(proto 1 0 (0) Success) v1  
> 24+0+0 (3859410672 0 0) 0x7f05ae70 con 0x7f05ac027c40
> 2015-03-11 01:00:13.527973 7f05abfff700  1 -- 192.168.0.250:0/1002795 --> 
> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f05ac029730 
> con 0x7f05ac027c40
> 2015-03-11 01:00:13.528124 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 
> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 
> 0x7f05ac029a50 con 0x7f05ac027c40
> 2015-03-11 01:00:13.528265 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 
> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 
> 0x7f05ac029f20 con 0x7f05ac027c40
> 2015-03-11 01:00:13.530359 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== 
> mon.0 192.168.0.10:6789/0 3  mon_map magic: 0 v1  191+0+0 (1112175541 
> 0 0) 0x7f05aab0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.530548 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== 
> mon.0 192.168.0.10:6789/0 4  mon_subscribe_ack(300s) v1  20+0+0 
> (3648139960 0 0) 0x7f05afb0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.531114 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== 
> mon.0 192.168.0.10:6789/0 5  osd_map(3277..3277 src has 2757..3277) v3 
>  5366+0+0 (3110999244 0 0) 0x7f05a0002800 con 0x7f05ac027c40
> 2015-03-11 01:00:13.531772 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== 
> mon.0 192.168.0.10:6789/0 6  mon_subscribe_ack(300s) v1  20+0+0 
> (3648139960 0 0) 0x7f05afb0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.532186 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== 
> mon.0 192.168.0.10:6789/0 7  osd_map(3277..3277 src has 2757..3277) v3 
>  5366+0+0 (3110999244 0 0) 0x7f05a0001250 con 0x7f05ac027c40
> 2015-03-11 01:00:13.532260 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== 
> mon.0 192.168.0.10:6789/0 8  mon_subscribe_ack(300s) v1  20+0+0 
> (3648139960 0 0) 0x7f05afb0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.556748 7f05b2f72700  1 

Re: [ceph-users] Add monitor unsuccesful

[Robert LeBlanc]

If I remember right, the mon key has to be the same between all the mon
hosts. I don't think I added an admin key to my second mon, it got all the
other keys once it joined the mon closure. I do remember the join taking a
while. Have you checked the firewall to make sure traffic is allowed? I
don't remember if you said you checked it.

Robert LeBlanc

Sent from a mobile device please excuse any typos.
On Mar 11, 2015 8:08 PM, "Jesus Chavez (jeschave)" 
wrote:

>  Thanks Steffen I have followed everything not sure what is going on, the
> mon keyring and client admin are individual? Per mon host? Or do I need to
> copy from the first initial mon node?
>
>  Thanks again!
>
>
> * Jesus Chavez*
> SYSTEMS ENGINEER-C.SALES
>
> jesch...@cisco.com
> Phone: *+52 55 5267 3146 <+52%2055%205267%203146>*
> Mobile: *+51 1 5538883255 <+51%201%205538883255>*
>
> CCIE - 44433
>
> On Mar 11, 2015, at 6:28 PM, Steffen W Sørensen  wrote:
>
>
> On 12/03/2015, at 00.55, Jesus Chavez (jeschave) 
> wrote:
>
> can anybody tell me a good blog link that explain how to add monitor? I
> have tried manually and also with ceph-deploy without success =(
>
> Dunno if these might help U:
>
>
> http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#adding-a-monitor-manual
>
> http://cephnotes.ksperis.com/blog/2013/08/29/mon-failed-to-start
>
> /Steffen
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Replication question

[Thomas Foster]

I am looking into how I can maximize my space with replication, and I am
trying to understand how I can do that.

I have 145TB of space and a replication of 3 for the pool and was thinking
that the max data I can have in the cluster is ~47TB in my cluster at one
time..is that correct?  Or is there a way to get more data into the cluster
with less space using erasure coding?

Any help would be greatly appreciated.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

[Joao Eduardo Luis]

On 03/12/2015 05:16 AM, Malcolm Haak wrote:

Sorry about all the unrelated grep issues..

So I've rebuilt and reinstalled and it's still broken.

On the working node, even with the new packages, everything works.
On the new broken node, I've added a mon and it works. But I still cannot start 
an OSD on the new node.

What else do you need from me? I'll get logs run any number of tests.

I've got data in this cluster already, and it's full so I need to expand it, 
I've already got hardware.

Thanks in advance for even having a look


Sam mentioned to me on IRC that the next step would be to grab the 
offending osdmaps.  Easiest way for that will be to stop a monitor and 
run 'ceph-monstore-tool' in order to obtain the full maps, and then use 
'ceph-kvstore-tool' to obtain incrementals.


Given the osd is crashing on version 2759, the following would be best:

(Assuming you have stopped a given monitor with id FOO, whose store is 
sitting at default path /var/lib/ceph/mon/ceph-FOO)


ceph-monstore-tool /var/lib/ceph/mon/ceph-FOO get osdmap -- --version 
2758 --out /tmp/osdmap.full.2758


ceph-monstore-tool /var/lib/ceph/mon/ceph-FOO get osdmap -- --version 
2759 --out /tmp/osdmap.full.2759


(please note the '--' between 'osdmap' and '--version', as that is 
required for the tool to do its thing)


and then

ceph-kvstore-tool /var/lib/ceph/mon/ceph-FOO/store.db get osdmap 2758 
out /tmp/osdmap.inc.2758


ceph-kvstore-tool /var/lib/ceph/mon/ceph-FOO/store.db get osdmap 2759 
out /tmp/osdmap.inc.2759


Cheers!

  -Joao





-Original Message-
From: Samuel Just [mailto:sj...@redhat.com]
Sent: Wednesday, 11 March 2015 1:41 AM
To: Malcolm Haak; jl...@redhat.com
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

Joao, it looks like map 2759 is causing trouble, how would he get the
full and incremental maps for that out of the mons?
-Sam

On Tue, 2015-03-10 at 14:12 +, Malcolm Haak wrote:

Hi Samuel,

The sha1? I'm going to admit ignorance as to what you are looking for. They are 
all running the same release if that is what you are asking.
Same tarball built into rpms using rpmbuild on both nodes...
Only difference being that the other node has been upgraded and the problem 
node is fresh.

added the requested config here is the command line output

microserver-1:/etc # /etc/init.d/ceph start osd.3
=== osd.3 ===
Mounting xfs on microserver-1:/var/lib/ceph/osd/ceph-3
2015-03-11 01:00:13.492279 7f05b2f72700  1 -- :/0 messenger.start
2015-03-11 01:00:13.492823 7f05b2f72700  1 -- :/1002795 --> 192.168.0.10:6789/0 
-- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x7f05ac0290b0 con 0x7f05ac027c40
2015-03-11 01:00:13.510814 7f05b07ef700  1 -- 192.168.0.250:0/1002795 learned 
my addr 192.168.0.250:0/1002795
2015-03-11 01:00:13.527653 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 
192.168.0.10:6789/0 1  mon_map magic: 0 v1  191+0+0 (1112175541 0 0) 
0x7f05aab0 con 0x7f05ac027c40
2015-03-11 01:00:13.527899 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 
192.168.0.10:6789/0 2  auth_reply(proto 1 0 (0) Success) v1  24+0+0 
(3859410672 0 0) 0x7f05ae70 con 0x7f05ac027c40
2015-03-11 01:00:13.527973 7f05abfff700  1 -- 192.168.0.250:0/1002795 --> 
192.168.0.10:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f05ac029730 con 
0x7f05ac027c40
2015-03-11 01:00:13.528124 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 
192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 
0x7f05ac029a50 con 0x7f05ac027c40
2015-03-11 01:00:13.528265 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 
192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 
0x7f05ac029f20 con 0x7f05ac027c40
2015-03-11 01:00:13.530359 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 
192.168.0.10:6789/0 3  mon_map magic: 0 v1  191+0+0 (1112175541 0 0) 
0x7f05aab0 con 0x7f05ac027c40
2015-03-11 01:00:13.530548 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 
192.168.0.10:6789/0 4  mon_subscribe_ack(300s) v1  20+0+0 (3648139960 0 0) 
0x7f05afb0 con 0x7f05ac027c40
2015-03-11 01:00:13.531114 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 
192.168.0.10:6789/0 5  osd_map(3277..3277 src has 2757..3277) v3  5366+0+0 
(3110999244 0 0) 0x7f05a0002800 con 0x7f05ac027c40
2015-03-11 01:00:13.531772 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 
192.168.0.10:6789/0 6  mon_subscribe_ack(300s) v1  20+0+0 (3648139960 0 0) 
0x7f05afb0 con 0x7f05ac027c40
2015-03-11 01:00:13.532186 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 
192.168.0.10:6789/0 7  osd_map(3277..3277 src has 2757..3277) v3  5366+0+0 
(3110999244 0 0) 0x7f05a0001250 con 0x7f05ac027c40
2015-03-11 01:00:13.532260 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 
192.168.0.10:6789/0 8  mon_subscribe_ack(300s) v1  20+0+0 (3648139960 0 0) 
0x7f05afb0 con 0x7f05ac027c40
2015-03-11 01:00:13.556748 7f05b2f727

Re: [ceph-users] Replication question

[Charles 'Boyo]

Hello,

On Thu, Mar 12, 2015 at 3:07 PM, Thomas Foster 
wrote:

> I am looking into how I can maximize my space with replication, and I am
> trying to understand how I can do that.
>
> I have 145TB of space and a replication of 3 for the pool and was thinking
> that the max data I can have in the cluster is ~47TB in my cluster at one
> time..is that correct?  Or is there a way to get more data into the cluster
> with less space using erasure coding?
>
>
Erasure coding would be the right way to do this.

See http://ceph.com/docs/master/rados/operations/erasure-code/

Charles
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Replication question

[Steve Anthony]

Actually, it's more like 41TB. It's a bad idea to run at near full
capacity (by default past 85%) because you need some space where Ceph
can replicate data as part of its healing process in the event of disk
or node failure. You'll get a health warning when you exceed this ratio.

You can use erasure coding to increase the amount of data you can store
beyond 41TB, but you'll still need some replicated disk as a caching
layer in front of the erasure coded pool if you're using RBD. See:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-December/036430.html

As to how much space you can save with erasure coding, that will depend
on if you're using RBD and need a cache layer and the values you set for
k and m (number of data chunks and coding chunks). There's been some
discussion on the list with regards to choosing those values.

-Steve

On 03/12/2015 10:07 AM, Thomas Foster wrote:
> I am looking into how I can maximize my space with replication, and I
> am trying to understand how I can do that.
>
> I have 145TB of space and a replication of 3 for the pool and was
> thinking that the max data I can have in the cluster is ~47TB in my
> cluster at one time..is that correct?  Or is there a way to get more
> data into the cluster with less space using erasure coding?  
>
> Any help would be greatly appreciated.
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Steve Anthony
LTS HPC Support Specialist
Lehigh University
sma...@lehigh.edu



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Replication question

[Thomas Foster]

Thank you!  That helps alot.
On Mar 12, 2015 10:40 AM, "Steve Anthony"  wrote:

>  Actually, it's more like 41TB. It's a bad idea to run at near full
> capacity (by default past 85%) because you need some space where Ceph can
> replicate data as part of its healing process in the event of disk or node
> failure. You'll get a health warning when you exceed this ratio.
>
> You can use erasure coding to increase the amount of data you can store
> beyond 41TB, but you'll still need some replicated disk as a caching layer
> in front of the erasure coded pool if you're using RBD. See:
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-December/036430.html
>
> As to how much space you can save with erasure coding, that will depend on
> if you're using RBD and need a cache layer and the values you set for k and
> m (number of data chunks and coding chunks). There's been some discussion
> on the list with regards to choosing those values.
>
> -Steve
>
> On 03/12/2015 10:07 AM, Thomas Foster wrote:
>
> I am looking into how I can maximize my space with replication, and I am
> trying to understand how I can do that.
>
>  I have 145TB of space and a replication of 3 for the pool and was
> thinking that the max data I can have in the cluster is ~47TB in my cluster
> at one time..is that correct?  Or is there a way to get more data into the
> cluster with less space using erasure coding?
>
>  Any help would be greatly appreciated.
>
>
>
>
> ___
> ceph-users mailing 
> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> --
> Steve Anthony
> LTS HPC Support Specialist
> Lehigh universitysma...@lehigh.edu
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Shadow files

[Yehuda Sadeh-Weinraub]

- Original Message -
> From: "Ben" 
> To: ceph-us...@ceph.com
> Sent: Wednesday, March 11, 2015 8:46:25 PM
> Subject: Re: [ceph-users] Shadow files
> 
> Anyone got any info on this?
> 
> Is it safe to delete shadow files?

It depends. Shadow files are badly named objects that represent part of the 
objects data. They are only safe to remove if you know that the corresponding 
objects no longer exist.

Yehuda

> 
> On 2015-03-11 10:03, Ben wrote:
> > We have a large number of shadow files in our cluster that aren't
> > being deleted automatically as data is deleted.
> > 
> > Is it safe to delete these files?
> > Is there something we need to be aware of when deleting them?
> > Is there a script that we can run that will delete these safely?
> > 
> > Is there something wrong with our cluster that it isn't deleting these
> > files when it should be?
> > 
> > We are using civetweb with radosgw, with tengine ssl proxy infront of
> > it
> > 
> > Any advice please
> > Thanks
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Sparse RBD instance snapshots in OpenStack

[Sebastien Han]

Several patches aim to solve that by using RBD snapshots instead of QEMU 
snapshots.
Unfortunately I doubt we will have something ready for OpenStack Juno.
Hopefully Liberty will be the release that fixes that.

Having RAW images is not that bad since booting from that snapshot will do a 
clone.
So not sure if doing sparsify a good idea (libguestfs should be able to do 
that).
However it’s better we could do that via RBD snapshots so we can have best of 
both worlds.

> On 12 Mar 2015, at 03:45, Charles 'Boyo  wrote:
> 
> Hello all.
> 
> The current behavior of snapshotting instances RBD-backed in OpenStack 
> involves uploading the snapshot into Glance.
> 
> The resulting Glance image is fully allocated, causing an explosion of 
> originally sparse RAW images. Is there a way to preserve the sparseness? Else 
> I can use qemu-img convert (or rbd export/import) to manually sparsify it?
> 
> On a related note, my Glance is also backed by the same Ceph cluster, in 
> another pool and I was wondering if Ceph snapshots would not be a better way 
> to do this. Any ideas?
> 
> Regards,
> 
> Charles
> 


Cheers.
 
Sébastien Han 
Cloud Architect 

"Always give 100%. Unless you're giving blood."

Phone: +33 (0)1 49 70 99 72 
Mail: sebastien@enovance.com 
Address : 11 bis, rue Roquépine - 75008 Paris
Web : www.enovance.com - Twitter : @enovance 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Replication question

[Kamil Kuramshin]

For example, here is my confuguration:

superuser@admin:~$ ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
242T  209T   20783G  8.38
POOLS:
NAME  ID USED  %USED MAX AVAIL OBJECTS
ec_backup-storage 4  9629G  3.88 137T 2465171
cache 5   136G  0.06 38393M   35036
block-devices 6  1953G  0.79 70202G  500060


*ec_backup-storage* - is Erasure Encoded pool, k=2, m=1 (default)
*cache* - is replicated pool consisting dedicated 12xSSDx60Gb disks, 
replica size=3, used as cache tier for EC pool
*block-devices* - is replicated pool, replica size=3, using same OSD's 
that inErasure Encoded pool


On*'**MAX AVAIL**'* column you can see that EC pool currently has 
*137Tb* of free space, but in same time if we will write to replicated 
pool there is only *70Tb, *but *both* pools are on the *same* *OSD's. 
*So using EC pool saves 2**times more effective space in my case!


12.03.2015 17:50, Thomas Foster пишет:


Thank you!  That helps alot.

On Mar 12, 2015 10:40 AM, "Steve Anthony" > wrote:


Actually, it's more like 41TB. It's a bad idea to run at near full
capacity (by default past 85%) because you need some space where
Ceph can replicate data as part of its healing process in the
event of disk or node failure. You'll get a health warning when
you exceed this ratio.

You can use erasure coding to increase the amount of data you can
store beyond 41TB, but you'll still need some replicated disk as a
caching layer in front of the erasure coded pool if you're using
RBD. See:

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-December/036430.html

As to how much space you can save with erasure coding, that will
depend on if you're using RBD and need a cache layer and the
values you set for k and m (number of data chunks and coding
chunks). There's been some discussion on the list with regards to
choosing those values.

-Steve

On 03/12/2015 10:07 AM, Thomas Foster wrote:

I am looking into how I can maximize my space with replication,
and I am trying to understand how I can do that.

I have 145TB of space and a replication of 3 for the pool and was
thinking that the max data I can have in the cluster is ~47TB in
my cluster at one time..is that correct?  Or is there a way to
get more data into the cluster with less space using erasure coding?

Any help would be greatly appreciated.




___
ceph-users mailing list
ceph-users@lists.ceph.com  
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


-- 
Steve Anthony

LTS HPC Support Specialist
Lehigh University
sma...@lehigh.edu  


___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Jesus Chavez (jeschave)]

Hi Robert yes I did disable completely actually with chkconfig off for not take 
the service up when booting, I have 2 networks 1 with internet for yum purposes 
and the network for the public network so before any configuration I specified 
on ceph.conf that public network but I am not sure if it is the cause for 
something...
The thing is that I am not sure about which steps should run in the new monitor 
host and which one should I run in the initial monitor, it seems like step 3 
and 4 that is generate keyring and mapping should be done in initial monitor 
server and also step 5 that is mkfs becuase if I try to run those steps in the 
new monitor host didnt work cause cant find the keys :(


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 7:54 AM, Robert LeBlanc 
mailto:rob...@leblancnet.us>> wrote:


If I remember right, the mon key has to be the same between all the mon hosts. 
I don't think I added an admin key to my second mon, it got all the other keys 
once it joined the mon closure. I do remember the join taking a while. Have you 
checked the firewall to make sure traffic is allowed? I don't remember if you 
said you checked it.

Robert LeBlanc

Sent from a mobile device please excuse any typos.

On Mar 11, 2015 8:08 PM, "Jesus Chavez (jeschave)" 
mailto:jesch...@cisco.com>> wrote:
Thanks Steffen I have followed everything not sure what is going on, the mon 
keyring and client admin are individual? Per mon host? Or do I need to copy 
from the first initial mon node?

Thanks again!


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 11, 2015, at 6:28 PM, Steffen W Sørensen 
mailto:ste...@me.com>> wrote:


On 12/03/2015, at 00.55, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:

can anybody tell me a good blog link that explain how to add monitor? I have 
tried manually and also with ceph-deploy without success =(
Dunno if these might help U:

http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#adding-a-monitor-manual

http://cephnotes.ksperis.com/blog/2013/08/29/mon-failed-to-start

/Steffen

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Robert LeBlanc]

Here is the procedure I wrote for our internal use (it is still a work in
progress) and may help you:

*Creating the First Monitor*

Once you Ceph installed, DNS and networking configured and have a ceph.conf
file built, you are ready to bootstrap the first monitor. The UUID is the
same from the ceph.conf file generated earlier, cluster-name is the name of
the Ceph cluster, usually just 'ceph', hostname is the short name of the
host and must match `hostname -s`.

1. SSH into the monitor host
2. Create the monitor keyring

ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon.
--cap mon 'allow *'

3. Create the admin keyring

ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring
--gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow
*' --cap mds 'allow'

4. Add the admin key to the monitor keyring so the Admin user can manage
the cluster

ceph-authtool /tmp/ceph.mon.keyring --import-keyring
/etc/ceph/ceph.client.admin.keyring

5. Create the initial monitor map

monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap

6. Create the directory the monitor will store persistent data

sudo mkdir /var/lib/ceph/mon/{cluster-name}-{hostname}

7. Create the file structure for the monitor

ceph-mon --mkfs -i {hostname} --monmap /tmp/monmap --keyring
/tmp/ceph.mon.keyring

8. Let the monitor know that everything is ready to go

sudo touch /var/lib/ceph/mon/{cluster-name}-{hostname}/done

9. Start the Monitor service

sudo start ceph-mon id=node1

10. Set the Monitor service to start at boot

*Adding Additional Monitors*

Adding additional monitors will make the cluster less susceptible to
outages due to hardware running the monitors going offline. As monitors are
added, the load on each monitor can both increase and decrease, so it is
import to understand when a new set of monitors should be added (to
understand why two monitors should be added, please see Monitors). Having
more monitors reduces the number of clients that each monitor has to
service. However, each time a CRUSH map changes (an OSD is marked out, a
new OSD is added, etc), a majority of monitors have to agree on the CRUSH
changes in the PAXOS algorithm and the CRUSH map has to be updated on each
Monitor. If the cluster is experiencing many CRUSH map changes it can add
additional load on the Monitors.

Monitors perform many file sync operations and are sensitive to latencies
with these operations. The large number of file sync operations can be very
disruptive to to OSD processes especially if residing on the same
traditional rotational disk. It is best to have a Monitor on its own
dedicated hardware. If dedicated hardware is not an option, then locate the
Monitor store on an SSD that is not primarily used for an OSD, an SSD
journal would be OK.

The process of adding a new monitor is detailed in (
http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#adding-a-monitor-manual)
and is outlined as follows:

   1. Copy the Monitor key and Monitor map from a running Monitor to the
   new monitor.
   2. Create a monitor directory on the new monitor.
   3. Add the new monitor to the Monitor map.
   4. Start the new monitor.


On Thu, Mar 12, 2015 at 9:58 AM, Jesus Chavez (jeschave)  wrote:

>  Hi Robert yes I did disable completely actually with chkconfig off for
> not take the service up when booting, I have 2 networks 1 with internet for
> yum purposes and the network for the public network so before any
> configuration I specified on ceph.conf that public network but I am not
> sure if it is the cause for something...
> The thing is that I am not sure about which steps should run in the new
> monitor host and which one should I run in the initial monitor, it seems
> like step 3 and 4 that is generate keyring and mapping should be done in
> initial monitor server and also step 5 that is mkfs becuase if I try to run
> those steps in the new monitor host didnt work cause cant find the keys :(
>
>
> * Jesus Chavez*
> SYSTEMS ENGINEER-C.SALES
>
> jesch...@cisco.com
> Phone: *+52 55 5267 3146 <+52%2055%205267%203146>*
> Mobile: *+51 1 5538883255 <+51%201%205538883255>*
>
> CCIE - 44433
>
> On Mar 12, 2015, at 7:54 AM, Robert LeBlanc  wrote:
>
>   If I remember right, the mon key has to be the same between all the mon
> hosts. I don't think I added an admin key to my second mon, it got all the
> other keys once it joined the mon closure. I do remember the join taking a
> while. Have you checked the firewall to make sure traffic is allowed? I
> don't remember if you said you checked it.
>
> Robert LeBlanc
>
> Sent from a mobile device please excuse any typos.
> On Mar 11, 2015 8:08 PM, "Jesus Chavez (jeschave)" 
> wrote:
>
>>  Thanks Steffen I have followed everything not sure what is going on,
>> the mon keyring and client admin are individual? Per mon host? Or do I need
>> to copy from the first initial mon node?
>>
>>  Thanks again!
>>
>>
>> * Jesus Chavez*
>> SYSTEMS ENGINEER-C.SALES
>>
>

Re: [ceph-users] Add monitor unsuccesful

[Jesus Chavez (jeschave)]

Great :) so just 1 point more, step 4 in adding monitors (Add the new 
monitor to the Monitor map.) this command actually runs in the new monitor 
right?

Thank you so much!


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 10:06 AM, Robert LeBlanc 
mailto:rob...@leblancnet.us>> wrote:

Add the new monitor to the Monitor map.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Robert LeBlanc]

That command (ceph mon add  [:]) can be run from any
client in the cluster with the admin key, it is a general Ceph command.

On Thu, Mar 12, 2015 at 10:33 AM, Jesus Chavez (jeschave) <
jesch...@cisco.com> wrote:

>  Great :) so just 1 point more, step 4 in adding monitors (Add the
> new monitor to the Monitor map.) this command actually runs in the new
> monitor right?
>
>  Thank you so much!
>
>
> * Jesus Chavez*
> SYSTEMS ENGINEER-C.SALES
>
> jesch...@cisco.com
> Phone: *+52 55 5267 3146 <+52%2055%205267%203146>*
> Mobile: *+51 1 5538883255 <+51%201%205538883255>*
>
> CCIE - 44433
>
> On Mar 12, 2015, at 10:06 AM, Robert LeBlanc  wrote:
>
>  Add the new monitor to the Monitor map.
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Jesus Chavez (jeschave)]

Thank you Robert! Ill try ;)


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 11:03 AM, Robert LeBlanc 
mailto:rob...@leblancnet.us>> wrote:

That command (ceph mon add  [:]) can be run from any client 
in the cluster with the admin key, it is a general Ceph command.

On Thu, Mar 12, 2015 at 10:33 AM, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:
Great :) so just 1 point more, step 4 in adding monitors (Add the new 
monitor to the Monitor map.) this command actually runs in the new monitor 
right?

Thank you so much!


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 10:06 AM, Robert LeBlanc 
mailto:rob...@leblancnet.us>> wrote:

Add the new monitor to the Monitor map.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Sparse RBD instance snapshots in OpenStack

[Charles 'Boyo]

Thanks for the feedback. I will be looking forward to those patches in Liberty.

In the meanwhile, it appears my best option would be to manually sparsify the 
Glance images using qemu-img convert.

Regards,

Charles

-Original Message-
From: Sebastien Han 
Date: Thu, 12 Mar 2015 10:44:43 
To: 
Cc: 
Subject: Re: Sparse RBD instance snapshots in OpenStack

Several patches aim to solve that by using RBD snapshots instead of QEMU 
snapshots.
Unfortunately I doubt we will have something ready for OpenStack Juno.
Hopefully Liberty will be the release that fixes that.

Having RAW images is not that bad since booting from that snapshot will do a 
clone.
So not sure if doing sparsify a good idea (libguestfs should be able to do 
that).
However it’s better we could do that via RBD snapshots so we can have best of 
both worlds.

> On 12 Mar 2015, at 03:45, Charles 'Boyo  wrote:
> 
> Hello all.
> 
> The current behavior of snapshotting instances RBD-backed in OpenStack 
> involves uploading the snapshot into Glance.
> 
> The resulting Glance image is fully allocated, causing an explosion of 
> originally sparse RAW images. Is there a way to preserve the sparseness? Else 
> I can use qemu-img convert (or rbd export/import) to manually sparsify it?
> 
> On a related note, my Glance is also backed by the same Ceph cluster, in 
> another pool and I was wondering if Ceph snapshots would not be a better way 
> to do this. Any ideas?
> 
> Regards,
> 
> Charles
> 


Cheers.
 
Sébastien Han 
Cloud Architect 

"Always give 100%. Unless you're giving blood."

Phone: +33 (0)1 49 70 99 72 
Mail: sebastien@enovance.com 
Address : 11 bis, rue Roquépine - 75008 Paris
Web : www.enovance.com - Twitter : @enovance 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Jesus Chavez (jeschave)]

Thanks I did everything as you mention and still have the same issue that hangs 
a lot:

[root@tauro ~]# ceph status
2015-03-12 11:40:50.441084 7f6e20336700  0 -- :/1005688 >> 192.168.4.35:6789/0 
pipe(0x7f6e1c0239a0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f6e1c023c30).fault
2015-03-12 11:40:53.441517 7f6e20235700  0 -- :/1005688 >> 192.168.4.35:6789/0 
pipe(0x7f6e1c00 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f6e1e90).fault
2015-03-12 11:40:57.443038 7f6e20336700  0 -- :/1005688 >> 192.168.4.35:6789/0 
pipe(0x7f6e100030e0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f6e10003370).fault
^CError connecting to cluster: InterruptedOrTimeoutError
[root@tauro ~]#

After I saw that my monmap doesn’t have the new entry:

monmaptool: monmap file /tmp/monmap
epoch 0
fsid ddb39f8d-c30f-4172-aeca-30e553a5e4f1
last_changed 2015-03-12 11:22:20.811525
created 2015-03-12 11:22:20.811525
0: 192.168.4.35:6789/0 mon.aries
[root@tauro ~]#


It only has the initial monitor (mon.aries) the new one is called (mon.tauro), 
the last command before “…ceph-tauro/done” was "sudo ceph-mon --cluster ceph 
--mkfs -i tauro --keyring /tmp/ceph.mon.keyring --monmap /tmp/monmap” also I 
have tried to add to monmap with:

[root@tauro ~]# ceph mon add tauro 192.168.4.36:6789
2015-03-12 11:45:26.807504 7fc53077a700  0 -- :/1007053 >> 192.168.4.35:6789/0 
pipe(0x7fc52c023970 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fc52c023c00).fault
2015-03-12 11:45:29.807800 7fc530679700  0 -- :/1007053 >> 192.168.4.35:6789/0 
pipe(0x7fc52c00 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fc52e90).fault
^CError connecting to cluster: InterruptedOrTimeoutError

and also I tried from the initial monitor:

[root@aries ~]#  ceph mon add tauro 192.168.4.36:6789
2015-03-12 11:46:21.561523 7f7432ffd700  0 monclient: hunting for new mon


and it hangs forever, I tried at the same time to get the service up on tauro 
(new mon):


[root@tauro ~]# sudo /etc/init.d/ceph start mon.tauro
=== mon.tauro ===
Starting Ceph mon.tauro on tauro...
Running as unit run-7463.service.
Starting ceph-create-keys on tauro...
[root@tauro ~]#


and of course still have nothing =(


those were my entire lines:

vi /etc/ceph/ceph.conf

i[global]
fsid = ddb39f8d-c30f-4172-aeca-30e553a5e4f1
mon initial members = aries
mon host = 192.168.4.35

ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap 
mon 'allow *'
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n 
client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 
'allow'
ceph-authtool /tmp/ceph.mon.keyring --import-keyring 
/etc/ceph/ceph.client.admin.keyring
monmaptool --create --add aries 192.168.4.35 --fsid 
ddb39f8d-c30f-4172-aeca-30e553a5e4f1 /tmp/monmap
sudo mkdir /var/lib/ceph/mon/ceph-aries
ceph-mon --mkfs -i aries --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
sudo touch /var/lib/ceph/mon/ceph-aries/done
sudo /etc/init.d/ceph start mon.aries

scp /etc/ceph/ceph.conf root@tauro:/etc/ceph/ceph.conf
scp /tmp/ceph.mon.keyring root@tauro:/tmp/ceph.mon.keyring
scp /tmp/monmap root@tauro:/tmp/monmap

mkdir /var/lib/ceph/mon/ceph-tauro
sudo ceph-mon --cluster ceph --mkfs -i tauro --keyring /tmp/ceph.mon.keyring 
--monmap /tmp/monmap
sudo ceph-mon -i tauro --mkfs --monmap /tmp/monmap --keyring 
/tmp/ceph.mon.keyring
sudo touch /var/lib/ceph/mon/ceph-tauro/done
sudo /etc/init.d/ceph start mon.tauro


=(




[cid:image005.png@01D00809.A6D502D0]


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433


Cisco.com





[cid:image006.gif@01D00809.A6D502D0]



  Think before you print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.

Please click 
here for 
Company Registration Information.





On Mar 12, 2015, at 10:33 AM, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:

Great :) so just 1 point more, step 4 in adding monitors (Add the new 
monitor to the Monitor map.) this command actually runs in the new monitor 
right?

Thank you so much!


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 10:06 AM, Robert LeBlanc 
mailto:rob...@leblancnet.us>> wrote:

Add the new monitor to the Monitor map.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osd replication

[tombo]

  

Hello, 

I need to understand how replication is accomplished or
who is taking care of replication, osd itsef? Because we are using
librados to read/write to cluster. If librados is not doing parallel
writes according desired number of object copies, it could happen that
objects are in journal waiting for flush and osd went down so objects
are hung in journal? Or do they already have their copies on other osds
which means that librados is resposible for redundancy? 

Thanks for
explanation. ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Robert LeBlanc]

I might be missing something, but is sounds like you already have a monitor
up and running. If you create a new key, the new monitor won't be able to
auth to the existing one. You need to get the monitor key from your
existing monitor and use that for the second (and third) monitor. Look at
step #3 at
http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#adding-a-monitor-manual.
You only have to build a fresh keyring for your first monitor. Your
subsequent monitors will use the first monitor's keyring.

On Thu, Mar 12, 2015 at 11:48 AM, Jesus Chavez (jeschave) <
jesch...@cisco.com> wrote:

>  Thanks I did everything as you mention and still have the same issue that
> hangs a lot:
>
>  [root@tauro ~]# ceph status
> 2015-03-12 11:40:50.441084 7f6e20336700  0 -- :/1005688 >>
> 192.168.4.35:6789/0 pipe(0x7f6e1c0239a0 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f6e1c023c30).fault
> 2015-03-12 11:40:53.441517 7f6e20235700  0 -- :/1005688 >>
> 192.168.4.35:6789/0 pipe(0x7f6e1c00 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f6e1e90).fault
> 2015-03-12 11:40:57.443038 7f6e20336700  0 -- :/1005688 >>
> 192.168.4.35:6789/0 pipe(0x7f6e100030e0 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f6e10003370).fault
> ^CError connecting to cluster: InterruptedOrTimeoutError
> [root@tauro ~]#
>
>  After I saw that my monmap doesn’t have the new entry:
>
>  monmaptool: monmap file /tmp/monmap
> epoch 0
> fsid ddb39f8d-c30f-4172-aeca-30e553a5e4f1
> last_changed 2015-03-12 11:22:20.811525
> created 2015-03-12 11:22:20.811525
> 0: 192.168.4.35:6789/0 mon.aries
> [root@tauro ~]#
>
>
>  It only has the initial monitor (mon.aries) the new one is called
> (mon.tauro), the last command before “…ceph-tauro/done” was "sudo ceph-mon
> --cluster ceph --mkfs -i tauro --keyring /tmp/ceph.mon.keyring --monmap
> /tmp/monmap” also I have tried to add to monmap with:
>
>  [root@tauro ~]# ceph mon add tauro 192.168.4.36:6789
> 2015-03-12 11:45:26.807504 7fc53077a700  0 -- :/1007053 >>
> 192.168.4.35:6789/0 pipe(0x7fc52c023970 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fc52c023c00).fault
> 2015-03-12 11:45:29.807800 7fc530679700  0 -- :/1007053 >>
> 192.168.4.35:6789/0 pipe(0x7fc52c00 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fc52e90).fault
> ^CError connecting to cluster: InterruptedOrTimeoutError
>
>  and also I tried from the initial monitor:
>
>  [root@aries ~]#  ceph mon add tauro 192.168.4.36:6789
> 2015-03-12 11:46:21.561523 7f7432ffd700  0 monclient: hunting for new mon
>
>
>  and it hangs forever, I tried at the same time to get the service up on
> tauro (new mon):
>
>
>  [root@tauro ~]# sudo /etc/init.d/ceph start mon.tauro
> === mon.tauro ===
> Starting Ceph mon.tauro on tauro...
> Running as unit run-7463.service.
> Starting ceph-create-keys on tauro...
> [root@tauro ~]#
>
>
>  and of course still have nothing =(
>
>
>  those were my entire lines:
>
>  vi /etc/ceph/ceph.conf
>
>  i[global]
> fsid = ddb39f8d-c30f-4172-aeca-30e553a5e4f1
> mon initial members = aries
> mon host = 192.168.4.35
>
>  ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon.
> --cap mon 'allow *'
> ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring
> --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow
> *' --cap mds 'allow'
> ceph-authtool /tmp/ceph.mon.keyring --import-keyring
> /etc/ceph/ceph.client.admin.keyring
> monmaptool --create --add aries 192.168.4.35 --fsid
> ddb39f8d-c30f-4172-aeca-30e553a5e4f1 /tmp/monmap
> sudo mkdir /var/lib/ceph/mon/ceph-aries
> ceph-mon --mkfs -i aries --monmap /tmp/monmap --keyring
> /tmp/ceph.mon.keyring
> sudo touch /var/lib/ceph/mon/ceph-aries/done
> sudo /etc/init.d/ceph start mon.aries
>
>  scp /etc/ceph/ceph.conf root@tauro:/etc/ceph/ceph.conf
> scp /tmp/ceph.mon.keyring root@tauro:/tmp/ceph.mon.keyring
> scp /tmp/monmap root@tauro:/tmp/monmap
>
>  mkdir /var/lib/ceph/mon/ceph-tauro
> sudo ceph-mon --cluster ceph --mkfs -i tauro --keyring
> /tmp/ceph.mon.keyring --monmap /tmp/monmap
> sudo ceph-mon -i tauro --mkfs --monmap /tmp/monmap --keyring
> /tmp/ceph.mon.keyring
> sudo touch /var/lib/ceph/mon/ceph-tauro/done
> sudo /etc/init.d/ceph start mon.tauro
>
>
>  =(
>
>
>
>
>
>
> * Jesus Chavez*
> SYSTEMS ENGINEER-C.SALES
>
> jesch...@cisco.com
> Phone: *+52 55 5267 3146 <%2B52%2055%205267%203146>*
> Mobile: *+51 1 5538883255*
>
> CCIE - 44433
>
>
> Cisco.com 
>
>
>
>
>
>   Think before you print.
>
> This email may contain confidential and privileged material for the sole
> use of the intended recipient. Any review, use, distribution or disclosure
> by others is strictly prohibited. If you are not the intended recipient (or
> authorized to receive for the recipient), please contact the sender by
> reply email and delete all copies of this message.
>
> Please click here
>  for
> Company Registration Information.
>
>
>
>
>  On Mar 12, 2015, at 10:33 AM, Jesus Chavez (jeschave) 
> wrote:
>
>  Great :) so ju

Re: [ceph-users] osd replication

[Robert LeBlanc]

The primary OSD for an object is responsible for the replication. In a
healthy cluster the workflow is as such:

   1. Client looks up primary OSD in CRUSH map
   2. Client sends object to be written to primary OSD
   3. Primary OSD looks up replication OSD(s) in its CRUSH map
   4. Primary OSD contacts replication OSD(s) and sends objects
   5. All OSDs commit object to local journal
   6. Replication OSD(s) report back to primary that the write is committed
   7. On the primary OSD, after ack of write from replication OSD(s) and
   it's own local journal does the primary OSD ack the write to the client
   8. Client receives ack and knows that the object is safely stored and
   replicated in the cluster

Ceph has a strong consistency model and will not tell the client the write
is complete until it is replicated in the cluster.

On Thu, Mar 12, 2015 at 12:26 PM, tombo  wrote:

>  Hello,
>
> I need to understand how replication is accomplished or who is taking care
> of replication, osd itsef? Because we are using librados to read/write to
> cluster. If librados is not doing parallel writes according desired number
> of object copies, it could  happen that objects are in journal waiting for
> flush and osd went down so objects are hung in journal? Or do they already
> have their copies on other osds which means that librados is resposible for
> redundancy?
>
> Thanks for explanation.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Jesus Chavez (jeschave)]

Thats what I thought and did actually the monmap and keyring were copied to the 
new monitor and there with 2 elements I did the mkfs thing and still have that 
Messages, do I need osd configured?  Because I have non and I am not sure if it 
is requiered ... Also is weird that monmap is not taking the new monitor I 
think I should try to configure the 3 monitors as initial monitors an see how 
it goes


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 12:44 PM, Robert LeBlanc 
mailto:rob...@leblancnet.us>> wrote:

I might be missing something, but is sounds like you already have a monitor up 
and running. If you create a new key, the new monitor won't be able to auth to 
the existing one. You need to get the monitor key from your existing monitor 
and use that for the second (and third) monitor. Look at step #3 at 
http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#adding-a-monitor-manual.
 You only have to build a fresh keyring for your first monitor. Your subsequent 
monitors will use the first monitor's keyring.

On Thu, Mar 12, 2015 at 11:48 AM, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:
Thanks I did everything as you mention and still have the same issue that hangs 
a lot:

[root@tauro ~]# ceph status
2015-03-12 11:40:50.441084 7f6e20336700  0 -- :/1005688 >> 
192.168.4.35:6789/0 pipe(0x7f6e1c0239a0 sd=3 :0 s=1 
pgs=0 cs=0 l=1 c=0x7f6e1c023c30).fault
2015-03-12 11:40:53.441517 7f6e20235700  0 -- :/1005688 >> 
192.168.4.35:6789/0 pipe(0x7f6e1c00 sd=3 :0 s=1 
pgs=0 cs=0 l=1 c=0x7f6e1e90).fault
2015-03-12 11:40:57.443038 7f6e20336700  0 -- :/1005688 >> 
192.168.4.35:6789/0 pipe(0x7f6e100030e0 sd=4 :0 s=1 
pgs=0 cs=0 l=1 c=0x7f6e10003370).fault
^CError connecting to cluster: InterruptedOrTimeoutError
[root@tauro ~]#

After I saw that my monmap doesn’t have the new entry:

monmaptool: monmap file /tmp/monmap
epoch 0
fsid ddb39f8d-c30f-4172-aeca-30e553a5e4f1
last_changed 2015-03-12 11:22:20.811525
created 2015-03-12 11:22:20.811525
0: 192.168.4.35:6789/0 mon.aries
[root@tauro ~]#


It only has the initial monitor (mon.aries) the new one is called (mon.tauro), 
the last command before “…ceph-tauro/done” was "sudo ceph-mon --cluster ceph 
--mkfs -i tauro --keyring /tmp/ceph.mon.keyring --monmap /tmp/monmap” also I 
have tried to add to monmap with:

[root@tauro ~]# ceph mon add tauro 192.168.4.36:6789
2015-03-12 11:45:26.807504 7fc53077a700  0 -- :/1007053 >> 
192.168.4.35:6789/0 pipe(0x7fc52c023970 sd=3 :0 s=1 
pgs=0 cs=0 l=1 c=0x7fc52c023c00).fault
2015-03-12 11:45:29.807800 7fc530679700  0 -- :/1007053 >> 
192.168.4.35:6789/0 pipe(0x7fc52c00 sd=3 :0 s=1 
pgs=0 cs=0 l=1 c=0x7fc52e90).fault
^CError connecting to cluster: InterruptedOrTimeoutError

and also I tried from the initial monitor:

[root@aries ~]#  ceph mon add tauro 192.168.4.36:6789
2015-03-12 11:46:21.561523 7f7432ffd700  0 monclient: hunting for new mon


and it hangs forever, I tried at the same time to get the service up on tauro 
(new mon):


[root@tauro ~]# sudo /etc/init.d/ceph start mon.tauro
=== mon.tauro ===
Starting Ceph mon.tauro on tauro...
Running as unit run-7463.service.
Starting ceph-create-keys on tauro...
[root@tauro ~]#


and of course still have nothing =(


those were my entire lines:

vi /etc/ceph/ceph.conf

i[global]
fsid = ddb39f8d-c30f-4172-aeca-30e553a5e4f1
mon initial members = aries
mon host = 192.168.4.35

ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap 
mon 'allow *'
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n 
client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 
'allow'
ceph-authtool /tmp/ceph.mon.keyring --import-keyring 
/etc/ceph/ceph.client.admin.keyring
monmaptool --create --add aries 192.168.4.35 --fsid 
ddb39f8d-c30f-4172-aeca-30e553a5e4f1 /tmp/monmap
sudo mkdir /var/lib/ceph/mon/ceph-aries
ceph-mon --mkfs -i aries --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
sudo touch /var/lib/ceph/mon/ceph-aries/done
sudo /etc/init.d/ceph start mon.aries

scp /etc/ceph/ceph.conf root@tauro:/etc/ceph/ceph.conf
scp /tmp/ceph.mon.keyring root@tauro:/tmp/ceph.mon.keyring
scp /tmp/monmap root@tauro:/tmp/monmap

mkdir /var/lib/ceph/mon/ceph-tauro
sudo ceph-mon --cluster ceph --mkfs -i tauro --keyring /tmp/ceph.mon.keyring 
--monmap /tmp/monmap
sudo ceph-mon -i tauro --mkfs --monmap /tmp/monmap --keyring 
/tmp/ceph.mon.keyring
sudo touch /var/lib/ceph/mon/ceph-tauro/done
sudo /etc/init.d/ceph start mon.tauro


=(







Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 

Re: [ceph-users] Add monitor unsuccesful

[Steffen W Sørensen]

> On 12/03/2015, at 20.00, Jesus Chavez (jeschave)  wrote:
> 
> Thats what I thought and did actually the monmap and keyring were copied to 
> the new monitor and there with 2 elements I did the mkfs thing and still have 
> that Messages, do I need osd configured?  Because I have non and I am not 
> sure if it is requiered ... Also is weird that monmap is not taking the new 
> monitor I think I should try to configure the 3 monitors as initial monitors 
> an see how it goes 
Dunno about your config, but I seem to remember when I decommissioned one mon 
instance and addition of a new on another node that I needed to have mon. 
section in ceph.conf inorder to be able to start the monitor.
 
ceph.conf snippet:

[osd]
 osd mount options xfs = 
"rw,noatime,nobarrier,logbsize=256k,logbufs=8,allocsize=4M,attr2,delaylog,inode64,noquota"
 keyring = /var/lib/ceph/osd/ceph-$id/keyring
 ; Tuning
 ;# By default, Ceph makes 3 replicas of objects. If you want to make 
four 
 ;# copies of an object the default value--a primary copy and three 
replica 
 ;# copies--reset the default values as shown in 'osd pool default 
size'.
 ;# If you want to allow Ceph to write a lesser number of copies in a 
degraded 
 ;# state, set 'osd pool default min size' to a number less than the
 ;# 'osd pool default size' value.

 osd pool default size = 2  # Write an object 2 times.
 osd pool default min size = 1 # Allow writing one copy in a degraded 
state.

 ;# Ensure you have a realistic number of placement groups. We recommend
 ;# approximately 100 per OSD. E.g., total number of OSDs multiplied by 
100 
 ;# divided by the number of replicas (i.e., osd pool default size). So 
for
 ;# 10 OSDs and osd pool default size = 3, we'd recommend approximately
 ;# (100 * 10) / 3 = 333.

 ;# got 24 OSDs => 1200 pg, but this is not a full production site, so 
let's settle for 1024 to lower cpu load
 osd pool default pg num = 1024
 osd pool default pgp num = 1024

 client cache size = 131072
 osd client op priority = 40
 osd op threads = 8
 osd client message size cap = 512
 filestore min sync interval = 10
 filestore max sync interval = 60
 ;filestore queue max bytes = 10485760
 ;filestore queue max ops = 50
 ;filestore queue committing max ops = 500
 ;filestore queue committing max bytes = 104857600
 ;filestore op threads = 2
 recovery max active = 2
 recovery op priority = 30
 osd max backfills = 2
 ; Journal Tuning
 journal size = 5120
 ;journal max write bytes = 1073714824
 ;journal max write entries = 1
 ;journal queue max ops = 5
 ;journal queue max bytes = 1048576
 


[mon.0]
 host = node4
 mon addr = 10.0.3.4:6789

[mon.1]
 host = node2
 mon addr = 10.0.3.2:6789

[mon.2]
 host = node1
 mon addr = 10.0.3.1:6789


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Could not find keyring file: /etc/ceph/ceph.client.admin.keyring

[Jesus Chavez (jeschave)]

Ok guys I decided to get back to ceph-deploy after mon create command I have 
got this:

[root@capricornio ~]# ceph-deploy gatherkeys capricornio
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.22): /usr/bin/ceph-deploy gatherkeys 
capricornio
[ceph_deploy.gatherkeys][DEBUG ] Checking capricornio for 
/etc/ceph/ceph.client.admin.keyring
[capricornio][DEBUG ] connected to host: capricornio
[capricornio][DEBUG ] detect platform information from remote host
[capricornio][DEBUG ] detect machine type
[capricornio][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find 
/etc/ceph/ceph.client.admin.keyring on capricornio
[ceph_deploy][ERROR ] KeyNotFoundError: Could not find keyring file: 
/etc/ceph/ceph.client.admin.keyring on host capricornio

[root@capricornio ~]#


Anybody can help me getting new keyring form monmap existing in the system?

Thanks
[cid:image005.png@01D00809.A6D502D0]


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433


Cisco.com





[cid:image006.gif@01D00809.A6D502D0]



  Think before you print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.

Please click 
here for 
Company Registration Information.





On Mar 12, 2015, at 10:33 AM, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:

Great :) so just 1 point more, step 4 in adding monitors (Add the new 
monitor to the Monitor map.) this command actually runs in the new monitor 
right?

Thank you so much!


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 10:06 AM, Robert LeBlanc 
mailto:rob...@leblancnet.us>> wrote:

Add the new monitor to the Monitor map.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Shadow files

[Italo Santos]

Hello Ben,

I’m facing with the same issue - #10295 (http://tracker.ceph.com/issues/10295) 
and I remove the object directly from that rados successfully. But is very 
important map all object before do that. I recommend you take a look to the 
links bellow to understand more about the objects name:

Translating a RadosGW object name into a filename on disk  
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg12161.html
http://www.spinics.net/lists/ceph-devel/msg20426.html




Regards.

Italo Santos
http://italosantos.com.br/


On Thursday, March 12, 2015 at 12:25 PM, Yehuda Sadeh-Weinraub wrote:

>  
>  
> - Original Message -
> > From: "Ben" mailto:b@benjackson.email)>
> > To: ceph-us...@ceph.com (mailto:ceph-us...@ceph.com)
> > Sent: Wednesday, March 11, 2015 8:46:25 PM
> > Subject: Re: [ceph-users] Shadow files
> >  
> > Anyone got any info on this?
> >  
> > Is it safe to delete shadow files?
>  
> It depends. Shadow files are badly named objects that represent part of the 
> objects data. They are only safe to remove if you know that the corresponding 
> objects no longer exist.
>  
> Yehuda
>  
> >  
> > On 2015-03-11 10:03, Ben wrote:
> > > We have a large number of shadow files in our cluster that aren't
> > > being deleted automatically as data is deleted.
> > >  
> > > Is it safe to delete these files?
> > > Is there something we need to be aware of when deleting them?
> > > Is there a script that we can run that will delete these safely?
> > >  
> > > Is there something wrong with our cluster that it isn't deleting these
> > > files when it should be?
> > >  
> > > We are using civetweb with radosgw, with tengine ssl proxy infront of
> > > it
> > >  
> > > Any advice please
> > > Thanks
> > >  
> >  
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com (mailto:ceph-users@lists.ceph.com)
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >  
>  
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com (mailto:ceph-users@lists.ceph.com)
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>  
>  


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Jesus Chavez (jeschave)]

Hi Steffen I already had them in my configuration 😞 I am stress now because it 
seems like none of the methods did help :( this is bad I think I am going to 
get back to rhel6.6 where xfs is a damn add on and I have to install from 
centos repo make ceph like patch :( but at last with RHEL6.6 work, Shame on 
RHEL7 the next time I will sell everything with ubuntu lol


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 1:56 PM, Steffen W Sørensen 
mailto:ste...@me.com>> wrote:


On 12/03/2015, at 20.00, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:

Thats what I thought and did actually the monmap and keyring were copied to the 
new monitor and there with 2 elements I did the mkfs thing and still have that 
Messages, do I need osd configured?  Because I have non and I am not sure if it 
is requiered ... Also is weird that monmap is not taking the new monitor I 
think I should try to configure the 3 monitors as initial monitors an see how 
it goes
Dunno about your config, but I seem to remember when I decommissioned one mon 
instance and addition of a new on another node that I needed to have mon. 
section in ceph.conf inorder to be able to start the monitor.

ceph.conf snippet:

[osd]
osd mount options xfs = 
"rw,noatime,nobarrier,logbsize=256k,logbufs=8,allocsize=4M,attr2,delaylog,inode64,noquota"
keyring = /var/lib/ceph/osd/ceph-$id/keyring
; Tuning
 ;# By default, Ceph makes 3 replicas of objects. If you want to make 
four
;# copies of an object the default value--a primary copy and three replica
;# copies--reset the default values as shown in 'osd pool default size'.
;# If you want to allow Ceph to write a lesser number of copies in a degraded
;# state, set 'osd pool default min size' to a number less than the
;# 'osd pool default size' value.

osd pool default size = 2  # Write an object 2 times.
osd pool default min size = 1 # Allow writing one copy in a degraded state.

;# Ensure you have a realistic number of placement groups. We recommend
;# approximately 100 per OSD. E.g., total number of OSDs multiplied by 100
;# divided by the number of replicas (i.e., osd pool default size). So for
;# 10 OSDs and osd pool default size = 3, we'd recommend approximately
;# (100 * 10) / 3 = 333.

;# got 24 OSDs => 1200 pg, but this is not a full production site, so let's 
settle for 1024 to lower cpu load
osd pool default pg num = 1024
osd pool default pgp num = 1024

client cache size = 131072
osd client op priority = 40
osd op threads = 8
osd client message size cap = 512
filestore min sync interval = 10
filestore max sync interval = 60
;filestore queue max bytes = 10485760
;filestore queue max ops = 50
;filestore queue committing max ops = 500
;filestore queue committing max bytes = 104857600
;filestore op threads = 2
recovery max active = 2
recovery op priority = 30
osd max backfills = 2
; Journal Tuning
journal size = 5120
;journal max write bytes = 1073714824
;journal max write entries = 1
;journal queue max ops = 5
;journal queue max bytes = 1048576




[mon.0]
host = node4
mon addr = 10.0.3.4:6789

[mon.1]
host = node2
mon addr = 10.0.3.2:6789

[mon.2]
host = node1
mon addr = 10.0.3.1:6789


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Robert LeBlanc]

I'm not sure why you are having such a hard time. I added monitors (and
removed them) on CentOS 7 by following what I had. The thing that kept
tripping me up was firewalld. Once I either shut it off or created a
service for Ceph, it worked fine.

What is in in /var/log/ceph/ceph-mon.tauro.log when it is hunting for a
monitor?

On Thu, Mar 12, 2015 at 2:31 PM, Jesus Chavez (jeschave)  wrote:

>  Hi Steffen I already had them in my configuration [image: 😞] I am
> stress now because it seems like none of the methods did help :( this is
> bad I think I am going to get back to rhel6.6 where xfs is a damn add on
> and I have to install from centos repo make ceph like patch :( but at last
> with RHEL6.6 work, Shame on RHEL7 the next time I will sell everything with
> ubuntu lol
>
>
> * Jesus Chavez*
> SYSTEMS ENGINEER-C.SALES
>
> jesch...@cisco.com
> Phone: *+52 55 5267 3146 <+52%2055%205267%203146>*
> Mobile: *+51 1 5538883255 <+51%201%205538883255>*
>
> CCIE - 44433
>
> On Mar 12, 2015, at 1:56 PM, Steffen W Sørensen  wrote:
>
>
>  On 12/03/2015, at 20.00, Jesus Chavez (jeschave) 
> wrote:
>
>  Thats what I thought and did actually the monmap and keyring were copied
> to the new monitor and there with 2 elements I did the mkfs thing and still
> have that Messages, do I need osd configured?  Because I have non and I am
> not sure if it is requiered ... Also is weird that monmap is not taking the
> new monitor I think I should try to configure the 3 monitors as initial
> monitors an see how it goes
>
> Dunno about your config, but I seem to remember when I decommissioned one
> mon instance and addition of a new on another node that I needed to have
> mon. section in ceph.conf inorder to be able to start the monitor.
>
> ceph.conf snippet:
>
>   [osd]
>  osd mount options xfs =
> "rw,noatime,nobarrier,logbsize=256k,logbufs=8,allocsize=4M,attr2,delaylog,inode64,noquota"
>  keyring = /var/lib/ceph/osd/ceph-$id/keyring
>  ; Tuning
>   ;# By default, Ceph makes 3 replicas of objects. If you want to
> make four
>  ;# copies of an object the default value--a primary copy and three
> replica
>  ;# copies--reset the default values as shown in 'osd pool default size'.
>  ;# If you want to allow Ceph to write a lesser number of copies in a
> degraded
>  ;# state, set 'osd pool default min size' to a number less than the
>  ;# 'osd pool default size' value.
>
>  osd pool default size = 2  # Write an object 2 times.
>  osd pool default min size = 1 # Allow writing one copy in a degraded
> state.
>
>  ;# Ensure you have a realistic number of placement groups. We recommend
>  ;# approximately 100 per OSD. E.g., total number of OSDs multiplied by
> 100
>  ;# divided by the number of replicas (i.e., osd pool default size). So
> for
>  ;# 10 OSDs and osd pool default size = 3, we'd recommend approximately
>  ;# (100 * 10) / 3 = 333.
>
>  ;# got 24 OSDs => 1200 pg, but this is not a full production site, so
> let's settle for 1024 to lower cpu load
>  osd pool default pg num = 1024
>  osd pool default pgp num = 1024
>
>  client cache size = 131072
>  osd client op priority = 40
>  osd op threads = 8
>  osd client message size cap = 512
>  filestore min sync interval = 10
>  filestore max sync interval = 60
>  ;filestore queue max bytes = 10485760
>  ;filestore queue max ops = 50
>  ;filestore queue committing max ops = 500
>  ;filestore queue committing max bytes = 104857600
>  ;filestore op threads = 2
>  recovery max active = 2
>  recovery op priority = 30
>  osd max backfills = 2
>  ; Journal Tuning
>  journal size = 5120
>  ;journal max write bytes = 1073714824
>  ;journal max write entries = 1
>  ;journal queue max ops = 5
>  ;journal queue max bytes = 1048576
>
>
>
>
>  [mon.0]
>  host = node4
>  mon addr = 10.0.3.4:6789
>
>  [mon.1]
>  host = node2
>  mon addr = 10.0.3.2:6789
>
>  [mon.2]
>  host = node1
>  mon addr = 10.0.3.1:6789
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Jesus Chavez (jeschave)]

This is awkard Robert all this time was the firewall :( I cant believe I spent 
2 days trying to figure out :(. Thank you so much!


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 2:48 PM, Robert LeBlanc 
mailto:rob...@leblancnet.us>> wrote:

I'm not sure why you are having such a hard time. I added monitors (and removed 
them) on CentOS 7 by following what I had. The thing that kept tripping me up 
was firewalld. Once I either shut it off or created a service for Ceph, it 
worked fine.

What is in in /var/log/ceph/ceph-mon.tauro.log when it is hunting for a monitor?

On Thu, Mar 12, 2015 at 2:31 PM, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:
Hi Steffen I already had them in my configuration  I am 
stress now because it seems like none of the methods did help :( this is bad I 
think I am going to get back to rhel6.6 where xfs is a damn add on and I have 
to install from centos repo make ceph like patch :( but at last with RHEL6.6 
work, Shame on RHEL7 the next time I will sell everything with ubuntu lol


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 1:56 PM, Steffen W Sørensen 
mailto:ste...@me.com>> wrote:


On 12/03/2015, at 20.00, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:

Thats what I thought and did actually the monmap and keyring were copied to the 
new monitor and there with 2 elements I did the mkfs thing and still have that 
Messages, do I need osd configured?  Because I have non and I am not sure if it 
is requiered ... Also is weird that monmap is not taking the new monitor I 
think I should try to configure the 3 monitors as initial monitors an see how 
it goes
Dunno about your config, but I seem to remember when I decommissioned one mon 
instance and addition of a new on another node that I needed to have mon. 
section in ceph.conf inorder to be able to start the monitor.

ceph.conf snippet:

[osd]
osd mount options xfs = 
"rw,noatime,nobarrier,logbsize=256k,logbufs=8,allocsize=4M,attr2,delaylog,inode64,noquota"
keyring = /var/lib/ceph/osd/ceph-$id/keyring
; Tuning
 ;# By default, Ceph makes 3 replicas of objects. If you want to make 
four
;# copies of an object the default value--a primary copy and three replica
;# copies--reset the default values as shown in 'osd pool default size'.
;# If you want to allow Ceph to write a lesser number of copies in a degraded
;# state, set 'osd pool default min size' to a number less than the
;# 'osd pool default size' value.

osd pool default size = 2  # Write an object 2 times.
osd pool default min size = 1 # Allow writing one copy in a degraded state.

;# Ensure you have a realistic number of placement groups. We recommend
;# approximately 100 per OSD. E.g., total number of OSDs multiplied by 100
;# divided by the number of replicas (i.e., osd pool default size). So for
;# 10 OSDs and osd pool default size = 3, we'd recommend approximately
;# (100 * 10) / 3 = 333.

;# got 24 OSDs => 1200 pg, but this is not a full production site, so let's 
settle for 1024 to lower cpu load
osd pool default pg num = 1024
osd pool default pgp num = 1024

client cache size = 131072
osd client op priority = 40
osd op threads = 8
osd client message size cap = 512
filestore min sync interval = 10
filestore max sync interval = 60
;filestore queue max bytes = 10485760
;filestore queue max ops = 50
;filestore queue committing max ops = 500
;filestore queue committing max bytes = 104857600
;filestore op threads = 2
recovery max active = 2
recovery op priority = 30
osd max backfills = 2
; Journal Tuning
journal size = 5120
;journal max write bytes = 1073714824
;journal max write entries = 1
;journal queue max ops = 5
;journal queue max bytes = 1048576




[mon.0]
host = node4
mon addr = 10.0.3.4:6789

[mon.1]
host = node2
mon addr = 10.0.3.2:6789

[mon.2]
host = node1
mon addr = 10.0.3.1:6789



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Shadow files

[ghislain.chevalier]

Hi
I just want to tell you there is a rgw object visualisation that could help you 
in our tool called inkscope available on github
Best regards



Envoyé de mon Galaxy Ace4 Orange


 Message d'origine 
De : Italo Santos 
Date :12/03/2015 21:26 (GMT+01:00)
À : Ben 
Cc : Yehuda Sadeh-Weinraub , ceph-us...@ceph.com
Objet : Re: [ceph-users] Shadow files

Hello Ben,

I’m facing with the same issue - #10295 
and I remove the object directly from that rados successfully. But is very 
important map all object before do that. I recommend you take a look to the 
links bellow to understand more about the objects name:

Translating a RadosGW object name into a filename on disk
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg12161.html
http://www.spinics.net/lists/ceph-devel/msg20426.html

Regards.

Italo Santos
http://italosantos.com.br/


On Thursday, March 12, 2015 at 12:25 PM, Yehuda Sadeh-Weinraub wrote:


- Original Message -
From: "Ben" mailto:b@benjackson.email>>
To: ceph-us...@ceph.com
Sent: Wednesday, March 11, 2015 8:46:25 PM
Subject: Re: [ceph-users] Shadow files

Anyone got any info on this?

Is it safe to delete shadow files?

It depends. Shadow files are badly named objects that represent part of the 
objects data. They are only safe to remove if you know that the corresponding 
objects no longer exist.

Yehuda


On 2015-03-11 10:03, Ben wrote:
We have a large number of shadow files in our cluster that aren't
being deleted automatically as data is deleted.

Is it safe to delete these files?
Is there something we need to be aware of when deleting them?
Is there a script that we can run that will delete these safely?

Is there something wrong with our cluster that it isn't deleting these
files when it should be?

We are using civetweb with radosgw, with tengine ssl proxy infront of
it

Any advice please
Thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Robert LeBlanc]

We all get burned by the firewall at one time or another. Hence the name
'fire'wall! :) I'm glad you got it working.

On Thu, Mar 12, 2015 at 2:53 PM, Jesus Chavez (jeschave)  wrote:

>  This is awkard Robert all this time was the firewall :( I cant believe I
> spent 2 days trying to figure out :(. Thank you so much!
>
>
> * Jesus Chavez*
> SYSTEMS ENGINEER-C.SALES
>
> jesch...@cisco.com
> Phone: *+52 55 5267 3146 <+52%2055%205267%203146>*
> Mobile: *+51 1 5538883255 <+51%201%205538883255>*
>
> CCIE - 44433
>
> On Mar 12, 2015, at 2:48 PM, Robert LeBlanc  wrote:
>
>   I'm not sure why you are having such a hard time. I added monitors (and
> removed them) on CentOS 7 by following what I had. The thing that kept
> tripping me up was firewalld. Once I either shut it off or created a
> service for Ceph, it worked fine.
>
>  What is in in /var/log/ceph/ceph-mon.tauro.log when it is hunting for a
> monitor?
>
> On Thu, Mar 12, 2015 at 2:31 PM, Jesus Chavez (jeschave) <
> jesch...@cisco.com> wrote:
>
>>  Hi Steffen I already had them in my configuration  I
>> am stress now because it seems like none of the methods did help :( this is
>> bad I think I am going to get back to rhel6.6 where xfs is a damn add on
>> and I have to install from centos repo make ceph like patch :( but at last
>> with RHEL6.6 work, Shame on RHEL7 the next time I will sell everything with
>> ubuntu lol
>>
>>
>>
>> * Jesus Chavez*
>> SYSTEMS ENGINEER-C.SALES
>>
>> jesch...@cisco.com
>> Phone: *+52 55 5267 3146 <+52%2055%205267%203146>*
>> Mobile: *+51 1 5538883255 <+51%201%205538883255>*
>>
>> CCIE - 44433
>>
>> On Mar 12, 2015, at 1:56 PM, Steffen W Sørensen  wrote:
>>
>>
>>  On 12/03/2015, at 20.00, Jesus Chavez (jeschave) 
>> wrote:
>>
>>  Thats what I thought and did actually the monmap and keyring were
>> copied to the new monitor and there with 2 elements I did the mkfs thing
>> and still have that Messages, do I need osd configured?  Because I have non
>> and I am not sure if it is requiered ... Also is weird that monmap is not
>> taking the new monitor I think I should try to configure the 3 monitors as
>> initial monitors an see how it goes
>>
>> Dunno about your config, but I seem to remember when I decommissioned one
>> mon instance and addition of a new on another node that I needed to have
>> mon. section in ceph.conf inorder to be able to start the monitor.
>>
>> ceph.conf snippet:
>>
>>  [osd]
>> osd mount options xfs =
>> "rw,noatime,nobarrier,logbsize=256k,logbufs=8,allocsize=4M,attr2,delaylog,inode64,noquota"
>> keyring = /var/lib/ceph/osd/ceph-$id/keyring
>> ; Tuning
>>  ;# By default, Ceph makes 3 replicas of objects. If you want to
>> make four
>> ;# copies of an object the default value--a primary copy and three
>> replica
>> ;# copies--reset the default values as shown in 'osd pool default size'.
>> ;# If you want to allow Ceph to write a lesser number of copies in a
>> degraded
>> ;# state, set 'osd pool default min size' to a number less than the
>> ;# 'osd pool default size' value.
>>
>>  osd pool default size = 2  # Write an object 2 times.
>> osd pool default min size = 1 # Allow writing one copy in a degraded
>> state.
>>
>>  ;# Ensure you have a realistic number of placement groups. We recommend
>> ;# approximately 100 per OSD. E.g., total number of OSDs multiplied by
>> 100
>> ;# divided by the number of replicas (i.e., osd pool default size). So for
>> ;# 10 OSDs and osd pool default size = 3, we'd recommend approximately
>> ;# (100 * 10) / 3 = 333.
>>
>>  ;# got 24 OSDs => 1200 pg, but this is not a full production site, so
>> let's settle for 1024 to lower cpu load
>> osd pool default pg num = 1024
>> osd pool default pgp num = 1024
>>
>>  client cache size = 131072
>> osd client op priority = 40
>> osd op threads = 8
>> osd client message size cap = 512
>> filestore min sync interval = 10
>> filestore max sync interval = 60
>> ;filestore queue max bytes = 10485760
>> ;filestore queue max ops = 50
>> ;filestore queue committing max ops = 500
>> ;filestore queue committing max bytes = 104857600
>> ;filestore op threads = 2
>> recovery max active = 2
>> recovery op priority = 30
>> osd max backfills = 2
>> ; Journal Tuning
>> journal size = 5120
>> ;journal max write bytes = 1073714824
>> ;journal max write entries = 1
>> ;journal queue max ops = 5
>> ;journal queue max bytes = 1048576
>>
>>
>>
>>
>>  [mon.0]
>> host = node4
>> mon addr = 10.0.3.4:6789
>>
>>  [mon.1]
>> host = node2
>> mon addr = 10.0.3.2:6789
>>
>>  [mon.2]
>> host = node1
>> mon addr = 10.0.3.1:6789
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Shadow files

[Ben]

Our cluster has millions of objects in it, there has to be an easy way 
to reconcile objects that no longer exist to shadow files?


We are in a critical position now because we have millions of objects, a 
large number of TB of data, and closing in on 42 osds near full 89% util 
out of 112 osds.


On 2015-03-13 02:25, Yehuda Sadeh-Weinraub wrote:

- Original Message -

From: "Ben" 
To: ceph-us...@ceph.com
Sent: Wednesday, March 11, 2015 8:46:25 PM
Subject: Re: [ceph-users] Shadow files

Anyone got any info on this?

Is it safe to delete shadow files?


It depends. Shadow files are badly named objects that represent part
of the objects data. They are only safe to remove if you know that the
corresponding objects no longer exist.

Yehuda



On 2015-03-11 10:03, Ben wrote:
> We have a large number of shadow files in our cluster that aren't
> being deleted automatically as data is deleted.
>
> Is it safe to delete these files?
> Is there something we need to be aware of when deleting them?
> Is there a script that we can run that will delete these safely?
>
> Is there something wrong with our cluster that it isn't deleting these
> files when it should be?
>
> We are using civetweb with radosgw, with tengine ssl proxy infront of
> it
>
> Any advice please
> Thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

[Malcolm Haak]

Sorry about this,

I sent this at 1AM last night and went to bed, I didn't realise the log was far 
too long and the email had been blocked... 

I've reattached all the requested files and trimmed the body of the email. 

Thank you again for looking at this.

-Original Message-
From: Malcolm Haak 
Sent: Friday, 13 March 2015 1:38 AM
To: 'Joao Eduardo Luis'; ceph-users@lists.ceph.com
Subject: RE: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

Ok,

So, I've been doing things in the meantime and as such the osd is now 
requesting 3008 and 3009 instead of 2758/9
I've included the problem OSD's log file.

And attached all the osdmap's as requested.

Regards

Malcolm Haak

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Joao 
Eduardo Luis
Sent: Friday, 13 March 2015 1:02 AM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing cluster

On 03/12/2015 05:16 AM, Malcolm Haak wrote:
> Sorry about all the unrelated grep issues..
>
> So I've rebuilt and reinstalled and it's still broken.
>
> On the working node, even with the new packages, everything works.
> On the new broken node, I've added a mon and it works. But I still cannot 
> start an OSD on the new node.
>
> What else do you need from me? I'll get logs run any number of tests.
>
> I've got data in this cluster already, and it's full so I need to expand it, 
> I've already got hardware.
>
> Thanks in advance for even having a look

Sam mentioned to me on IRC that the next step would be to grab the 
offending osdmaps.  Easiest way for that will be to stop a monitor and 
run 'ceph-monstore-tool' in order to obtain the full maps, and then use 
'ceph-kvstore-tool' to obtain incrementals.

Given the osd is crashing on version 2759, the following would be best:

(Assuming you have stopped a given monitor with id FOO, whose store is 
sitting at default path /var/lib/ceph/mon/ceph-FOO)

ceph-monstore-tool /var/lib/ceph/mon/ceph-FOO get osdmap -- --version 
2758 --out /tmp/osdmap.full.2758

ceph-monstore-tool /var/lib/ceph/mon/ceph-FOO get osdmap -- --version 
2759 --out /tmp/osdmap.full.2759

(please note the '--' between 'osdmap' and '--version', as that is 
required for the tool to do its thing)

and then

ceph-kvstore-tool /var/lib/ceph/mon/ceph-FOO/store.db get osdmap 2758 
out /tmp/osdmap.inc.2758

ceph-kvstore-tool /var/lib/ceph/mon/ceph-FOO/store.db get osdmap 2759 
out /tmp/osdmap.inc.2759

Cheers!

   -Joao


>
>
> -Original Message-
> From: Samuel Just [mailto:sj...@redhat.com]
> Sent: Wednesday, 11 March 2015 1:41 AM
> To: Malcolm Haak; jl...@redhat.com
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Issues with fresh 0.93 OSD adding to existing 
> cluster
>
> Joao, it looks like map 2759 is causing trouble, how would he get the
> full and incremental maps for that out of the mons?
> -Sam
>
> On Tue, 2015-03-10 at 14:12 +, Malcolm Haak wrote:
>> Hi Samuel,
>>
>> The sha1? I'm going to admit ignorance as to what you are looking for. They 
>> are all running the same release if that is what you are asking.
>> Same tarball built into rpms using rpmbuild on both nodes...
>> Only difference being that the other node has been upgraded and the problem 
>> node is fresh.
>>
>> added the requested config here is the command line output
>>
>> microserver-1:/etc # /etc/init.d/ceph start osd.3
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


ceph-osd.3.log
Description: ceph-osd.3.log


osdmap.full.3008
Description: osdmap.full.3008


osdmap.full.3009
Description: osdmap.full.3009


osdmap.inc.3008
Description: osdmap.inc.3008


osdmap.inc.3009
Description: osdmap.inc.3009
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] rados duplicate object name

[Kapil Sharma]

Hi Cephers,
Has anyone tested the behavior of rados by adding an object to the 
cluster with an object name which already exists in the cluster ?
with command - "rados put -p testpool myobject testfile"

I notice that even if I already have an object called 'myobject' in testpool, 
I can still add a new object with same name and it overwrites my previous 
object without any error message.

With RBD this is not an issue. I do see a proper error message when
I try to add an RBD with name which already exists - 
rbd: create error: (17) File exists2015-03-13 00:16:09.800355 7fe2c4c47780 -1 
librbd: rbd image foo already exists



Regards,
Kapil.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Add monitor unsuccesful

[Jesus Chavez (jeschave)]

Jajaja :) damn fire wall :P well thank you for your patience! Have a great day 
Robert and Steffen


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 3:20 PM, Robert LeBlanc 
mailto:rob...@leblancnet.us>> wrote:

We all get burned by the firewall at one time or another. Hence the name 
'fire'wall! :) I'm glad you got it working.

On Thu, Mar 12, 2015 at 2:53 PM, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:
This is awkard Robert all this time was the firewall :( I cant believe I spent 
2 days trying to figure out :(. Thank you so much!


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 2:48 PM, Robert LeBlanc 
mailto:rob...@leblancnet.us>> wrote:

I'm not sure why you are having such a hard time. I added monitors (and removed 
them) on CentOS 7 by following what I had. The thing that kept tripping me up 
was firewalld. Once I either shut it off or created a service for Ceph, it 
worked fine.

What is in in /var/log/ceph/ceph-mon.tauro.log when it is hunting for a monitor?

On Thu, Mar 12, 2015 at 2:31 PM, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:
Hi Steffen I already had them in my configuration  I am 
stress now because it seems like none of the methods did help :( this is bad I 
think I am going to get back to rhel6.6 where xfs is a damn add on and I have 
to install from centos repo make ceph like patch :( but at last with RHEL6.6 
work, Shame on RHEL7 the next time I will sell everything with ubuntu lol



Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433

On Mar 12, 2015, at 1:56 PM, Steffen W Sørensen 
mailto:ste...@me.com>> wrote:


On 12/03/2015, at 20.00, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:

Thats what I thought and did actually the monmap and keyring were copied to the 
new monitor and there with 2 elements I did the mkfs thing and still have that 
Messages, do I need osd configured?  Because I have non and I am not sure if it 
is requiered ... Also is weird that monmap is not taking the new monitor I 
think I should try to configure the 3 monitors as initial monitors an see how 
it goes
Dunno about your config, but I seem to remember when I decommissioned one mon 
instance and addition of a new on another node that I needed to have mon. 
section in ceph.conf inorder to be able to start the monitor.

ceph.conf snippet:

[osd]
osd mount options xfs = 
"rw,noatime,nobarrier,logbsize=256k,logbufs=8,allocsize=4M,attr2,delaylog,inode64,noquota"
keyring = /var/lib/ceph/osd/ceph-$id/keyring
; Tuning
 ;# By default, Ceph makes 3 replicas of objects. If you want to make 
four
;# copies of an object the default value--a primary copy and three replica
;# copies--reset the default values as shown in 'osd pool default size'.
;# If you want to allow Ceph to write a lesser number of copies in a degraded
;# state, set 'osd pool default min size' to a number less than the
;# 'osd pool default size' value.

osd pool default size = 2  # Write an object 2 times.
osd pool default min size = 1 # Allow writing one copy in a degraded state.

;# Ensure you have a realistic number of placement groups. We recommend
;# approximately 100 per OSD. E.g., total number of OSDs multiplied by 100
;# divided by the number of replicas (i.e., osd pool default size). So for
;# 10 OSDs and osd pool default size = 3, we'd recommend approximately
;# (100 * 10) / 3 = 333.

;# got 24 OSDs => 1200 pg, but this is not a full production site, so let's 
settle for 1024 to lower cpu load
osd pool default pg num = 1024
osd pool default pgp num = 1024

client cache size = 131072
osd client op priority = 40
osd op threads = 8
osd client message size cap = 512
filestore min sync interval = 10
filestore max sync interval = 60
;filestore queue max bytes = 10485760
;filestore queue max ops = 50
;filestore queue committing max ops = 500
;filestore queue committing max bytes = 104857600
;filestore op threads = 2
recovery max active = 2
recovery op priority = 30
osd max backfills = 2
; Journal Tuning
journal size = 5120
;journal max write bytes = 1073714824
;journal max write entries = 1
;journal queue max ops = 5
;journal queue max bytes = 1048576




[mon.0]
host = node4
mon addr = 10.0.3.4:6789

[mon.1]
host = node2
mon addr = 10.0.3.2:6789

[mon.2]
host = node1
mon addr = 10.0.3.1:6789



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://l

[ceph-users] Strange Monitor Appearance after Update

[Georgios Dimitrakakis]

Hi all!

I have updated from 0.80.8 to 0.80.9 and every time I try to restart 
CEPH a monitor a strange monitor is appearing!


Here is the output:


#/etc/init.d/ceph restart mon
=== mon.master ===
=== mon.master ===
Stopping Ceph mon.master on master...kill 10766...done
=== mon.master ===
Starting Ceph mon.master on master...
Starting ceph-create-keys on master...
=== mon.master_192.168.0.10 ===
=== mon.master_192.168.0.10 ===
Stopping Ceph mon.master_192.168.0.10 on master...done
=== mon.master_192.168.0.10 ===
Starting Ceph mon.master_192.168.0.10 on master...
2015-03-13 03:06:22.964493 7f06256fa7a0 -1 
mon.master_192.168.0.10@-1(probing) e2 not in monmap and have been in a 
quorum before; must have been removed
2015-03-13 03:06:22.964497 7f06256fa7a0 -1 
mon.master_192.168.0.10@-1(probing) e2 commit suicide!

2015-03-13 03:06:22.964499 7f06256fa7a0 -1 failed to initialize
failed: 'ulimit -n 32768;  /usr/bin/ceph-mon -i master_192.168.0.10 
--pid-file /var/run/ceph/mon.master_192.168.0.10.pid -c 
/etc/ceph/ceph.conf --cluster ceph '



I have two monitors which are:

mon.master and mon.client1

and have defined them in ceph.conf as:

mon_initial_members = master,client1
mon_host = 192.168.0.10,192.168.0.11



Why is the "mon.master_192.168.0.10" appearing and how can I stop it 
from happening?



The above is the problem on one node. Obviously the problem is 
appearing on the other node as well but instead I have


"mon.client1_192.168.0.11" appearing



Any ideas?


Regards,


George
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Strange Monitor Appearance after Update

[Georgios Dimitrakakis]

I forgot to say that the monitors form a quorum and the cluster's 
health is OK

so there aren't any serious troubles other than the annoying message.

Best,

George


Hi all!

I have updated from 0.80.8 to 0.80.9 and every time I try to restart
CEPH a monitor a strange monitor is appearing!

Here is the output:


#/etc/init.d/ceph restart mon
=== mon.master ===
=== mon.master ===
Stopping Ceph mon.master on master...kill 10766...done
=== mon.master ===
Starting Ceph mon.master on master...
Starting ceph-create-keys on master...
=== mon.master_192.168.0.10 ===
=== mon.master_192.168.0.10 ===
Stopping Ceph mon.master_192.168.0.10 on master...done
=== mon.master_192.168.0.10 ===
Starting Ceph mon.master_192.168.0.10 on master...
2015-03-13 03:06:22.964493 7f06256fa7a0 -1
mon.master_192.168.0.10@-1(probing) e2 not in monmap and have been in
a quorum before; must have been removed
2015-03-13 03:06:22.964497 7f06256fa7a0 -1
mon.master_192.168.0.10@-1(probing) e2 commit suicide!
2015-03-13 03:06:22.964499 7f06256fa7a0 -1 failed to initialize
failed: 'ulimit -n 32768;  /usr/bin/ceph-mon -i master_192.168.0.10
--pid-file /var/run/ceph/mon.master_192.168.0.10.pid -c
/etc/ceph/ceph.conf --cluster ceph '


I have two monitors which are:

mon.master and mon.client1

and have defined them in ceph.conf as:

mon_initial_members = master,client1
mon_host = 192.168.0.10,192.168.0.11



Why is the "mon.master_192.168.0.10" appearing and how can I stop it
from happening?


The above is the problem on one node. Obviously the problem is
appearing on the other node as well but instead I have

"mon.client1_192.168.0.11" appearing



Any ideas?


Regards,


George
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Strange Monitor Appearance after Update

[Robert LeBlanc]

Two monitors don't work very well and really don't but you anything. I
would either add another monitor or remove one. Paxos is most effective
with an odd number of monitors.

I don't know about the problem you are experiencing and how to help you. An
even number of monitors should work.

Robert LeBlanc

Sent from a mobile device please excuse any typos.
On Mar 12, 2015 7:19 PM, "Georgios Dimitrakakis" 
wrote:

> I forgot to say that the monitors form a quorum and the cluster's health
> is OK
> so there aren't any serious troubles other than the annoying message.
>
> Best,
>
> George
>
>  Hi all!
>>
>> I have updated from 0.80.8 to 0.80.9 and every time I try to restart
>> CEPH a monitor a strange monitor is appearing!
>>
>> Here is the output:
>>
>>
>> #/etc/init.d/ceph restart mon
>> === mon.master ===
>> === mon.master ===
>> Stopping Ceph mon.master on master...kill 10766...done
>> === mon.master ===
>> Starting Ceph mon.master on master...
>> Starting ceph-create-keys on master...
>> === mon.master_192.168.0.10 ===
>> === mon.master_192.168.0.10 ===
>> Stopping Ceph mon.master_192.168.0.10 on master...done
>> === mon.master_192.168.0.10 ===
>> Starting Ceph mon.master_192.168.0.10 on master...
>> 2015-03-13 03:06:22.964493 7f06256fa7a0 -1
>> mon.master_192.168.0.10@-1(probing) e2 not in monmap and have been in
>> a quorum before; must have been removed
>> 2015-03-13 03:06:22.964497 7f06256fa7a0 -1
>> mon.master_192.168.0.10@-1(probing) e2 commit suicide!
>> 2015-03-13 03:06:22.964499 7f06256fa7a0 -1 failed to initialize
>> failed: 'ulimit -n 32768;  /usr/bin/ceph-mon -i master_192.168.0.10
>> --pid-file /var/run/ceph/mon.master_192.168.0.10.pid -c
>> /etc/ceph/ceph.conf --cluster ceph '
>>
>>
>> I have two monitors which are:
>>
>> mon.master and mon.client1
>>
>> and have defined them in ceph.conf as:
>>
>> mon_initial_members = master,client1
>> mon_host = 192.168.0.10,192.168.0.11
>>
>>
>>
>> Why is the "mon.master_192.168.0.10" appearing and how can I stop it
>> from happening?
>>
>>
>> The above is the problem on one node. Obviously the problem is
>> appearing on the other node as well but instead I have
>>
>> "mon.client1_192.168.0.11" appearing
>>
>>
>>
>> Any ideas?
>>
>>
>> Regards,
>>
>>
>> George
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Strange Monitor Appearance after Update

[Georgios Dimitrakakis]

Hi Robert!

Thanks for the feedback! I am aware of the fact that the number of the 
monitors should be odd
but this is a very basic setup just to test CEPH functionality and 
perform tasks there before

doing it to our production cluster.

So I am not concerned about that and I really don't believe that this 
is why the problem has appeared!


What concerns me is how this "new" monitor that has the same name 
followed by an underscore and

the IP address appeared out of nowhere and how to stop it!

Regards,

George


Two monitors dont work very well and really dont but you anything. I
would either add another monitor or remove one. Paxos is most
effective with an odd number of monitors.

I dont know about the problem you are experiencing and how to help
you. An even number of monitors should work.

Robert LeBlanc

Sent from a mobile device please excuse any typos.
On Mar 12, 2015 7:19 PM, "Georgios Dimitrakakis"  wrote:


I forgot to say that the monitors form a quorum and the clusters
health is OK
so there arent any serious troubles other than the annoying
message.

Best,

George


Hi all!

I have updated from 0.80.8 to 0.80.9 and every time I try to
restart
CEPH a monitor a strange monitor is appearing!

Here is the output:

#/etc/init.d/ceph restart mon
=== mon.master ===
=== mon.master ===
Stopping Ceph mon.master on master...kill 10766...done
=== mon.master ===
Starting Ceph mon.master on master...
Starting ceph-create-keys on master...
=== mon.master_192.168.0.10 ===
=== mon.master_192.168.0.10 ===
Stopping Ceph mon.master_192.168.0.10 on master...done
=== mon.master_192.168.0.10 ===
Starting Ceph mon.master_192.168.0.10 on master...
2015-03-13 03:06:22.964493 7f06256fa7a0 -1
mon.master_192.168.0.10@-1(probing) e2 not in monmap and have
been in
a quorum before; must have been removed
2015-03-13 03:06:22.964497 7f06256fa7a0 -1
mon.master_192.168.0.10@-1(probing) e2 commit suicide!
2015-03-13 03:06:22.964499 7f06256fa7a0 -1 failed to initialize
failed: ulimit -n 32768;  /usr/bin/ceph-mon -i
master_192.168.0.10
--pid-file /var/run/ceph/mon.master_192.168.0.10.pid -c
/etc/ceph/ceph.conf --cluster ceph

I have two monitors which are:

mon.master and mon.client1

and have defined them in ceph.conf as:

mon_initial_members = master,client1
mon_host = 192.168.0.10,192.168.0.11

Why is the "mon.master_192.168.0.10" appearing and how can I stop
it
from happening?

The above is the problem on one node. Obviously the problem is
appearing on the other node as well but instead I have

"mon.client1_192.168.0.11" appearing

Any ideas?

Regards,

George
___
ceph-users mailing list
ceph-users@lists.ceph.com [1]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [2]

___
ceph-users mailing list
ceph-users@lists.ceph.com [3]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [4]



Links:
--
[1] mailto:ceph-users@lists.ceph.com
[2] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[3] mailto:ceph-users@lists.ceph.com
[4] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[5] mailto:gior...@acmac.uoc.gr

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Strange Monitor Appearance after Update

[Robert LeBlanc]

Having two monitors should not be causing the problem you are seeing like
you say. What is in /var/log/ceph/ceph.mon.*.log?

Robert LeBlanc

Sent from a mobile device please excuse any typos.
On Mar 12, 2015 7:39 PM, "Georgios Dimitrakakis" 
wrote:

> Hi Robert!
>
> Thanks for the feedback! I am aware of the fact that the number of the
> monitors should be odd
> but this is a very basic setup just to test CEPH functionality and perform
> tasks there before
> doing it to our production cluster.
>
> So I am not concerned about that and I really don't believe that this is
> why the problem has appeared!
>
> What concerns me is how this "new" monitor that has the same name followed
> by an underscore and
> the IP address appeared out of nowhere and how to stop it!
>
> Regards,
>
> George
>
>  Two monitors dont work very well and really dont but you anything. I
>> would either add another monitor or remove one. Paxos is most
>> effective with an odd number of monitors.
>>
>> I dont know about the problem you are experiencing and how to help
>> you. An even number of monitors should work.
>>
>> Robert LeBlanc
>>
>> Sent from a mobile device please excuse any typos.
>> On Mar 12, 2015 7:19 PM, "Georgios Dimitrakakis"  wrote:
>>
>>  I forgot to say that the monitors form a quorum and the clusters
>>> health is OK
>>> so there arent any serious troubles other than the annoying
>>> message.
>>>
>>> Best,
>>>
>>> George
>>>
>>>  Hi all!

 I have updated from 0.80.8 to 0.80.9 and every time I try to
 restart
 CEPH a monitor a strange monitor is appearing!

 Here is the output:

 #/etc/init.d/ceph restart mon
 === mon.master ===
 === mon.master ===
 Stopping Ceph mon.master on master...kill 10766...done
 === mon.master ===
 Starting Ceph mon.master on master...
 Starting ceph-create-keys on master...
 === mon.master_192.168.0.10 ===
 === mon.master_192.168.0.10 ===
 Stopping Ceph mon.master_192.168.0.10 on master...done
 === mon.master_192.168.0.10 ===
 Starting Ceph mon.master_192.168.0.10 on master...
 2015-03-13 03:06:22.964493 7f06256fa7a0 -1
 mon.master_192.168.0.10@-1(probing) e2 not in monmap and have
 been in
 a quorum before; must have been removed
 2015-03-13 03:06:22.964497 7f06256fa7a0 -1
 mon.master_192.168.0.10@-1(probing) e2 commit suicide!
 2015-03-13 03:06:22.964499 7f06256fa7a0 -1 failed to initialize
 failed: ulimit -n 32768;  /usr/bin/ceph-mon -i
 master_192.168.0.10
 --pid-file /var/run/ceph/mon.master_192.168.0.10.pid -c
 /etc/ceph/ceph.conf --cluster ceph

 I have two monitors which are:

 mon.master and mon.client1

 and have defined them in ceph.conf as:

 mon_initial_members = master,client1
 mon_host = 192.168.0.10,192.168.0.11

 Why is the "mon.master_192.168.0.10" appearing and how can I stop
 it
 from happening?

 The above is the problem on one node. Obviously the problem is
 appearing on the other node as well but instead I have

 "mon.client1_192.168.0.11" appearing

 Any ideas?

 Regards,

 George
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com [1]
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [2]

>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com [3]
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [4]
>>>
>>
>>
>> Links:
>> --
>> [1] mailto:ceph-users@lists.ceph.com
>> [2] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> [3] mailto:ceph-users@lists.ceph.com
>> [4] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> [5] mailto:gior...@acmac.uoc.gr
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS: delayed objects deletion ?

[Yan, Zheng]

On Fri, Mar 13, 2015 at 1:17 AM, Florent B  wrote:
> Hi all,
>
> I test CephFS again on Giant release.
>
> I use ceph-fuse.
>
> After deleting a large directory (few hours ago), I can see that my pool
> still contains 217 GB of objects.
>
> Even if my root directory on CephFS is empty.
>
> And metadata pool is 46 MB.
>
> Is it expected ? If not, how to debug this ?

Old mds does not work well in this area. Try umounting clients and
restarting MDS.

Regards
Yan, Zheng


>
> Thank you.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] OSD booting down

[Jesus Chavez (jeschave)]

Hi all, after adding osds manually and reboot the server the osd didnt come up 
automatically am I missing something?

Thanks


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD booting down

[Robert LeBlanc]

My experience with CentOS 7 is that ceph-disk works the best. Systemd has a
fit with extra arguments common in the upstart and SysV scripts. Ceph
installs udev rules that will automatically mount and start OSDs.

The udev rules look for GPT partition UUIDs that are set aside for Ceph to
find partitions that should be mounted and started. You can do it by hand
(I've done it to understand the process) but it is a lot of work. Since
we've gone to using ceph-disk we haven't had any problems with OSDs
starting at boot. If I need to restart an OSD, I just kill the process and
then run ceph-disk activate. Ceph-disk is just a script so you can open it
up and take a look.

So I guess it depends on which automatically you want to happen.

Robert LeBlanc

Sent from a mobile device please excuse any typos.
On Mar 12, 2015 9:54 PM, "Jesus Chavez (jeschave)" 
wrote:

>  Hi all, after adding osds manually and reboot the server the osd didnt
> come up automatically am I missing something?
>
>  Thanks
>
>
> * Jesus Chavez*
> SYSTEMS ENGINEER-C.SALES
>
> jesch...@cisco.com
> Phone: *+52 55 5267 3146 <+52%2055%205267%203146>*
> Mobile: *+51 1 5538883255 <+51%201%205538883255>*
>
> CCIE - 44433
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Does ceph zero out RBD volumes when deleted?

[Sreenath BH]

When a RBD volume is deleted, does Ceph fill used 4 MB chunks with zeros?

thanks,
Sreenath
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Mapping users to different rgw pools

[Sreenath BH]

Hi all,

Can one Radow gateway support more than one pool for storing objects?

And as a follow-up question, is there a way to map different users to
separate rgw pools so that their obejcts get stored in different
pools?

thanks,
Sreenath
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com