[ceph-users] python binding - snap rollback - progress reporting

[Nikola Ciprich]

Hello,

I'd like to ask - I'm using python RBD/rados bindings. Everything
works well for me, the only thing I'd like to improve is snapshots rollback
as the operation is quite time consuming, I would like to report it's progress.

is this somehow possible? even at the cost of implementing whole rollback 
operation by myself?

thanks a lot in advance!

BR

nik


-- 
-
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:+420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: ser...@linuxbox.cz
-


pgpGcfrLSPucy.pgp
Description: PGP signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issue activating OSDs

[Oliver Dzombic]

Hi Robert,

did you already tried ceph-deply gatherkeys ?

And did you already tried to reboot the nodes completely ?

--
Mit freundlichen Gruessen / Best regards

Oliver Dzombic
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Multiple Cache Pool with Single Storage Pool

[Lazuardi Nasution]

Hi,

I'm new with CEPH cache tiering. Is it possible to have multiple cache pool
with single storage pool backend? For example I have some compute nodes
with its own local SSDs. I want each compute node has its own cache by
using its own local SSDs. The target is to minimize load to storage pool
backend.

If yes, what will be used for RBD client pool name, cache pool name or
storage pool name? How to make sure that RBD client will only use cache
pool constructed with its own local SSDs, not other nodes local SSDs?

Frankly, I'm looking for RBD client cache on SSDs, but it seem has not
available yet.

Best regards,
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph RBD LIO ESXi Advice?

[Timofey Titovets]

Big thanks Nick, any way
Now i catch hangs of ESXi and  Proxy =_=''
/* Proxy VM: Ubuntu 15.10/Kernel 4.3/LIO/Ceph 0.94/ESXi 6.0 Software iSCSI*/
I've moved to NFS-RBD proxy and now try to make it HA

2015-11-07 18:59 GMT+03:00 Nick Fisk :
> Hi Timofey,
>
> You are most likely experiencing the effects of Ceph's write latency in 
> combination with the sync write behaviour of ESXi. You will probably struggle 
> to get much under 2ms write latency with Ceph, assuming a minimum of 2 copies 
> in Ceph. This will limit you to around 500iops for a QD of 1. Because of this 
> you will also experience slow file/VM copies, as ESXi moves the blocks of 
> data around in 64kb sync IO's. 500x64kb = ~30MB/s.
>
> Moving to 10GB end to end may get you a reasonable boost in performance as 
> you will be removing a 1ms or so of latency from the network for each write. 
> Also search the mailing list for small performance tweaks you can do, like 
> disabling logging.
>
> Other than that the only thing I have found that has chance of giving you 
> performance similar to other products and/or legacy SAN's is to use some sort 
> of RBD caching with something like flashcache/enhanceio/bcache o nyour proxy 
> nodes. However this brings its on challenges and I still haven't got to a 
> point that I'm happy to deploy it.
>
> I'm surprised you are also not seeing LIO hangs, which several people 
> including me experience when using RBD+LIO+ESXi, although I haven't checked 
> recently to see if this is now working better. I would be interesting in 
> hearing your feedback on this. They normally manifest themselves when an OSD 
> drops out and IO is suspended for more than 5-10s.
>
> Sorry I couldn't be of more help.
>
> Nick
>
>> -Original Message-
>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
>> Timofey Titovets
>> Sent: 07 November 2015 11:44
>> To: ceph-users@lists.ceph.com
>> Subject: [ceph-users] Ceph RBD LIO ESXi Advice?
>>
>> Hi List,
>> I Searching for advice from somebody, who use Legacy client like ESXi with
>> Ceph
>>
>> I try to build High-performance fault-tolerant storage with Ceph 0.94
>>
>> In production i have 50+ TB of VMs (~800 VMs)
>> 8 NFS servers each:
>> 2xIntel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz 12xSeagate ST2000NM0023
>> 1xLSI Nytro™ MegaRAID® NMR 8110-4i
>> 96 GB of RAM
>> 4x 1 GBE links in Balance-ALB mode (I don't have problem with network
>> throughput)
>>
>> Now in lab. i have build 3 node cluster like:
>> Kernel 4.2
>> Intel(R) Xeon(R) CPU 5130  @ 2.00GHz
>> 16 Gb of RAM
>> 6xSeagate ST2000NM0033
>> 2x 1GBE in Balance-ALB
>> i.e. each node is a MON and 6 OSDs
>>
>>
>> Config like:
>> osd journal size = 16384
>> osd pool default size = 2
>> osd pool default min size = 2
>> osd pool default pg num = 256
>> osd pool default pgp num = 256
>> osd crush chooseleaf type = 1
>> filestore max sync interval = 180
>>
>> For attach RBD Storage to ESXi i create a 2 VMs:
>> 2 cores
>> 2 GB RAM
>> Kernel 4.3
>> Each vm map big RBD volume and proxy it by LIO to ESXi ESXi see VMs like
>> iSCSI Target server in Active/Passive mode
>>
>> RBD created with --image-shared and --image-format 2 keys
>>
>> My Questions:
>> 1. I have architecture problem?
>> 2. May be you have ideas?
>> 3. ESXi working with iSCSI storage very slow(30-60 Mb/s read/write), but this
>> is can be a ESXi problem, later i will test this with more modern Hypervisor
>> server 4. Proxy VMs working not too bad with storage, but fio shows too low
>> numbers:
>> [global]
>> size=128g   # File size
>> filename=/storage/testfile.fio
>> numjobs=1   # One tread
>> runtime=600 # 10m for each test
>> ioengine=libaio # Use async io
>> # Pseude random data, can be compressed by 15%
>> buffer_compress_percentage=15
>> overwrite=1 # Overwrite data in file
>> end_fsync=1 # Doing fsync, at the and of test, for sync OS buffers
>> direct=1# Bypass OS cache
>> startdelay=30   # Pause between tests
>> bs=4k   # Block size for io requests
>> iodepth=64  # Count of IO request, what can be requested asynchronously
>> rw=randrw   # Random Read/Write
>> 
>> # IOMeter defines the server loads as the following:
>> # iodepth=1   # Linear
>> # iodepth=4   # Very Light
>> # iodepth=8   # Light
>> # iodepth=64  # Moderate
>> # iodepth=256 # Heavy
>> 
>> [Disk-4k-randomrw-depth-1]
>> rwmixread=50
>> iodepth=1
>> stonewall # Do each test separated
>> 
>> [Disk-4k-randomrw-depth-8]
>> rwmixread=50
>> iodepth=8
>> stonewall
>> 
>> [Disk-4k-randomrw-depth-64]
>> rwmixread=50
>> stonewall
>> 
>> [Disk-4k-randomrw-depth-256]
>> rwmixread=50
>> iodepth=256
>> stonewall
>> 
>> [Disk-4k-randomrw-depth-512]
>> rwmixread=50
>> iodepth=5

Re: [ceph-users] Ceph RBD LIO ESXi Advice?

[Nick Fisk]

You might find NFS even slower as you will probably have 2IO's for every ESXi 
IO due to the journal for whatever FS you are using on the NFS server. If you 
are not seeing it even slower, you most likely have the NFS server set to run 
in async mode.

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Timofey Titovets
> Sent: 08 November 2015 22:07
> To: Nick Fisk 
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Ceph RBD LIO ESXi Advice?
> 
> Big thanks Nick, any way
> Now i catch hangs of ESXi and  Proxy =_=''
> /* Proxy VM: Ubuntu 15.10/Kernel 4.3/LIO/Ceph 0.94/ESXi 6.0 Software
> iSCSI*/ I've moved to NFS-RBD proxy and now try to make it HA
> 
> 2015-11-07 18:59 GMT+03:00 Nick Fisk :
> > Hi Timofey,
> >
> > You are most likely experiencing the effects of Ceph's write latency in
> combination with the sync write behaviour of ESXi. You will probably struggle
> to get much under 2ms write latency with Ceph, assuming a minimum of 2
> copies in Ceph. This will limit you to around 500iops for a QD of 1. Because 
> of
> this you will also experience slow file/VM copies, as ESXi moves the blocks of
> data around in 64kb sync IO's. 500x64kb = ~30MB/s.
> >
> > Moving to 10GB end to end may get you a reasonable boost in
> performance as you will be removing a 1ms or so of latency from the
> network for each write. Also search the mailing list for small performance
> tweaks you can do, like disabling logging.
> >
> > Other than that the only thing I have found that has chance of giving you
> performance similar to other products and/or legacy SAN's is to use some
> sort of RBD caching with something like flashcache/enhanceio/bcache o
> nyour proxy nodes. However this brings its on challenges and I still haven't
> got to a point that I'm happy to deploy it.
> >
> > I'm surprised you are also not seeing LIO hangs, which several people
> including me experience when using RBD+LIO+ESXi, although I haven't
> checked recently to see if this is now working better. I would be interesting
> in hearing your feedback on this. They normally manifest themselves when
> an OSD drops out and IO is suspended for more than 5-10s.
> >
> > Sorry I couldn't be of more help.
> >
> > Nick
> >
> >> -Original Message-
> >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf
> >> Of Timofey Titovets
> >> Sent: 07 November 2015 11:44
> >> To: ceph-users@lists.ceph.com
> >> Subject: [ceph-users] Ceph RBD LIO ESXi Advice?
> >>
> >> Hi List,
> >> I Searching for advice from somebody, who use Legacy client like ESXi
> >> with Ceph
> >>
> >> I try to build High-performance fault-tolerant storage with Ceph 0.94
> >>
> >> In production i have 50+ TB of VMs (~800 VMs)
> >> 8 NFS servers each:
> >> 2xIntel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz 12xSeagate ST2000NM0023
> >> 1xLSI Nytro™ MegaRAID® NMR 8110-4i
> >> 96 GB of RAM
> >> 4x 1 GBE links in Balance-ALB mode (I don't have problem with network
> >> throughput)
> >>
> >> Now in lab. i have build 3 node cluster like:
> >> Kernel 4.2
> >> Intel(R) Xeon(R) CPU 5130  @ 2.00GHz
> >> 16 Gb of RAM
> >> 6xSeagate ST2000NM0033
> >> 2x 1GBE in Balance-ALB
> >> i.e. each node is a MON and 6 OSDs
> >>
> >>
> >> Config like:
> >> osd journal size = 16384
> >> osd pool default size = 2
> >> osd pool default min size = 2
> >> osd pool default pg num = 256
> >> osd pool default pgp num = 256
> >> osd crush chooseleaf type = 1
> >> filestore max sync interval = 180
> >>
> >> For attach RBD Storage to ESXi i create a 2 VMs:
> >> 2 cores
> >> 2 GB RAM
> >> Kernel 4.3
> >> Each vm map big RBD volume and proxy it by LIO to ESXi ESXi see VMs
> >> like iSCSI Target server in Active/Passive mode
> >>
> >> RBD created with --image-shared and --image-format 2 keys
> >>
> >> My Questions:
> >> 1. I have architecture problem?
> >> 2. May be you have ideas?
> >> 3. ESXi working with iSCSI storage very slow(30-60 Mb/s read/write),
> >> but this is can be a ESXi problem, later i will test this with more
> >> modern Hypervisor server 4. Proxy VMs working not too bad with
> >> storage, but fio shows too low
> >> numbers:
> >> [global]
> >> size=128g   # File size
> >> filename=/storage/testfile.fio
> >> numjobs=1   # One tread
> >> runtime=600 # 10m for each test
> >> ioengine=libaio # Use async io
> >> # Pseude random data, can be compressed by 15%
> >> buffer_compress_percentage=15
> >> overwrite=1 # Overwrite data in file
> >> end_fsync=1 # Doing fsync, at the and of test, for sync OS buffers
> >> direct=1# Bypass OS cache
> >> startdelay=30   # Pause between tests
> >> bs=4k   # Block size for io requests
> >> iodepth=64  # Count of IO request, what can be requested
> asynchronously
> >> rw=randrw   # Random Read/Write
> >> 
> >> # IOMeter defines the server loads as the following:
> >> # iodepth=1   # Linear
> >> # iodepth=4   # Very Light
> >> # iodepth=8   # Light
>

Re: [ceph-users] Erasure coded pools and 'feature set mismatch' issue

[Gregory Farnum]

With that release it shouldn't be the EC pool causing trouble; it's the
CRUSH tunables also mentioned in that thread. Instructions should be
available in the docs for using older tunable that are compatible with
kernel 3.13.
-Greg

On Saturday, November 7, 2015, Bogdan SOLGA  wrote:

> Hello, everyone!
>
> I have recently created a Ceph cluster (v 0.94.5) on Ubuntu 14.04.3 and I
> have created an erasure coded pool, which has a caching pool in front of it.
>
> When trying to map RBD images, regardless if they are created in the rbd
> or in the erasure coded pool, the operation fails with 'rbd: map failed:
> (5) Input/output error'. Searching the internet for a solution... I came
> across this
> 
> page, which seems to detail exactly the same issue - a 'misunderstanding'
> between erasure coded pools and the 3.13 kernel (used by Ubuntu).
>
> Can you please advise on a fix for that issue? As we would prefer to use
> erasure coded pools, the only solutions which came into my mind were:
>
>- upgrade to the Infernalis Ceph release, although I'm not sure the
>issue is fixed in that version;
>
>
>- upgrade the kernel (on all the OSDs and Ceph clients) to the 3.14+
>kernel;
>
> Any better / easier solution is highly appreciated.
>
> Regards,
>
> Bogdan
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Radosgw admin MNG Tools to create and report usage of Object accounts

[Florent MONTHEL]

Hi Cephers,

I’ve just released toolkit on python to report usage and inventory of buckets / 
accounts / S3 and Swift keys - In the same way, we have script to create 
account and S3/Swift keys (and initial buckets)
Tool is using rgwadmin python module 

https://github.com/fmonthel/radosgw-admin-mng-tools 


Thanks



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[WD_Hwang]

Hi, Craig:
  I used 10 VMs for federated gateway testing. There are 5 nodes for us-east, 
and the others are for us-west. The two zones are independent.
  Before the configurations of the region and zone, I have the two zones with 
the same 'client.radosgw.[zone]' setting of ceph.conf.
-
[client.radosgw.us-east-1]
rgw region = us
rgw region root pool = .us.rgw.root
rgw zone = us-east
rgw zone root pool = .us-east.rgw.root
host = node1-east
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /tmp/radosgw.us-east.sock
log file = /var/log/ceph/radosgw.us-east.log
rgw dns name = node1-east.ceph.com

[client.radosgw.us-west-1]
rgw region = us
rgw region root pool = .us.rgw.root
rgw zone = us-west
rgw zone root pool = .us-west.rgw.root
host = node1-west
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /tmp/radosgw.us-west.sock
log file = /var/log/ceph/radosgw.us-west.log
rgw dns name = node1-west.ceph.com
-

  The needed pools were created by manual. And I also created a normal user in 
the two zones with same access key and secret key. After that, I used 's3cmd' 
to create a bucket as 'BUCKETa' and put 'ceph.conf' in the bucket to test the 
synchronization. There was some error message in the logging file of 
'radosgw-agent'. I found the file of 'ceph.conf' was sent to the secondary zone 
and DELETED later by unknown reason.
-
application/json; charset=UTF-8
Fri, 06 Nov 2015 09:18:01 GMT
x-amz-copy-source:BUCKETa/ceph.conf
/BUCKETa/ceph.conf
2015-11-06 17:18:01,174 4558 [boto][DEBUG ] Signature:
AWS us_access_key:p6+AscqnndOpcWfJMBO7ADDpNek=
2015-11-06 17:18:01,175 4558 [boto][DEBUG ] url = 
'http://node1-west.ceph.com/BUCKETa/ceph.conf'
params={'rgwx-op-id': 'admin1:4463:2', 'rgwx-source-zone': u'us-east', 
'rgwx-client-id': 'radosgw-agent'}
headers={'Content-Length': '0', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 
Linux/3.13.0-66-generic', 'x-amz-copy-source': 'BUCKETa/ceph.conf', 'Date': 
'Fri, 06 Nov 2015 09
:18:01 GMT', 'Content-Type': 'application/json; charset=UTF-8', 
'Authorization': 'AWS us_access_key:p6+AscqnndOpcWfJMBO7ADDpNek='}
data=None
2015-11-06 17:18:01,175 4558 [boto][DEBUG ] Method: PUT
2015-11-06 17:18:01,175 4558 [boto][DEBUG ] Path: 
/BUCKETa/ceph.conf?rgwx-op-id=admin1%3A4463%3A2&rgwx-source-zone=us-east&rgwx-client-id=radosgw-agent
2015-11-06 17:18:01,175 4558 [boto][DEBUG ] Data:
2015-11-06 17:18:01,176 4558 [boto][DEBUG ] Headers: {'Content-Type': 
'application/json; charset=UTF-8', 'x-amz-copy-source': 'BUCKETa/ceph.conf'}
2015-11-06 17:18:01,176 4558 [boto][DEBUG ] Host: node1-west.ceph.com
2015-11-06 17:18:01,176 4558 [boto][DEBUG ] Port: 80
2015-11-06 17:18:01,176 4558 [boto][DEBUG ] Params: {'rgwx-op-id': 
'admin1%3A4463%3A2', 'rgwx-source-zone': 'us-east', 'rgwx-client-id': 
'radosgw-agent'}
2015-11-06 17:18:01,177 4558 [boto][DEBUG ] Token: None
2015-11-06 17:18:01,177 4558 [boto][DEBUG ] StringToSign:
PUT

application/json; charset=UTF-8
Fri, 06 Nov 2015 09:18:01 GMT
x-amz-copy-source:BUCKETa/ceph.conf
/BUCKETa/ceph.conf
2015-11-06 17:18:01,177 4558 [boto][DEBUG ] Signature:
AWS us_access_key:p6+AscqnndOpcWfJMBO7ADDpNek=
2015-11-06 17:18:01,203 4558 [radosgw_agent.worker][DEBUG ] object 
"BUCKETa/ceph.conf" not found on master, deleting from secondary
2015-11-06 17:18:01,203 4558 [boto][DEBUG ] path=/BUCKETa/
2015-11-06 17:18:01,203 4558 [boto][DEBUG ] auth_path=/BUCKETa/
2015-11-06 17:18:01,203 4558 [boto][DEBUG ] path=/BUCKETa/?max-keys=0
2015-11-06 17:18:01,204 4558 [boto][DEBUG ] auth_path=/BUCKETa/?max-keys=0
2015-11-06 17:18:01,204 4558 [boto][DEBUG ] Method: GET
2015-11-06 17:18:01,204 4558 [boto][DEBUG ] Path: /BUCKETa/?max-keys=0
2015-11-06 17:18:01,204 4558 [boto][DEBUG ] Data:
2015-11-06 17:18:01,204 4558 [boto][DEBUG ] Headers: {}
2015-11-06 17:18:01,205 4558 [boto][DEBUG ] Host: node1-west.ceph.com
2015-11-06 17:18:01,205 4558 [boto][DEBUG ] Port: 80
2015-11-06 17:18:01,205 4558 [boto][DEBUG ] Params: {}
2015-11-06 17:18:01,206 4558 [boto][DEBUG ] establishing HTTP connection: 
kwargs={'port': 80, 'timeout': 70}
2015-11-06 17:18:01,206 4558 [boto][DEBUG ] Token: None
2015-11-06 17:18:01,206 4558 [boto][DEBUG ] StringToSign:
-

Any help would be much appreciated.

Best Regards,
Wdhwang


-Original Message-
From: Craig Lewis [mailto:cle...@centraldesktop.com] 
Sent: Saturday, November 07, 2015 3:59 AM
To: WD Hwang/WHQ/Wistron
Cc: Ceph Users
Subject: Re: [ceph-users] Federated gateways

You are updating [radosgw-admin] in ceph.conf, in steps 1.4 and 2.4?

I recall restarting things more often.  IIRC, I would restart everything after 
every regionmap update or a ceph.conf update.

I man

Re: [ceph-users] v9.2.0 Infernalis released

[Alexandre DERUMIER]

Hi,

debian repository seem to miss librbd1 package for debian jessie

http://download.ceph.com/debian-infernalis/pool/main/c/ceph/

(ubuntu trusty librbd1 is present)


- Mail original -
De: "Sage Weil" 
À: ceph-annou...@ceph.com, "ceph-devel" , 
"ceph-users" , ceph-maintain...@ceph.com
Envoyé: Vendredi 6 Novembre 2015 23:05:54
Objet: [ceph-users] v9.2.0 Infernalis released

[I'm going to break my own rule and do this on a Friday only because this 
has been built and in the repos for a couple of days now; I've just been 
traveling and haven't had time to announce it.] 

This major release will be the foundation for the next stable series. 
There have been some major changes since v0.94.x Hammer, and the 
upgrade process is non-trivial. Please read these release notes carefully. 

Major Changes from Hammer 
- 

- General: 

* Ceph daemons are now managed via systemd (with the exception of 
Ubuntu Trusty, which still uses upstart). 
* Ceph daemons run as 'ceph' user instead root. 
* On Red Hat distros, there is also an SELinux policy. 

- RADOS: 

* The RADOS cache tier can now proxy write operations to the base 
tier, allowing writes to be handled without forcing migration of 
an object into the cache. 
* The SHEC erasure coding support is no longer flagged as 
experimental. SHEC trades some additional storage space for faster 
repair. 
* There is now a unified queue (and thus prioritization) of client 
IO, recovery, scrubbing, and snapshot trimming. 
* There have been many improvements to low-level repair tooling 
(ceph-objectstore-tool). 
* The internal ObjectStore API has been significantly cleaned up in order 
to faciliate new storage backends like NewStore. 

- RGW: 

* The Swift API now supports object expiration. 
* There are many Swift API compatibility improvements. 

- RBD: 

* The ``rbd du`` command shows actual usage (quickly, when 
object-map is enabled). 
* The object-map feature has seen many stability improvements. 
* Object-map and exclusive-lock features can be enabled or disabled 
dynamically. 
* You can now store user metadata and set persistent librbd options 
associated with individual images. 
* The new deep-flatten features allows flattening of a clone and all 
of its snapshots. (Previously snapshots could not be flattened.) 
* The export-diff command command is now faster (it uses aio). There is also 
a new fast-diff feature. 
* The --size argument can be specified with a suffix for units 
(e.g., ``--size 64G``). 
* There is a new ``rbd status`` command that, for now, shows who has 
the image open/mapped. 

- CephFS: 

* You can now rename snapshots. 
* There have been ongoing improvements around administration, diagnostics, 
and the check and repair tools. 
* The caching and revocation of client cache state due to unused 
inodes has been dramatically improved. 
* The ceph-fuse client behaves better on 32-bit hosts. 

Distro compatibility 
 

We have decided to drop support for many older distributions so that we can 
move to a newer compiler toolchain (e.g., C++11). Although it is still possible 
to build Ceph on older distributions by installing backported development 
tools, 
we are not building and publishing release packages for ceph.com. 

We now build packages for: 

* CentOS 7 or later. We have dropped support for CentOS 6 (and other 
RHEL 6 derivatives, like Scientific Linux 6). 
* Debian Jessie 8.x or later. Debian Wheezy 7.x's g++ has incomplete 
support for C++11 (and no systemd). 
* Ubuntu Trusty 14.04 or later. Ubuntu Precise 12.04 is no longer 
supported. 
* Fedora 22 or later. 

Upgrading from Firefly 
-- 

Upgrading directly from Firefly v0.80.z is not recommended. It is 
possible to do a direct upgrade, but not without downtime. We 
recommend that clusters are first upgraded to Hammer v0.94.4 or a 
later v0.94.z release; only then is it possible to upgrade to 
Infernalis 9.2.z for an online upgrade (see below). 

To do an offline upgrade directly from Firefly, all Firefly OSDs must 
be stopped and marked down before any Infernalis OSDs will be allowed 
to start up. This fencing is enforced by the Infernalis monitor, so 
use an upgrade procedure like: 

1. Upgrade Ceph on monitor hosts 
2. Restart all ceph-mon daemons 
3. Upgrade Ceph on all OSD hosts 
4. Stop all ceph-osd daemons 
5. Mark all OSDs down with something like:: 
ceph osd down `seq 0 1000` 
6. Start all ceph-osd daemons 
7. Upgrade and restart remaining daemons (ceph-mds, radosgw) 

Upgrading from Hammer 
- 

* For all distributions that support systemd (CentOS 7, Fedora, Debian 
Jessie 8.x, OpenSUSE), ceph daemons are now managed using native systemd 
files instead of the legacy sysvinit scripts. For example,:: 

systemctl start ceph.target # start all daemons 
systemctl status ceph-osd@12 # check status of osd.12 

The main notable distro that is *not* yet using systemd is Ubuntu trusty 
14.04. (The next Ubunt

Re: [ceph-users] v9.2.0 Infernalis released

[Francois Lafont]

Hi,

I have just upgraded a cluster to 9.2.0 from 9.1.0.
All seems to be well except I have this little error
message :

~# ceph tell mon.* version --format plain
mon.1: ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314e58)
mon.2: ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314e58)
mon.3: Error ENOENT: problem getting command descriptions from mon.3 < 
Here. ;)
mon.3: problem getting command descriptions from mon.3

Except this little message, all seems to be fine.

~# ceph -s
cluster f875b4c1-535a-4f17-9883-2793079d410a
 health HEALTH_OK
 monmap e3: 3 mons at 
{1=10.0.2.101:6789/0,2=10.0.2.102:6789/0,3=10.0.2.103:6789/0}
election epoch 104, quorum 0,1,2 1,2,3
 mdsmap e66: 1/1/1 up {0=3=up:active}, 2 up:standby
 osdmap e256: 15 osds: 15 up, 15 in
flags sortbitwise
  pgmap v1094: 192 pgs, 3 pools, 31798 bytes data, 20 objects
560 MB used, 55862 GB / 55863 GB avail
 192 active+clean

I have tried to restart mon.3 but no success. Should I ignore the
message?

-- 
François Lafont
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v9.2.0 Infernalis released

[Francois Lafont]

On 09/11/2015 06:28, Francois Lafont wrote:
 
> I have just upgraded a cluster to 9.2.0 from 9.1.0.
> All seems to be well except I have this little error
> message :
> 
> ~# ceph tell mon.* version --format plain
> mon.1: ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314e58)
> mon.2: ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314e58)
> mon.3: Error ENOENT: problem getting command descriptions from mon.3 < 
> Here. ;)
> mon.3: problem getting command descriptions from mon.3
> 
> Except this little message, all seems to be fine.
> 
> ~# ceph -s
> cluster f875b4c1-535a-4f17-9883-2793079d410a
>  health HEALTH_OK
>  monmap e3: 3 mons at 
> {1=10.0.2.101:6789/0,2=10.0.2.102:6789/0,3=10.0.2.103:6789/0}
> election epoch 104, quorum 0,1,2 1,2,3
>  mdsmap e66: 1/1/1 up {0=3=up:active}, 2 up:standby
>  osdmap e256: 15 osds: 15 up, 15 in
> flags sortbitwise
>   pgmap v1094: 192 pgs, 3 pools, 31798 bytes data, 20 objects
> 560 MB used, 55862 GB / 55863 GB avail
>  192 active+clean
> 
> I have tried to restart mon.3 but no success. Should I ignore the
> message?

In fact, it's curious:

~# ceph mon dump
dumped monmap epoch 3
epoch 3
fsid f875b4c1-535a-4f17-9883-2793079d410a
last_changed 2015-11-04 08:25:37.700420
created 2015-11-04 07:31:38.790832
0: 10.0.2.101:6789/0 mon.1
1: 10.0.2.102:6789/0 mon.2
2: 10.0.2.103:6789/0 mon.3


~# ceph tell mon.1 version 
ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314e58)

~# ceph tell mon.2 version 
ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314e58)

~# ceph tell mon.3 version 
Error ENOENT: problem getting command descriptions from mon.3
[2] root@ceph03 06:35 ~

~# ceph tell mon.0 version 
ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314e58)

Concerning monitors, I have this in my ceph.conf:

 [mon.1]
  host = ceph01
  mon addr = 10.0.2.101

[mon.2]
  host = ceph02
  mon addr = 10.0.2.102

[mon.3]
  host = ceph03
  mon addr = 10.0.2.103

So the ID of my monitors are 1, 2, 3. But there is a little
problem because I have :

~# ceph tell mon.0 version 
ceph version 9.2.0 (17df5d2948d929e997b9d320b228caffc8314e58)

So what is this mon.0 ??

-- 
François Lafont
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Erasure coded pools and 'feature set mismatch' issue

[Bogdan SOLGA]

Hello Greg!

Thank you for your advice, first of all!

I have tried to adjust the Ceph tunables detailed in this
 page, but
without success. I have tried both '*ceph osd crush tunables optimal*'
and '*ceph
osd crush tunables hammer*', but both lead to the same 'feature set
mismatch' issue, whenever I tried to create a new RBD image, afterwards.
The only way I could restore the proper functioning of the cluster was to
set the tunables to default ('*ceph osd crush tunables default*'), which
are the default values for a new cluster.

So... either I'm doing something incompletely, or I'm doing something
wrong. Any further advice on how to be able to use EC pools is highly
welcomed.

Thank you!

Regards,
Bogdan


On Mon, Nov 9, 2015 at 12:20 AM, Gregory Farnum  wrote:

> With that release it shouldn't be the EC pool causing trouble; it's the
> CRUSH tunables also mentioned in that thread. Instructions should be
> available in the docs for using older tunable that are compatible with
> kernel 3.13.
> -Greg
>
>
> On Saturday, November 7, 2015, Bogdan SOLGA 
> wrote:
>
>> Hello, everyone!
>>
>> I have recently created a Ceph cluster (v 0.94.5) on Ubuntu 14.04.3 and I
>> have created an erasure coded pool, which has a caching pool in front of it.
>>
>> When trying to map RBD images, regardless if they are created in the rbd
>> or in the erasure coded pool, the operation fails with 'rbd: map failed:
>> (5) Input/output error'. Searching the internet for a solution... I came
>> across this
>> 
>> page, which seems to detail exactly the same issue - a 'misunderstanding'
>> between erasure coded pools and the 3.13 kernel (used by Ubuntu).
>>
>> Can you please advise on a fix for that issue? As we would prefer to use
>> erasure coded pools, the only solutions which came into my mind were:
>>
>>- upgrade to the Infernalis Ceph release, although I'm not sure the
>>issue is fixed in that version;
>>
>>
>>- upgrade the kernel (on all the OSDs and Ceph clients) to the 3.14+
>>kernel;
>>
>> Any better / easier solution is highly appreciated.
>>
>> Regards,
>>
>> Bogdan
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Erasure coded pools and 'feature set mismatch' issue

[Adam Tygart]

The problem is that "hammer" tunables (i.e. "optimal" in v0.94.x) are
incompatible with the kernel interfaces before Linux 4.1 (namely due
to straw2 buckets). To make use of the kernel interfaces in 3.13, I
believe you'll need "firefly" tunables.

--
Adam

On Sun, Nov 8, 2015 at 11:48 PM, Bogdan SOLGA  wrote:
> Hello Greg!
>
> Thank you for your advice, first of all!
>
> I have tried to adjust the Ceph tunables detailed in this page, but without
> success. I have tried both 'ceph osd crush tunables optimal' and 'ceph osd
> crush tunables hammer', but both lead to the same 'feature set mismatch'
> issue, whenever I tried to create a new RBD image, afterwards. The only way
> I could restore the proper functioning of the cluster was to set the
> tunables to default ('ceph osd crush tunables default'), which are the
> default values for a new cluster.
>
> So... either I'm doing something incompletely, or I'm doing something wrong.
> Any further advice on how to be able to use EC pools is highly welcomed.
>
> Thank you!
>
> Regards,
> Bogdan
>
>
> On Mon, Nov 9, 2015 at 12:20 AM, Gregory Farnum  wrote:
>>
>> With that release it shouldn't be the EC pool causing trouble; it's the
>> CRUSH tunables also mentioned in that thread. Instructions should be
>> available in the docs for using older tunable that are compatible with
>> kernel 3.13.
>> -Greg
>>
>>
>> On Saturday, November 7, 2015, Bogdan SOLGA 
>> wrote:
>>>
>>> Hello, everyone!
>>>
>>> I have recently created a Ceph cluster (v 0.94.5) on Ubuntu 14.04.3 and I
>>> have created an erasure coded pool, which has a caching pool in front of it.
>>>
>>> When trying to map RBD images, regardless if they are created in the rbd
>>> or in the erasure coded pool, the operation fails with 'rbd: map failed: (5)
>>> Input/output error'. Searching the internet for a solution... I came across
>>> this page, which seems to detail exactly the same issue - a
>>> 'misunderstanding' between erasure coded pools and the 3.13 kernel (used by
>>> Ubuntu).
>>>
>>> Can you please advise on a fix for that issue? As we would prefer to use
>>> erasure coded pools, the only solutions which came into my mind were:
>>>
>>> upgrade to the Infernalis Ceph release, although I'm not sure the issue
>>> is fixed in that version;
>>>
>>> upgrade the kernel (on all the OSDs and Ceph clients) to the 3.14+
>>> kernel;
>>>
>>> Any better / easier solution is highly appreciated.
>>>
>>> Regards,
>>>
>>> Bogdan
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v9.2.0 Infernalis released

[Dan van der Ster]

On Mon, Nov 9, 2015 at 6:39 AM, Francois Lafont  wrote:
> 0: 10.0.2.101:6789/0 mon.1
> 1: 10.0.2.102:6789/0 mon.2
> 2: 10.0.2.103:6789/0 mon.3

Mon rank vs. Mon id is super confusing, especially if you use a number
for the mon id. In your case:

0 -> mon.0 (which has id mon.1)
1 -> mon.1 (which has id mon.2)
...

We renamed our mons to mon.`hostname -s` and it's no longer confusing.

Cheers, Dan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com