Hi Jaime
On 11/08/14 10:28, Jaime Melis wrote:
By the way, unfortunately this won't make it 4.8 as it's already
feature freeze (and about to be released). I know you reported this a
while back and in the end we didn't include it because we weren't
sure. Sorry for not revisiting this issue earlier :S
Before providing the patch for the docs, let me discuss this with the
team, see if we can include it for 4.8.1
Ok don't worry, we will include everything into a single new pull
request to simplify the patch.
Cheers
Alvaro
...
cheers,
Jaime
On Mon, Aug 11, 2014 at 10:25 AM, Jaime Melis <[email protected]
<mailto:[email protected]>> wrote:
Hi Alvaro,
Could you also provide a patch for the documentation?
On Mon, Aug 11, 2014 at 10:22 AM, Alvaro Simon Garcia
<[email protected] <mailto:[email protected]>>
wrote:
Hi Jaime, Javier
Do you think that it would be possible to include these ceph
patches into next 4.8 release? At this moment we are applying
theses patches by hand in our configuration to use different
Ceph pools.
https://github.com/OpenNebula/one/pull/27
https://github.com/OpenNebula/one/pull/28
https://github.com/OpenNebula/one/pull/29
https://github.com/OpenNebula/one/pull/30
https://github.com/OpenNebula/one/pull/31
Cheers
Alvaro
On 17/07/14 14:11, Alvaro Simon Garcia wrote:
Hi Jaime
Sorry for the late reply, I didn't see your mail before
because I was included in cc. With this patch you don't need
to include a new user into your ceph keyring. If you want to
use a ceph datastore you only need to include the user and
pool into the datastore template that's all. In our case we
use livbirt datastore and we have created two ceph datastore
for testing porposes (each one use a different pool as well):
$ onedatastore show ceph
DATASTORE 103 INFORMATION
ID : 103
NAME : ceph
USER : oneadmin
GROUP : oneadmin
CLUSTER : -
TYPE : IMAGE
DS_MAD : ceph
TM_MAD : ceph
BASE PATH : /var/lib/one//datastores/103
DISK_TYPE : RBD
DATASTORE CAPACITY
TOTAL: : 87.6T
FREE: : 59.2T
USED: : 28.4T
LIMIT: : -
PERMISSIONS
OWNER : um-
GROUP : u--
OTHER : ---
DATASTORE TEMPLATE
BASE_PATH="/var/lib/one//datastores/"
BRIDGE_LIST="hyp004.cubone.os"
CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
CEPH_USER="libvirt"
CLONE_TARGET="SELF"
DATASTORE_CAPACITY_CHECK="yes"
DISK_TYPE="RBD"
DS_MAD="ceph"
LN_TARGET="NONE"
POOL_NAME="one"
TM_MAD="ceph"
IMAGES
7
8
31
32
34
35
and the second one:
$ onedatastore show "ceph two"
DATASTORE 104 INFORMATION
ID : 104
NAME : ceph two
USER : oneadmin
GROUP : oneadmin
CLUSTER : -
TYPE : IMAGE
DS_MAD : ceph
TM_MAD : ceph
BASE PATH : /var/lib/one//datastores/104
DISK_TYPE : RBD
DATASTORE CAPACITY
TOTAL: : 87.6T
FREE: : 59.2T
USED: : 28.4T
LIMIT: : -
PERMISSIONS
OWNER : um-
GROUP : u--
OTHER : ---
DATASTORE TEMPLATE
BASE_PATH="/var/lib/one//datastores/"
BRIDGE_LIST="hyp004.cubone.os"
CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
CEPH_USER="libvirt"
CLONE_TARGET="SELF"
DATASTORE_CAPACITY_CHECK="yes"
DISK_TYPE="RBD"
DS_MAD="ceph"
LN_TARGET="NONE"
POOL_NAME="two"
TM_MAD="ceph"
As you can see we are using different pools in each one (so
we don't need to include the pool name into ceph.conf either,
and we are able to use several ceph clusters) and this change
simplifies the ONE conf and ceph cluster admin as well.
Cheers
Alvaro
On 03/07/14 17:08, Jaime Melis wrote:
Hi Alvaro,
thanks a lot for the contribution. I haven't tested it yet,
but the code seems to be perfect.
My main issue here is that I'm not entirely sure if we need
this. Doesn't it make sense for the oneadmin user to be in
the keyring and be able to run rados? You mentioned this,
but I'm having some trouble understanding what do you mean:
> This feature could be useful as well if you want to monitor
several datastores that are using different ceph pools and
users ids.
> You only have to include the id and pool info into the ONE
datastore template and the monitoring script will use one or
another depending on the DS conf.
With the current system you can monitor multiple ceph pools
as long as the oneadmin user has right, isn't that so?
Joel, would you like to weigh in? would you find this useful?
cheers,
Jaime
On Fri, Jun 20, 2014 at 3:23 PM, Javier Fontan
<[email protected] <mailto:[email protected]>> wrote:
Great, thanks! Our ceph dev is traveling right now, he
will review the
patches when he arrives.
Cheers
On Thu, Jun 19, 2014 at 3:21 PM, Alvaro Simon Garcia
<[email protected]
<mailto:[email protected]>> wrote:
> Hi Javier
>
> We have modified the ceph monitor to take into account
the datastore ceph
> pool and also the ceph user. This is a generic
solution that could be useful
> for other datacenters as well, we have created a pull
request in github if
> you are agree about this change and you want to
include it in the next
> release.
>
> https://github.com/OpenNebula/one/pull/27
>
> We only have modified these lines in
> /var/lib/one/remotes/datastore/ceph/monitor:
>
>> --- monitor.orig.190614 2014-06-19
14:35:24.022755989 +0200
>> +++ monitor 2014-06-19 14:49:34.043187892 +0200
>> @@ -46,10 +46,12 @@
>> while IFS= read -r -d '' element; do
>> XPATH_ELEMENTS[i++]="$element"
>> done < <($XPATH
/DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/BRIDGE_LIST \
>> - /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME)
>> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME \
>> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/CEPH_USER)
>>
>> BRIDGE_LIST="${XPATH_ELEMENTS[j++]}"
>> POOL_NAME="${XPATH_ELEMENTS[j++]:-$POOL_NAME}"
>> +CEPH_USER="${XPATH_ELEMENTS[j++]}"
>>
>> HOST=`get_destination_host`
>>
>> @@ -61,7 +63,7 @@
>> # ------------ Compute datastore usage -------------
>>
>> MONITOR_SCRIPT=$(cat <<EOF
>> -$RADOS df | $AWK '{
>> +$RADOS df -p ${POOL_NAME} --id ${CEPH_USER}| $AWK '{
>> if (\$1 == "total") {
>>
>> space = int(\$3/1024)
>
>
> CEPH_USER and POOL_NAME should be mandatory to create
the ceph datastore.
>
> Cheers
> Alvaro
>
>
>
>> Hola Javi
>>
>> Thanks a lot for your feedback. Yes we will modify
the current
>> monitoring scripts to take into account this. This
feature could be
>> useful as well if you want to monitor several
datastores that are using
>> different ceph pools and users ids. You only have to
include the id and
>> pool info into the ONE datastore template and the
monitoring script will
>> use one or another depending on the DS conf.
>>
>>
>> Cheers and thanks!
>> Alvaro
>> On 2014-06-17 14:55, Javier Fontan wrote:
>>>
>>> CEPH_USER is used when generating the libvirt/kvm
deployment file but
>>> not for DS monitoring:
>>>
>>> * Deployment file generation:
>>>
>>>
https://github.com/OpenNebula/one/blob/one-4.6/src/vmm/LibVirtDriverKVM.cc#L461
>>> * Monitoring:
>>>
https://github.com/OpenNebula/one/blob/one-4.6/src/datastore_mad/remotes/ceph/monitor#L64
>>>
>>> Ceph is not may area of expertise but you may need
to add those
>>> parameters to the monitor script a maybe to other
scripts that use the
>>> "rados" command. It may also be possible to modify
the RADOS command
>>> to have those parameters instead of modifying all
the scripts:
>>>
>>>
>>>
https://github.com/OpenNebula/one/blob/one-4.6/src/mad/sh/scripts_common.sh#L40
>>>
>>> As I said I don't know much about Ceph and it may be
those credentials
>>> could be set in a config file or so.
>>>
>>> On Tue, Jun 17, 2014 at 11:19 AM, Alvaro Simon Garcia
>>> <[email protected]
<mailto:[email protected]>> wrote:
>>>>
>>>> Hi
>>>>
>>>> We have included the admin keyring instead of
libvirt user and it
>>>> works...
>>>> that means that we can run rbd or qemu-img wihtout
the libvirt id, but
>>>> is
>>>> not the best solution. We have included the user
into datastore conf:
>>>>
>>>> CEPH_USER="libvirt"
>>>>
>>>> but it seems that is not used by opennebula at the end
>>>>
>>>> Cheers
>>>> Alvaro
>>>>
>>>>
>>>> On 2014-06-17 10:09, Alvaro Simon Garcia wrote:
>>>>>
>>>>> Hi all
>>>>>
>>>>>
>>>>> We have included our ONE nodes into Ceph cluster,
cephx auth is working
>>>>> but OpenNebula is not able to detect the free space:
>>>>>
>>>>>
>>>>>
>>>>>> $ onedatastore show 103
>>>>>> DATASTORE 103 INFORMATION
>>>>>> ID : 103
>>>>>> NAME : ceph
>>>>>> USER : oneadmin
>>>>>> GROUP : oneadmin
>>>>>> CLUSTER : -
>>>>>> TYPE : IMAGE
>>>>>> DS_MAD : ceph
>>>>>> TM_MAD : ceph
>>>>>> BASE PATH : /var/lib/one//datastores/103
>>>>>> DISK_TYPE : RBD
>>>>>>
>>>>>> DATASTORE CAPACITY
>>>>>> TOTAL: : 0M
>>>>>> FREE: : 0M
>>>>>> USED: : 0M
>>>>>> LIMIT: : -
>>>>>>
>>>>>> PERMISSIONS
>>>>>> OWNER : um-
>>>>>> GROUP : u--
>>>>>> OTHER : ---
>>>>>
>>>>>
>>>>>> $ onedatastore list
>>>>>> ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS
>>>>>> TM
>>>>>> 0 system 114.8G 85% - 0 sys -
>>>>>> shared
>>>>>> 1 default 114.9G 84% - 2 img fs
>>>>>> ssh
>>>>>> 2 files 114.9G 84% - 0 fil fs
>>>>>> ssh
>>>>>> 103 ceph 0M - - 0 img ceph
>>>>>> ceph
>>>>>
>>>>>
>>>>> but if we run rados as oneadmin user:
>>>>>
>>>>>> $ rados df -p one --id libvirt
>>>>>> pool name category KB
objects clones
>>>>>> degraded unfound rd rd KB
wr wr
>>>>>> KB
>>>>>> one - 0 0 0
>>>>>> 0 0 0 0 0
0
>>>>>> total used 1581852 37
>>>>>> total avail 140846865180
>>>>>> total space 140848447032
>>>>>
>>>>>
>>>>> It's working correctly (we are using one pool and
libvirt ceph id)
>>>>>
>>>>> the oned.log only shows this info:
>>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring
datastore default (1)
>>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring
datastore files (2)
>>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring
datastore ceph (103)
>>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore
default (1) successfully
>>>>> monitored.
>>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore files
(2) successfully
>>>>> monitored.
>>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore ceph
(103) successfully
>>>>> monitored.
>>>>>
>>>>> Any clue about how to debug this issue?
>>>>>
>>>>> Thanks in advance!
>>>>> Alvaro
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> [email protected]
<mailto:[email protected]>
>>>>>
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> [email protected]
<mailto:[email protected]>
>>>>
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>
>>>
>> _______________________________________________
>> Users mailing list
>> [email protected]
<mailto:[email protected]>
>>
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>
--
Javier Fontán Muiños
Developer
OpenNebula - Flexible Enterprise Cloud Made Simple
www.OpenNebula.org <http://www.OpenNebula.org> |
@OpenNebula | github.com/jfontan <http://github.com/jfontan>
_______________________________________________
Users mailing list
[email protected]
<mailto:[email protected]>
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
--
Jaime Melis
Project Engineer
OpenNebula - Flexible Enterprise Cloud Made Simple
www.OpenNebula.org <http://www.OpenNebula.org> |
[email protected] <mailto:[email protected]>
_______________________________________________
Users mailing list
[email protected] <mailto:[email protected]>
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
--
Jaime Melis
Project Engineer
OpenNebula - Flexible Enterprise Cloud Made Simple
www.OpenNebula.org <http://www.OpenNebula.org> |
[email protected] <mailto:[email protected]>
--
Jaime Melis
Project Engineer
OpenNebula - Flexible Enterprise Cloud Made Simple
www.OpenNebula.org <http://www.OpenNebula.org> | [email protected]
<mailto:[email protected]>
_______________________________________________
Users mailing list
[email protected]
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
