Re: [ceph-users] Fwd: [Ceph-community]Improve Read Performance

[Le Quang Long]

Thanks for your reply.

I intend use Ceph RBD as shared storage for Oracle Database RAC.
My Ceph deployment has 3 nodes with 8 1TB 15k SAS per node, I do not have
SSD at the moment, so I design every SAS will be Journal and OSD.

Can you suggest me a way to get highest performance for Oracle Cluster with
this deployment?

Many thanks.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: [Ceph-community]Improve Read Performance

[Shinobu]

How heavy transaction are you expecting?

 Shinobu

On Sun, Aug 30, 2015 at 8:33 PM, Le Quang Long 
wrote:

> Thanks for your reply.
>
> I intend use Ceph RBD as shared storage for Oracle Database RAC.
> My Ceph deployment has 3 nodes with 8 1TB 15k SAS per node, I do not have
> SSD at the moment, so I design every SAS will be Journal and OSD.
>
> Can you suggest me a way to get highest performance for Oracle Cluster
> with this deployment?
>
> Many thanks.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
Email:
 shin...@linux.com
 ski...@redhat.com

Life with Distributed Computational System based on OpenSource

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: [Ceph-community]Improve Read Performance

[Somnath Roy]

And what kind of performance are you looking for?
I assume your workload will be small block random read/write?
Btw, without SSD journal write performance will be very bad specially when your 
cluster is small..

Sent from my iPhone

On Aug 30, 2015, at 4:33 AM, Le Quang Long 
mailto:longlq.openst...@gmail.com>> wrote:


Thanks for your reply.

I intend use Ceph RBD as shared storage for Oracle Database RAC.
My Ceph deployment has 3 nodes with 8 1TB 15k SAS per node, I do not have SSD 
at the moment, so I design every SAS will be Journal and OSD.

Can you suggest me a way to get highest performance for Oracle Cluster with 
this deployment?

Many thanks.



PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] OSD activate hangs

[pavana bhat]

Hi,

I am trying to ceph-deploy with Hammer on rhel7. While trying to activate
the OSD using ceph-deploy on admin-node, the below step hangs. I tried to
run it manually on the osd-node and tried tracing using "python -m trace
--trace" . It looks like it is stuck in some threading.py code. Can someone
please help?

[*ceph-vm-osd1*][*WARNIN*] INFO:ceph-disk:Running command: /usr/bin/ceph
--cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise
c2d19639-f3ec-447d-9a7c-a180a226dded

[*ceph-vm-osd1*][*WARNIN*] No data was received after 300 seconds,
disconnecting...



*Manual run with --verbose option:*


[cloud-user@ceph-vm-osd1 ~]$ sudo /usr/bin/ceph --verbose --cluster ceph
--name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring osd create
c2d19639-f3ec-447d-9a7c-a180a226dded

parsed_args: Namespace(admin_socket=None, admin_socket_nope=None,
cephconf=None, client_id=None, client_name='client.bootstrap-osd',
cluster='ceph', cluster_timeout=None, completion=False, help=False,
input_file=None, output_file=None, output_format=None, status=False,
verbose=True, version=False, watch=False, watch_debug=False,
watch_error=False, watch_info=False, watch_sec=False, watch_warn=False),
childargs: ['--keyring', '/var/lib/ceph/bootstrap-osd/ceph.keyring', 'osd',
'create', 'c2d19639-f3ec-447d-9a7c-a180a226dded']

^CError connecting to cluster: InterruptedOrTimeoutError


*Manual run with python -m trace --trace :*

<>


 --- modulename: threading, funcname: _note

threading.py(64): if self.__verbose:

threading.py(946): self.__block.acquire()

threading.py(947): try:

threading.py(948): if timeout is None:

threading.py(954): deadline = _time() + timeout

threading.py(955): while not self.__stopped:

threading.py(956): delay = deadline - _time()

threading.py(957): if delay <= 0:

threading.py(961): self.__block.wait(delay, balancing)

 --- modulename: threading, funcname: wait

threading.py(331): if not self._is_owned():

 --- modulename: threading, funcname: _is_owned

threading.py(302): if self.__lock.acquire(0):

threading.py(306): return True

threading.py(333): waiter = _allocate_lock()

threading.py(334): waiter.acquire()

threading.py(335): self.__waiters.append(waiter)

threading.py(336): saved_state = self._release_save()

 --- modulename: threading, funcname: _release_save

threading.py(294): self.__lock.release()   # No state to
save

threading.py(337): try:# restore state no matter what (e.g.,
KeyboardInterrupt)

threading.py(338): if timeout is None:

threading.py(348): endtime = _time() + timeout

threading.py(349): delay = 0.0005 # 500 us -> initial delay
of 1 ms

threading.py(350): while True:

threading.py(351): gotit = waiter.acquire(0)

threading.py(352): if gotit:

threading.py(354): remaining = endtime - _time()

threading.py(355): if remaining <= 0:

threading.py(357): if balancing:

threading.py(358): delay = min(delay * 2,
remaining, 0.05)

threading.py(361): _sleep(delay)

 --- modulename: threading, funcname: _note

threading.py(64): if self.__verbose:

 --- modulename: threading, funcname: __stop

threading.py(870): if not hasattr(self, '_Thread__block'):

threading.py(872): self.__block.acquire()

threading.py(873): self.__stopped = True

threading.py(874): self.__block.notify_all()

 --- modulename: threading, funcname: notifyAll

threading.py(409): self.notify(len(self.__waiters))

 --- modulename: threading, funcname: notify

threading.py(385): if not self._is_owned():

 --- modulename: threading, funcname: _is_owned

threading.py(302): if self.__lock.acquire(0):

threading.py(306): return True

threading.py(387): __waiters = self.__waiters

threading.py(388): waiters = __waiters[:n]

threading.py(389): if not waiters:

threading.py(393): self._note("%s.notify(): notifying %d waiter%s",
self, n,

threading.py(394):n!=1 and "s" or "")

 --- modulename: threading, funcname: _note

threading.py(64): if self.__verbose:

threading.py(395): for waiter in waiters:

threading.py(396): waiter.release()

threading.py(397): try:

threading.py(398): __waiters.remove(waiter)

threading.py(395): for waiter in waiters:

threading.py(875): self.__block.release()

threading.py(350): while True:

threading.py(351): gotit = waiter.acquire(0)

threading.py(352):   

Re: [ceph-users] Ceph-deploy error

[pavana bhat]

In case someone else runs into the same issue in future:

I came out of this issue by installing epel-release before installing
ceph-deploy. If the order of installation is ceph-deploy followed by
epel-release, the issue is being hit.

Thanks,
Pavana

On Sat, Aug 29, 2015 at 10:02 AM, pavana bhat 
wrote:

> Hi,
>
> I'm trying to install ceph for the first time following the quick
> installation guide. I'm getting the below error, can someone please help?
>
> ceph-deploy install --release=firefly ceph-vm-mon1
>
> [*ceph_deploy.conf*][*DEBUG* ] found configuration file at:
> /home/cloud-user/.cephdeploy.conf
>
> [*ceph_deploy.cli*][*INFO*  ] Invoked (1.5.28): /usr/bin/ceph-deploy
> install --release=firefly ceph-vm-mon1
>
> [*ceph_deploy.cli*][*INFO*  ] ceph-deploy options:
>
> [*ceph_deploy.cli*][*INFO*  ]  verbose   : False
>
> [*ceph_deploy.cli*][*INFO*  ]  testing   : None
>
> [*ceph_deploy.cli*][*INFO*  ]  cd_conf   :
> 
>
> [*ceph_deploy.cli*][*INFO*  ]  cluster   : ceph
>
> [*ceph_deploy.cli*][*INFO*  ]  install_mds   : False
>
> [*ceph_deploy.cli*][*INFO*  ]  stable: None
>
> [*ceph_deploy.cli*][*INFO*  ]  default_release   : False
>
> [*ceph_deploy.cli*][*INFO*  ]  username  : None
>
> [*ceph_deploy.cli*][*INFO*  ]  adjust_repos  : True
>
> [*ceph_deploy.cli*][*INFO*  ]  func  :  install at 0x7f34b410e938>
>
> [*ceph_deploy.cli*][*INFO*  ]  install_all   : False
>
> [*ceph_deploy.cli*][*INFO*  ]  repo  : False
>
> [*ceph_deploy.cli*][*INFO*  ]  host  :
> ['ceph-vm-mon1']
>
> [*ceph_deploy.cli*][*INFO*  ]  install_rgw   : False
>
> [*ceph_deploy.cli*][*INFO*  ]  repo_url  : None
>
> [*ceph_deploy.cli*][*INFO*  ]  ceph_conf : None
>
> [*ceph_deploy.cli*][*INFO*  ]  install_osd   : False
>
> [*ceph_deploy.cli*][*INFO*  ]  version_kind  : stable
>
> [*ceph_deploy.cli*][*INFO*  ]  install_common: False
>
> [*ceph_deploy.cli*][*INFO*  ]  overwrite_conf: False
>
> [*ceph_deploy.cli*][*INFO*  ]  quiet : False
>
> [*ceph_deploy.cli*][*INFO*  ]  dev   : master
>
> [*ceph_deploy.cli*][*INFO*  ]  local_mirror  : None
>
> [*ceph_deploy.cli*][*INFO*  ]  release   : firefly
>
> [*ceph_deploy.cli*][*INFO*  ]  install_mon   : False
>
> [*ceph_deploy.cli*][*INFO*  ]  gpg_url   : None
>
> [*ceph_deploy.install*][*DEBUG* ] Installing stable version firefly on
> cluster ceph hosts ceph-vm-mon1
>
> [*ceph_deploy.install*][*DEBUG* ] Detecting platform for host
> ceph-vm-mon1 ...
>
> [*ceph-vm-mon1*][*DEBUG* ] connection detected need for sudo
>
> [*ceph-vm-mon1*][*DEBUG* ] connected to host: ceph-vm-mon1
>
> [*ceph-vm-mon1*][*DEBUG* ] detect platform information from remote host
>
> [*ceph-vm-mon1*][*DEBUG* ] detect machine type
>
> [*ceph_deploy.install*][*INFO*  ] Distro info: Red Hat Enterprise Linux
> Server 7.1 Maipo
>
> [*ceph-vm-mon1*][*INFO*  ] installing Ceph on ceph-vm-mon1
>
> [*ceph-vm-mon1*][*INFO*  ] Running command: sudo yum clean all
>
> [*ceph-vm-mon1*][*DEBUG* ] Loaded plugins: fastestmirror, priorities
>
> [*ceph-vm-mon1*][*DEBUG* ] Cleaning repos: epel rhel-7-ha-rpms
> rhel-7-optional-rpms rhel-7-server-rpms
>
> [*ceph-vm-mon1*][*DEBUG* ]   : rhel-7-supplemental-rpms
>
> [*ceph-vm-mon1*][*DEBUG* ] Cleaning up everything
>
> [*ceph-vm-mon1*][*DEBUG* ] Cleaning up list of fastest mirrors
>
> [*ceph-vm-mon1*][*INFO*  ] Running command: sudo yum -y install
> epel-release
>
> [*ceph-vm-mon1*][*DEBUG* ] Loaded plugins: fastestmirror, priorities
>
> [*ceph-vm-mon1*][*DEBUG* ] Determining fastest mirrors
>
> [*ceph-vm-mon1*][*DEBUG* ]  * epel: kdeforge2.unl.edu
>
> [*ceph-vm-mon1*][*DEBUG* ]  * rhel-7-ha-rpms:
> rhel-repo.eu-biere-1.t-systems.cloud.cisco.com
>
> [*ceph-vm-mon1*][*DEBUG* ]  * rhel-7-optional-rpms:
> rhel-repo.eu-biere-1.t-systems.cloud.cisco.com
>
> [*ceph-vm-mon1*][*DEBUG* ]  * rhel-7-server-rpms:
> rhel-repo.eu-biere-1.t-systems.cloud.cisco.com
>
> [*ceph-vm-mon1*][*DEBUG* ]  * rhel-7-supplemental-rpms:
> rhel-repo.eu-biere-1.t-systems.cloud.cisco.com
>
> [*ceph-vm-mon1*][*DEBUG* ] Package epel-release-7-5.noarch already
> installed and latest version
>
> [*ceph-vm-mon1*][*DEBUG* ] Nothing to do
>
> [*ceph-vm-mon1*][*INFO*  ] Running command: sudo yum -y install
> yum-plugin-priorities
>
> [*ceph-vm-mon1*][*DEBUG* ] Loaded plugins: fastestmirror, priorities
>
> [*ceph-vm-mon1*][*DEBUG* ] Loading mirror speeds from cached hostfile
>
> [*ceph-vm-mon1*][*DEBUG* ]  * epel: kdeforge2.unl.edu
>
> [*ceph-vm-mon1*][*DEBUG* ]  * rhel-7-ha-rpms:
> rhel-repo.eu-biere-1.t-systems.cloud.cisco.com
>
> [*ceph

Re: [ceph-users] Fwd: [Ceph-community]Improve Read Performance

[Le Quang Long]

Yes, I will use Ceph RBD as shared Storage for Oracle Database Cluster, so
I need high I/O read write random. With 3 nodes and 24 SAS 15K 1TB, what is
the most optimized solution to get it ?
On Aug 31, 2015 2:01 AM, "Somnath Roy"  wrote:

And what kind of performance are you looking for?
I assume your workload will be small block random read/write?
Btw, without SSD journal write performance will be very bad specially when
your cluster is small..

Sent from my iPhone

On Aug 30, 2015, at 4:33 AM, Le Quang Long 
wrote:

Thanks for your reply.

I intend use Ceph RBD as shared storage for Oracle Database RAC.
My Ceph deployment has 3 nodes with 8 1TB 15k SAS per node, I do not have
SSD at the moment, so I design every SAS will be Journal and OSD.

Can you suggest me a way to get highest performance for Oracle Cluster with
this deployment?

Many thanks.


--

PLEASE NOTE: The information contained in this electronic mail message is
intended only for the use of the designated recipient(s) named above. If
the reader of this message is not the intended recipient, you are hereby
notified that you have received this message in error and that any review,
dissemination, distribution, or copying of this message is strictly
prohibited. If you have received this communication in error, please notify
the sender by telephone or e-mail (as shown above) immediately and destroy
any and all copies of this message in your possession (whether hard copies
or electronically stored copies).
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Is Ceph appropriate for small installations?

[Lindsay Mathieson]

On 29 August 2015 at 00:53, Tony Nelson  wrote:

> I recently built a 3 node Proxmox cluster for my office.  I’d like to get
> HA setup, and the Proxmox book recommends Ceph.  I’ve been reading the
> documentation and watching videos, and I think I have a grasp on the
> basics, but I don’t need anywhere near a petabyte of storage.
>
>
>
> I’m considering servers w/ 12 drive bays, 2 SDD mirrored for the OS, 2
> SDDs for journals and the other 8 for OSDs.  I was going to purchase 3
> identical servers, and use my 3 Proxmox servers as the monitors, with of
> course GB networking in between.  Obviously this is very vague, but I’m
> just getting started on the research.
>
>
>

I run a small 3 node Proxmox cluster for our office as well with Ceph, but
I'd now recommend against using Ceph for small setups like ours.

- Maintenance headache. Ceph requires a lot of tweaking to get started and
a lot of ongoing monitoring, plus a fair bit of skill. If you're running
the show yourself (as typical in small businesses) its quite stressful.
Who's going to fix the ceph cluster when a osd goes down when you're on
holiday?

- Performance. Its terrible on small clusters. I've setup a iSCSI over ZFS
for a server and its orders of magnitude better at I/O. And I haven't even
configured multipath yet.

- Flexibility. Much much easier to expand or replace disks on my ZFS server.

The redundancy is good, I can reboot a ceph node for maintenance and it
recovers very quickly (much quicker than glusterfs), but cluster
performance suffers badly when a node is down so in practice its of limited
utility.

I'm coming to the realisation that for us performance and ease of
administration is more valuable than 100% uptime. Worst case (Storage
server dies) we could rebuild from backups in a day. Essentials could be
restored in a hour. I could experiment with ongoing ZFS replications to a
backup server that makes that even quicker.

Thats for use - your requirements may be different. And of course once you
get into truly large deployments, ceph comes into its own.




-- 
Lindsay
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Storage node refurbishing, a "freeze" OSD feature would be nice

[Christian Balzer]

Hello,

I'm about to add another storage node to small firefly cluster here and
refurbish 2 existing nodes (more RAM, different OSD disks).

Insert rant about not going to start using ceph-deploy as I would have to
set the cluster to no-in since "prepare" also activates things due to the
udev magic...

This cluster is quite at the limits of its IOPS capacity (the HW was
requested ages ago, but the mills here grind slowly and not particular
fine either), so the plan is to:

a) phase in the new node (lets call it C), one OSD at a time (in the dead
of night)
b) empty out old node A (weight 0), one OSD at a time. When
done, refurbish and bring it back in, like above.
c) repeat with 2nd old node B.

Looking at this it's obvious where the big optimization in this procedure
would be, having the ability to "freeze" the OSDs on node B.
That is making them ineligible for any new PGs while preserving their
current status. 
So that data moves from A to C (which is significantly faster than A or B)
and then back to A when it is refurbished, avoiding any heavy lifting by B.

Does that sound like something other people might find useful as well and
is it feasible w/o upsetting the CRUSH applecart?

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Fusion Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Is Ceph appropriate for small installations?

[Alexandre DERUMIER]

>>True, true. But I personally think that Ceph doesn't perform well on
>>small <10 node clusters.

Hi, I can reach 60 iops 4k read with 3 nodes (6ssd each).



- Mail original -
De: "Lindsay Mathieson" 
À: "Tony Nelson" 
Cc: "ceph-users" 
Envoyé: Lundi 31 Août 2015 03:10:14
Objet: Re: [ceph-users] Is Ceph appropriate for small installations?


On 29 August 2015 at 00:53, Tony Nelson < tnel...@starpoint.com > wrote: 




I recently built a 3 node Proxmox cluster for my office. I’d like to get HA 
setup, and the Proxmox book recommends Ceph. I’ve been reading the 
documentation and watching videos, and I think I have a grasp on the basics, 
but I don’t need anywhere near a petabyte of storage. 



I’m considering servers w/ 12 drive bays, 2 SDD mirrored for the OS, 2 SDDs for 
journals and the other 8 for OSDs. I was going to purchase 3 identical servers, 
and use my 3 Proxmox servers as the monitors, with of course GB networking in 
between. Obviously this is very vague, but I’m just getting started on the 
research. 





I run a small 3 node Proxmox cluster for our office as well with Ceph, but I'd 
now recommend against using Ceph for small setups like ours. 

- Maintenance headache. Ceph requires a lot of tweaking to get started and a 
lot of ongoing monitoring, plus a fair bit of skill. If you're running the show 
yourself (as typical in small businesses) its quite stressful. Who's going to 
fix the ceph cluster when a osd goes down when you're on holiday? 

- Performance. Its terrible on small clusters. I've setup a iSCSI over ZFS for 
a server and its orders of magnitude better at I/O. And I haven't even 
configured multipath yet. 

- Flexibility. Much much easier to expand or replace disks on my ZFS server. 

The redundancy is good, I can reboot a ceph node for maintenance and it 
recovers very quickly (much quicker than glusterfs), but cluster performance 
suffers badly when a node is down so in practice its of limited utility. 

I'm coming to the realisation that for us performance and ease of 
administration is more valuable than 100% uptime. Worst case (Storage server 
dies) we could rebuild from backups in a day. Essentials could be restored in a 
hour. I could experiment with ongoing ZFS replications to a backup server that 
makes that even quicker. 

Thats for use - your requirements may be different. And of course once you get 
into truly large deployments, ceph comes into its own. 




-- 
Lindsay 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Storage node refurbishing, a "freeze" OSD feature would be nice

[Udo Lembke]

Hi Christian,
for my setup "b" takes too long - too much data movement and stress to all 
nodes.
I have simply (with replica 3) "set noout", reinstall one node (with new 
filesystem on the OSDs, but leave them in the
crushmap) and start all OSDs (at friday night) - takes app. less than one day 
for rebuild (11*4TB 1*8TB).
Do also stress the other nodes, but less than with weigting to zero.

Udo

On 31.08.2015 06:07, Christian Balzer wrote:
> 
> Hello,
> 
> I'm about to add another storage node to small firefly cluster here and
> refurbish 2 existing nodes (more RAM, different OSD disks).
> 
> Insert rant about not going to start using ceph-deploy as I would have to
> set the cluster to no-in since "prepare" also activates things due to the
> udev magic...
> 
> This cluster is quite at the limits of its IOPS capacity (the HW was
> requested ages ago, but the mills here grind slowly and not particular
> fine either), so the plan is to:
> 
> a) phase in the new node (lets call it C), one OSD at a time (in the dead
> of night)
> b) empty out old node A (weight 0), one OSD at a time. When
> done, refurbish and bring it back in, like above.
> c) repeat with 2nd old node B.
> 
> Looking at this it's obvious where the big optimization in this procedure
> would be, having the ability to "freeze" the OSDs on node B.
> That is making them ineligible for any new PGs while preserving their
> current status. 
> So that data moves from A to C (which is significantly faster than A or B)
> and then back to A when it is refurbished, avoiding any heavy lifting by B.
> 
> Does that sound like something other people might find useful as well and
> is it feasible w/o upsetting the CRUSH applecart?
> 
> Christian
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com