Re: [ceph-users] v9.1.0 Infernalis release candidate released

Hi Goncalo, On Wed, Oct 14, 2015 at 6:51 AM, Goncalo Borges wrote: > Hi Sage... > > I've seen that the rh6 derivatives have been ruled out. > > This is a problem in our case since the OS choice in our systems is, > somehow, imposed by CERN. The experiments software is certified for SL6 and > the

Re: [ceph-users] ceph same rbd on multiple client

Hi Tyler, Thanks for your reply. We have disabled rbd_cache but still issue is persist. Please find our configuration file. # cat /etc/ceph/ceph.conf [global] fsid = 944fa0af-b7be-45a9-93ff-b9907cfaee3f mon_initial_members = integ-hm5, integ-hm6, integ-hm7 mon_host = 192.168.112.

[ceph-users] Ceph journal - isn't it a bit redundant sometimes?

Hi, I've been thinking about this for a while now - does Ceph really need a journal? Filesystems are already pretty good at committing data to disk when asked (and much faster too), we have external journals in XFS and Ext4... In a scenario where client does an ordinary write, there's no need to

Re: [ceph-users] avoid 3-mds fs laggy on 1 rejoin?

Yan, Zheng пишет: 2) I have 3 active mds now. I try, it works, keep it. Restart still problematic. multiple active MDS is not ready for production. OK, so if I run 3 active now (looks good) - better to turn back to 1? 3) Yes, more caps on master VM (4.2.3 kernel mount, there are web+mail+

Re: [ceph-users] Ceph OSD on ZFS

Hello, On Wed, 14 Oct 2015 09:25:41 +1000 Lindsay Mathieson wrote: > I'm adding a node (4 * WD RED 3TB) to our small cluster to bring it up to > replica 3. Can we assume from this node that the your current setup is something like 2 nodes with 4 drives each? > Given how much headache it has b

Re: [ceph-users] v9.1.0 Infernalis release candidate released

On Wed, 14 Oct 2015, Dan van der Ster wrote: > Hi Goncalo, > > On Wed, Oct 14, 2015 at 6:51 AM, Goncalo Borges > wrote: > > Hi Sage... > > > > I've seen that the rh6 derivatives have been ruled out. > > > > This is a problem in our case since the OS choice in our systems is, > > somehow, imposed

[ceph-users] download.ceph.com unreachable IPv6 [was: v9.1.0 Infernalis release candidate released]

On 10/13/2015 11:01 PM, Sage Weil wrote: > http://download.ceph.com/debian-testing unfortunately this site is not reachable at the moment. $ wget http://download.ceph.com/debian-testing/dists/wheezy/InRelease -O - --2015-10-14 16:06:55-- http://download.ceph.com/debian-testing/dists/wheezy/In

[ceph-users] Ceph PGs stuck creating after running force_create_pg

Hello, We recently had 2 nodes go down in our ceph cluster, one was repaired and the other had all 12 osds destroyed when it went down. We brought everything back online, there were several PGs that were showing as down+peering as well as down. After marking the failed OSDs as lost and removing

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

Jan, Journal helps FileStore to maintain the transactional integrity in the event of a crash. That's the main reason. Thanks & Regards Somnath -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Jan Schermer Sent: Wednesday, October 14, 2015 2:28

Re: [ceph-users] download.ceph.com unreachable IPv6 [was: v9.1.0 Infernalis release candidate released]

On 14-10-15 16:30, Björn Lässig wrote: > On 10/13/2015 11:01 PM, Sage Weil wrote: >> http://download.ceph.com/debian-testing > > unfortunately this site is not reachable at the moment. > > > $ wget http://download.ceph.com/debian-testing/dists/wheezy/InRelease -O - > --2015-10-14 16:06:55--

[ceph-users] What are linger_ops in the output of objecter_requests ?

Hello, debugging slow requests behaviour of our Rados Gateway, I run into this linger_ops field and I cannot understand the meaning. I would expect in the "ops" field to find slow requests stucked there. Actually most of the time I have "ops": [], and looks like ops gets empty very quickly. Howe

Re: [ceph-users] What are linger_ops in the output of objecter_requests ?

On Wed, Oct 14, 2015 at 5:13 PM, Saverio Proto wrote: > Hello, > > debugging slow requests behaviour of our Rados Gateway, I run into > this linger_ops field and I cannot understand the meaning. > > I would expect in the "ops" field to find slow requests stucked there. > Actually most of the time

Re: [ceph-users] Potential OSD deadlock?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 It seems in our situation the cluster is just busy, usually with really small RBD I/O. We have gotten things to where it doesn't happen as much in a steady state, but when we have an OSD fail (mostly from an XFS log bug we hit at least once a week),

[ceph-users] Proc for Impl XIO mess with Infernalis

Hi all, I would like to know if with this new release of Infernalis is there somewhere a procedure in order to implement xio messager with ib and ceph. Also if it's possible to change an existing ceph cluster to this kind of new setup (the existing cluster does not had any production data yet). T

[ceph-users] Cross-posting to users and ceph-devel

Hi, Not to complain or flame about it, but I see a lot of messages which are being send to both users and ceph-devel. Imho that beats the purpose of having a users and a devel list, isn't it? The problem is that messages go to both lists and users hit reply-all again and so it continues. For ex

Re: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

Hi Mark, The Async result in 128K drops quickly after some point, is that because of the testing methodology? Other conclusion looks to me like simple messenger + Jemalloc is the best practice till now as it has the same performance as async but using much less memory? -Xiaoxi

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

But that's exactly what filesystems and their own journals do already :-) Jan > On 14 Oct 2015, at 17:02, Somnath Roy wrote: > > Jan, > Journal helps FileStore to maintain the transactional integrity in the event > of a crash. That's the main reason. > > Thanks & Regards > Somnath > > -O

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

On Wed, Oct 14, 2015 at 6:05 PM, Jan Schermer wrote: > But that's exactly what filesystems and their own journals do already :-) They do it for filesystem "transactions", not ceph transactions. It's true that there is quite a bit of double journaling going on - newstore should help with that qui

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

Can you elaborate on that? I don't think there needs to be a difference. Ceph is hosting mostly filesystems, so it's all just a bunch of filesystem transactions anyway... Jan > On 14 Oct 2015, at 18:14, Ilya Dryomov wrote: > > On Wed, Oct 14, 2015 at 6:05 PM, Jan Schermer wrote: >> But that'

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

> Can you elaborate on that? I don't think there needs to be a difference. Ceph > is hosting mostly filesystems, so it's all just a bunch of filesystem > transactions anyway... > There is some additional background information here [1]. The XFS journal protects "atomic" (for lack of a better wo

Re: [ceph-users] download.ceph.com unreachable IPv6 [was: v9.1.0 Infernalis release candidate released]

On 10/14/2015 05:11 PM, Wido den Hollander wrote: > > > On 14-10-15 16:30, Björn Lässig wrote: >> On 10/13/2015 11:01 PM, Sage Weil wrote: >>> http://download.ceph.com/debian-testing >> >> unfortunately this site is not reachable at the moment. >> > wido@wido-desktop:~$ wget -6 > http://downloa

Re: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

Hi Xiaoxi, I would ignore the tails on those tests. I suspect it's just some fio processes finishing earlier than others and the associated aggregate performance dropping off. These reads tests are so fast that my original guess at reasonable volume sizes for 300 second tests appear to be o

Re: [ceph-users] Potential OSD deadlock?

On Wed, 14 Oct 2015, Robert LeBlanc wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > It seems in our situation the cluster is just busy, usually with > really small RBD I/O. We have gotten things to where it doesn't happen > as much in a steady state, but when we have an OSD fail (mo

[ceph-users] Fwd: Proc for Impl XIO mess with Infernalis

Let me be more specific about what I need in order to move forward with this kind of install: setup: 3 mon servers 8 osd servers (4 with SAS disks and SSD journal - relation 1:3) and (4 with SSD disks osd & journal on the same disk) running ceph version 0.94.3 I've already install and test *fi

Re: [ceph-users] Potential OSD deadlock?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I'm sure I have a log of a 1,000 second block somewhere, I'll have to look around for it. I'll try turning that knob and see what happens. I'll come back with the results. Thanks, - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45

Re: [ceph-users] v9.1.0 Infernalis release candidate released

A couple of questions related to this, especially since we have a hammer bug that's biting us so we're anxious to upgrade to Infernalis. 1) RE: ibrbd and librados ABI compatibility is broken. Be careful installing this RC on client machines (e.g., those running qemu). It will be fixed in the fina

Re: [ceph-users] v9.1.0 Infernalis release candidate released

On Wed, 14 Oct 2015, Kyle Hutson wrote: > A couple of questions related to this, especially since we have a hammer > bug that's biting us so we're anxious to upgrade to Infernalis. Which bug? We want to fix hammer, too! > 1) RE: ibrbd and librados ABI compatibility is broken. Be careful install

Re: [ceph-users] v9.1.0 Infernalis release candidate released

> Which bug? We want to fix hammer, too! This one: https://www.mail-archive.com/ceph-users@lists.ceph.com/msg23915.html (Adam sits about 5' from me.) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ce

Re: [ceph-users] v9.1.0 Infernalis release candidate released

On Wed, 14 Oct 2015, Kyle Hutson wrote: > > Which bug?  We want to fix hammer, too! > > This > one: https://www.mail-archive.com/ceph-users@lists.ceph.com/msg23915.html > > (Adam sits about 5' from me.) Oh... that fix is already in the hammer branch and will be in 0.94.4. Since you have to go

Re: [ceph-users] v9.1.0 Infernalis release candidate released

Nice! Thanks! On Wed, Oct 14, 2015 at 1:23 PM, Sage Weil wrote: > On Wed, 14 Oct 2015, Kyle Hutson wrote: > > > Which bug? We want to fix hammer, too! > > > > This > > one: > https://www.mail-archive.com/ceph-users@lists.ceph.com/msg23915.html > > > > (Adam sits about 5' from me.) > > Oh... tha

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

FileSystem like XFS guarantees a single file write but in Ceph transaction we are touching file/xattrs/leveldb (omap), so no way filesystem can guarantee that transaction. That's why FileStore has implemented a write_ahead journal. Basically, it is writing the entire transaction object there and

Re: [ceph-users] CephFS "corruption" -- Nulled bytes

Not thoroughly tested, but I've got a quick and dirty script to fix these up. Worst case scenario, it does nothing. In my limited testing, the contents of the files comes back without a remount of cephfs. https://github.com/BeocatKSU/admin/blob/master/ec_cephfs_fixer.py -- Adam On Thu, Oct 8, 20

Re: [ceph-users] download.ceph.com unreachable IPv6 [was: v9.1.0 Infernalis release candidate released]

On 10/14/2015 06:50 PM, Björn Lässig wrote: > On 10/14/2015 05:11 PM, Wido den Hollander wrote: >> >> >> On 14-10-15 16:30, Björn Lässig wrote: >>> On 10/13/2015 11:01 PM, Sage Weil wrote: http://download.ceph.com/debian-testing >>> >>> unfortunately this site is not reachable at the moment.

Re: [ceph-users] download.ceph.com unreachable IPv6 [was: v9.1.0 Infernalis release candidate released]

On 10/14/2015 06:50 PM, Björn Lässig wrote: > On 10/14/2015 05:11 PM, Wido den Hollander wrote: >> >> >> On 14-10-15 16:30, Björn Lässig wrote: >>> On 10/13/2015 11:01 PM, Sage Weil wrote: http://download.ceph.com/debian-testing >>> >>> unfortunately this site is not reachable at the moment.

[ceph-users] Can we place the release key on download.ceph.com?

Hi, Currently the public keys for signing the packages can be found on git.ceph.com: https://git.ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc git.ceph.com doesn't have IPv6, but it also isn't mirrored to any system. It would be handy if http://download.ceph.com/release.asc would exis

[ceph-users] mds issue

Hi, After I set up more than 1 mds servers, it sometimes gets stuck or slow from client end. I tried to stop one mds and then the client end will hang there. I accidentally set up bal frag=true. Not sure if it matters. Later I disabled this feature. Is there any reason for the abo

[ceph-users] Hitsets and Cache Tiering

Hi All, I'm just trying to get my head around the hitset behaviour in cache tiering. I'm running latest master. If I have only a single hitset with say a hitset_period of 60s, when does a new one get created? Doing a simple PUT test, I get the feeling that the current hitset stays for eve

Re: [ceph-users] mds issue

On Wed, Oct 14, 2015 at 10:30 PM, Erming Pei wrote: > Hi, > >After I set up more than 1 mds servers, it sometimes gets stuck or slow > from client end. I tried to stop one mds and then the client end will hang > there. > >I accidentally set up bal frag=true. Not sure if it matters. Later I

Re: [ceph-users] v9.1.0 Infernalis release candidate released

Hi Sage, Dan... In our case, we have strongly invested in the testing of CephFS. It seems as a good solution to some of the issues we currently experience regarding the use cases from our researchers. While I do not see a problem in deploying Ceph cluster in SL7, I suspect that we will need

Re: [ceph-users] v9.1.0 Infernalis release candidate released

Hi and thanks at all for this good news, ;) On 13/10/2015 23:01, Sage Weil wrote: >#. Fix the data ownership during the upgrade. This is the preferred > option, > but is more work. The process for each host would be to: > > #. Upgrade the ceph package. This creates the ceph u

Re: [ceph-users] v9.1.0 Infernalis release candidate released

Sorry, another remark. On 13/10/2015 23:01, Sage Weil wrote: > The v9.1.0 packages are pushed to the development release repositories:: > > http://download.ceph.com/rpm-testing > http://download.ceph.com/debian-testing I don't see the 9.1.0 available for Ubuntu Trusty : http://downloa

Re: [ceph-users] v9.1.0 Infernalis release candidate released

On Thu, 15 Oct 2015, Goncalo Borges wrote: > Hi Sage, Dan... > > In our case, we have strongly invested in the testing of CephFS. It seems as a > good solution to some of the issues we currently experience regarding the use > cases from our researchers. > > While I do not see a problem in deployi

Re: [ceph-users] v9.1.0 Infernalis release candidate released

On Thu, 15 Oct 2015, Francois Lafont wrote: > Hi and thanks at all for this good news, ;) > > On 13/10/2015 23:01, Sage Weil wrote: > > >#. Fix the data ownership during the upgrade. This is the preferred > > option, > > but is more work. The process for each host would be to: > > >

Re: [ceph-users] v9.1.0 Infernalis release candidate released

On Thu, 15 Oct 2015, Francois Lafont wrote: > Sorry, another remark. > > On 13/10/2015 23:01, Sage Weil wrote: > > > The v9.1.0 packages are pushed to the development release repositories:: > > > > http://download.ceph.com/rpm-testing > > http://download.ceph.com/debian-testing > > I don't

[ceph-users] Does SSD Journal improve the performance?

Hi, It should be sure SSD Journal will improve the performance of IOPS. But unfortunately it's not in my test. I have two pools with the same number of osds: pool1, ssdj_sas: 9 osd servers, 8 OSDs(SAS) on every server Journal on SSD, one SSD disk for 4 SAS disks. pool 2, sas: 9 osd servers, 8

Re: [ceph-users] Does SSD Journal improve the performance?

Hello, Firstly, this is clearly a ceph-users question, don't cross post to ceph-devel. On Thu, 15 Oct 2015 09:29:03 +0800 hzwuli...@gmail.com wrote: > Hi, > > It should be sure SSD Journal will improve the performance of IOPS. But > unfortunately it's not in my test. > > I have two pools wit

Re: [ceph-users] CephFS file to rados object mapping

Hi, On 14/10/2015 06:45, Gregory Farnum wrote: >> Ok, however during my tests I had been careful to replace the correct >> file by a bad file with *exactly* the same size (the content of the >> file was just a little string and I have changed it by a string with >> exactly the same size). I had b

Re: [ceph-users] ceph same rbd on multiple client

Hi Tyler, Can please send me the next setup action to be taken on this issue. Regards Prabu On Wed, 14 Oct 2015 13:43:29 +0530 gjprabu wrote Hi Tyler, Thanks for your reply. We have disabled rbd_cache but still issue is persist. Ple

Re: [ceph-users] download.ceph.com unreachable IPv6 [was: v9.1.0 Infernalis release candidate released]

On 10/14/2015 09:19 PM, Wido den Hollander wrote: unfortunately this site is not reachable at the moment. http://eu.ceph.com/debian-testing/dists/wheezy/InRelease this one works fine for all sites. I'll tell my puppetmaster to deploy the source and we will debug this issue later in detail.