[ceph-users] Cannot mount CephFS after irreversible OSD lost

Dear ceph experts, I've built and administrating 12 OSD ceph cluster (spanning over 3 nodes) with replication count of 2. The ceph version is ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) The cluster hosts two pools (data and metadata) that are exported over CephFS. At some

Re: [ceph-users] Cannot mount CephFS after irreversible OSD lost

Dear John, Thanks for such a prompt reply! Seems like something happens on the mon side, since there are no mount-specific requests logged on the mds side (see below). FYI, some hours ago I've disabled auth completely, but it didn't help. The serialized metadata pool is 9.7G. I can try to compre

Re: [ceph-users] Cannot mount CephFS after irreversible OSD lost

to flush/reset the MDS cache? On 17 November 2015 at 13:26, John Spray wrote: > On Tue, Nov 17, 2015 at 12:17 PM, Mykola Dvornik > wrote: > > Dear John, > > > > Thanks for such a prompt reply! > > > > Seems like something happens on the mon side, since t

Re: [ceph-users] Cannot mount CephFS after irreversible OSD lost

t if it is, the damage is marginal. So the question is cephfs-data-scan designed to resolve problems with duplicated inodes? On 19 November 2015 at 04:17, Yan, Zheng wrote: > On Wed, Nov 18, 2015 at 5:21 PM, Mykola Dvornik > wrote: > >> Hi John, >> >> It turned out

Re: [ceph-users] Cannot mount CephFS after irreversible OSD lost

d you do at this stage is mount your filesystem read-only, back it up, and then create a new filesystem and restore from backup. Ok. Is it somehow possible to have multiple FSs on the same ceph cluster? On 19 November 2015 at 10:43, John Spray wrote: > On Wed, Nov 18, 2015 at 9:21 AM, Myk

Re: [ceph-users] Cannot mount CephFS after irreversible OSD lost

Thanks for the tip. I will stay of the safe side and wait until it will be merged into master) Many thanks for all your help. -Mykola On 19 November 2015 at 11:10, John Spray wrote: > On Thu, Nov 19, 2015 at 10:07 AM, Mykola Dvornik > wrote: > > I'm guessing in this context

Re: [ceph-users] Can't activate osd in infernalis

cat /etc/udev/rules.d/89-ceph-journal.rules KERNEL=="sdd?" SUBSYSTEM=="block" OWNER="ceph" GROUP="disk" MODE="0660" On 19 November 2015 at 13:54, Mykola wrote: > I am afraid one would need an udev rule to make it persistent. > > > > Sent from Outlook Mail

Re: [ceph-users] Can't activate osd in infernalis

he folder > exists, but all the other udev rules are in /usr/lib/udev/rules.d/. > Can I just create a new file named "89-ceph-journal.rules" in the > /usr/lib/udev/rules.d/ folder? > > > Regards > > David > > > On 19.11.2015 14:02, Mykola Dvornik wrote: > &

Re: [ceph-users] ceph osd prepare cmd on infernalis 9.2.0

*'Could not create partition 2 from 10485761 to 10485760'.* Perhaps try to zap the disks first? On 19 November 2015 at 16:22, German Anders wrote: > Hi cephers, > > I had some issues while running the prepare osd command: > > ceph version: infernalis 9.2.0 > > disk: /dev/sdf (745.2G) >

Re: [ceph-users] osd become unusable, blocked by xfsaild (?) and load > 5000

The same thing happens to my setup with CentOS7.x + non-stock kernel (kernel-ml from elrepo). I was not happy with IOPS I got out of the stock CentOS7.x so I did the kernel upgrade and crashes started to happen until some of the OSDs become non-bootable at all. The funny thing is that I was no

[ceph-users] CephFS: number of PGs for metadata pool

Hi guys, I am creating a 4-node/16OSD/32TB CephFS from scratch. According to the ceph documentation the metadata pool should have small amount of PGs since it contains some negligible amount of data compared to data pool. This makes me feel it might not be safe. So I was wondering how to cho

Re: [ceph-users] CephFS: number of PGs for metadata pool

ebody did some research in this direction? On Wed, Dec 9, 2015 at 1:13 PM, Jan Schermer wrote: Number of PGs doesn't affect the number of replicas, so don't worry about it. Jan On 09 Dec 2015, at 13:03, Mykola Dvornik wrote: Hi guys, I am creating a 4-node/16OSD/3

Re: [ceph-users] CephFS: number of PGs for metadata pool

25 PM, Mykola Dvornik wrote: Hi Jan, Thanks for the reply. I see your point about replicas. However my motivation was a bit different. Consider some given amount of objects that are stored in the metadata pool. If I understood correctly ceph data placement approach, the number of object

[ceph-users] CephFS quotas in kernel client

Any plans to support quotas in CephFS kernel client? -Mykola___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS quotas in kernel client

Thanks for a quick reply. On Mon, 2016-05-23 at 20:08 +0800, Yan, Zheng wrote: > No plan so far.  Current quota design requires client to do > bottom-to-top path walk, which is unfriendly for kernel client (due > to > lock design of kernel). > > On Mon, May 23, 2016 at 4:55

[ceph-users] rados complexity

Are there any ceph users with pools containing >2 kobjects? If so, have you noticed any instabilities of the clusters once this threshold is reached? -Mykola___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/cep

Re: [ceph-users] rados complexity

00 schrieb Mykola Dvornik: > Are there any ceph users with pools containing >2 kobjects? > > If so, have you noticed any instabilities of the clusters once this > threshold > is reached? > > -Mykola > ___ > ceph-users mailing

Re: [ceph-users] CephFS mds cache pressure

I have the same issues with the variety of kernel clients running 4.6.3 and 4.4.12 and fuse clients from 10.2.2. -Mykola -Original Message- From: xiaoxi chen To: João Castro , ceph-users@lists.ceph.com Subject: Re: [ceph-users] CephFS mds cache pressure Date: Wed, 29 Jun 2016 01:00:40 +

[ceph-users] Maximum possible IOPS for the given configuration

Dear ceph-users, Are there any expressions / calculators available to calculate the maximum expected random write IOPS of the ceph cluster? To my understanding of the ceph IO, this should be something like MAXIOPS = (1-OVERHEAD) * OSD_BACKENDSTORAGE_IOPS * NUM_OSD / REPLICA_COUNT So the questio

Re: [ceph-users] Lessons learned upgrading Hammer -> Jewel

I would also advice people to mind the SELinux if it is enabled on the OSD's nodes. The re-labeling should be done as the part of the upgrade and this is rather time consuming process. -Original Message- From: Mart van Santen To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Lessons

[ceph-users] Recovering full OSD

Dear ceph community, One of the OSDs in my cluster cannot start due to the *ERROR: osd init failed: (28) No space left on device* A while ago it was recommended to manually delete PGs on the OSD to let it start. So I am wondering was is the recommended way to fix this issue for the cluster runn

Re: [ceph-users] Recovering full OSD

> On Mon, Aug 8, 2016 at 8:01 PM, Mykola Dvornik > wrote: > > Dear ceph community, > > > > One of the OSDs in my cluster cannot start due to the > > > > ERROR: osd init failed: (28) No space left on device > > > > A while ago it was recommended to man

[ceph-users] CephFS: No space left on device

After upgrading to 10.2.3 we frequently see messages like 'rm: cannot remove '...': No space left on device The folders we are trying to delete contain approx. 50K files 193 KB each. The cluster state and storage available are both OK: cluster 98d72518-6619-4b5c-b148-9a781ef13bcb healt

Re: [ceph-users] CephFS: No space left on device

to time. I am aware of quite discussion about this issue on the ML, but cannot really follow how to debug it. Regards, -Mykola On 2 October 2016 at 22:27, John Spray wrote: > On Sun, Oct 2, 2016 at 11:09 AM, Mykola Dvornik > wrote: > > After upgrading to 10.2.3 we frequently

Re: [ceph-users] CephFS: No space left on device

to roughly correspond > to mds_max_purge_files, so I'd definitely try changing that to get > things purging faster. > > John > > On Mon, Oct 3, 2016 at 3:21 PM, Mykola Dvornik > wrote: > > Hi John, > > > > This is how the daemonperf looks like : > &g

Re: [ceph-users] CephFS: No space left on device

up to 23K. No inconsistent PGs or any other problems happened to the cluster within this time scale. -Mykola On 5 October 2016 at 05:49, Yan, Zheng wrote: > On Mon, Oct 3, 2016 at 5:48 AM, Mykola Dvornik > wrote: > > Hi Johan, > > > > Many thanks for your reply. I will

Re: [ceph-users] CephFS: No space left on device

you use before upgrading to 10.2.3 ? > > Regards > Yan, Zheng > > > > > > > -Mykola > > > > > > > > From: Yan, Zheng > > Sent: Thursday, 6 October 2016 04:48 > > To: Mykola Dvornik > > Cc: John Spray; ceph-users > >

Re: [ceph-users] CentOS 7.2, Infernalis, preparing osd's and partprobe issues.

I had more or less the same problem. This most likely synchronization issue. I have been deploying 16 OSD each running exactly the same hardware/software. The issue appeared randomly with no obvious correlations with other stuff. The dirty workaround was to put time.sleep(5) before invoking partpro

[ceph-users] ceph-fuse inconsistent filesystem view from different clients

Hi guys, I have 16 OSD/1MON/1MDS ceph cluster serving CephFS. The FS is mounted on 11 clients using ceph-fuse. In some cases there are multiple ceph-fuse processes per client, each with its own '-r' option. The problem is that some of the clients get significantly out-of-sync. Flushing caches, '

Re: [ceph-users] ceph-fuse inconsistent filesystem view from different clients

regards, Mykola On Wed, Dec 30, 2015 at 7:49 AM, Yan, Zheng wrote: On Wed, Dec 30, 2015 at 5:59 AM, Mykola Dvornik wrote: Hi guys, I have 16 OSD/1MON/1MDS ceph cluster serving CephFS. The FS is mounted on 11 clients using ceph-fuse. In some cases there are multiple ceph-fuse processes

[ceph-users] CephFS is not maintianing conistency

Hi guys, This is sort of rebuttal. I have a CephFS deployed and mounted on a couple of clients via ceph-fuse (due to quota support and possibility to kill the ceph-fuse process to avoid stale mounts). So the problems is that some times the changes made on one client are not visible on the o

Re: [ceph-users] CephFS is not maintianing conistency

nt io 4381 B/s wr, 2 op In addition on the clients' side I have cat /etc/fuse.conf user_allow_other auto_cache large_read max_write = 16777216 max_read = 16777216 -Mykola On Mon, Feb 1, 2016 at 5:06 PM, Gregory Farnum wrote: On Monday, February 1, 2016, Mykola Dvornik wrote: Hi guys,

Re: [ceph-users] CephFS is not maintianing conistency

:27 AM, Mykola Dvornik wrote: What version are you running on your servers and clients? Are you using 4.1 or 4.2 kernel? https://bugzilla.kernel.org/show_bug.cgi?id=104911. Upgrade to 4.3+ kernel or 4.1.17 kernel or 4.2.8 kernel can resolve this issue. On the clients: ceph-fuse --version

Re: [ceph-users] CephFS is not maintianing conistency

16 at 5:32 PM, Mykola Dvornik wrote: One of my clients is using 4.3.5-300.fc23.x86_64 (Fedora release 23) did you encounter this problem on client using 4.3.5 kernel? If you did, this issue should be ceph-fuse bug. while all the other clients reply on 3.10.0-327.4.4.el7.x86_64 (CentOS

Re: [ceph-users] CephFS is not maintianing conistency

No, I have not had any issues with 4.3.x. On Tue, Feb 2, 2016 at 3:28 PM, Yan, Zheng wrote: On Tue, Feb 2, 2016 at 8:28 PM, Mykola Dvornik wrote: No, I've never seen this issue on the Fedora stock kernels. So either my workflow is not triggering it on the Fedora software stack or

Re: [ceph-users] Urgent help needed for ceph storage "mount error 5 = Input/output error"

I would strongly(!) suggest you to add few more OSDs to cluster before things get worse / corrupted. -Mykola On Tue, Feb 2, 2016 at 6:45 PM, Zhao Xu wrote: Hi All, Recently our ceph storage is running at low performance. Today, we can not write to the folder. We tried to unmount the ceph

Re: [ceph-users] Urgent help needed for ceph storage "mount error 5 = Input/output error"

maker. For now, I just want to mount the ceph drive again, even in read only mode, so that they can read the data. Any idea on how to achieve this? Thanks, X On Tue, Feb 2, 2016 at 9:57 AM, Mykola Dvornik wrote: I would strongly(!) suggest you to add few more OSDs to cluster before things get