Out of curiosity, are you guys re-exporting the fs to clients over something like nfs or running applications directly on the OSD nodes?
On Tue, 12 Mar 2019, 18:28 Paul Emmerich, <[email protected]> wrote: > Mounting kernel CephFS on an OSD node works fine with recent kernels > (4.14+) and enough RAM in the servers. > > We did encounter problems with older kernels though > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Tue, Mar 12, 2019 at 10:07 AM Hector Martin <[email protected]> > wrote: > > > > It's worth noting that most containerized deployments can effectively > > limit RAM for containers (cgroups), and the kernel has limits on how > > many dirty pages it can keep around. > > > > In particular, /proc/sys/vm/dirty_ratio (default: 20) means at most 20% > > of your total RAM can be dirty FS pages. If you set up your containers > > such that the cumulative memory usage is capped below, say, 70% of RAM, > > then this might effectively guarantee that you will never hit this issue. > > > > On 08/03/2019 02:17, Tony Lill wrote: > > > AFAIR the issue is that under memory pressure, the kernel will ask > > > cephfs to flush pages, but that this in turn causes the osd (mds?) to > > > require more memory to complete the flush (for network buffers, etc). > As > > > long as cephfs and the OSDs are feeding from the same kernel mempool, > > > you are susceptible. Containers don't protect you, but a full VM, like > > > xen or kvm? would. > > > > > > So if you don't hit the low memory situation, you will not see the > > > deadlock, and you can run like this for years without a problem. I > have. > > > But you are most likely to run out of memory during recovery, so this > > > could compound your problems. > > > > > > On 3/7/19 3:56 AM, Marc Roos wrote: > > >> > > >> > > >> Container = same kernel, problem is with processes using the same > > >> kernel. > > >> > > >> > > >> > > >> > > >> > > >> > > >> -----Original Message----- > > >> From: Daniele Riccucci [mailto:[email protected]] > > >> Sent: 07 March 2019 00:18 > > >> To: [email protected] > > >> Subject: Re: [ceph-users] mount cephfs on ceph servers > > >> > > >> Hello, > > >> is the deadlock risk still an issue in containerized deployments? For > > >> example with OSD daemons in containers and mounting the filesystem on > > >> the host machine? > > >> Thank you. > > >> > > >> Daniele > > >> > > >> On 06/03/19 16:40, Jake Grimmett wrote: > > >>> Just to add "+1" on this datapoint, based on one month usage on Mimic > > >>> 13.2.4 essentially "it works great for us" > > >>> > > >>> Prior to this, we had issues with the kernel driver on 12.2.2. This > > >>> could have been due to limited RAM on the osd nodes (128GB / 45 OSD), > > >>> and an older kernel. > > >>> > > >>> Upgrading the RAM to 256GB and using a RHEL 7.6 derived kernel has > > >>> allowed us to reliably use the kernel driver. > > >>> > > >>> We keep 30 snapshots ( one per day), have one active metadata server, > > >>> and change several TB daily - it's much, *much* faster than with > fuse. > > >>> > > >>> Cluster has 10 OSD nodes, currently storing 2PB, using ec 8:2 coding. > > >>> > > >>> ta ta > > >>> > > >>> Jake > > >>> > > >>> > > >>> > > >>> > > >>> On 3/6/19 11:10 AM, Hector Martin wrote: > > >>>> On 06/03/2019 12:07, Zhenshi Zhou wrote: > > >>>>> Hi, > > >>>>> > > >>>>> I'm gonna mount cephfs from my ceph servers for some reason, > > >>>>> including monitors, metadata servers and osd servers. I know it's > > >>>>> not a best practice. But what is the exact potential danger if I > > >>>>> mount cephfs from its own server? > > >>>> > > >>>> As a datapoint, I have been doing this on two machines (single-host > > >>>> Ceph > > >>>> clusters) for months with no ill effects. The FUSE client performs a > > >>>> lot worse than the kernel client, so I switched to the latter, and > > >>>> it's been working well with no deadlocks. > > >>>> > > >>> _______________________________________________ > > >>> ceph-users mailing list > > >>> [email protected] > > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >>> > > >> _______________________________________________ > > >> ceph-users mailing list > > >> [email protected] > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >> > > >> > > >> _______________________________________________ > > >> ceph-users mailing list > > >> [email protected] > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >> > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > [email protected] > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > -- > > Hector Martin ([email protected]) > > Public Key: https://mrcn.st/pub > > _______________________________________________ > > ceph-users mailing list > > [email protected] > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
