On Sat, Jun 21, 2025 at 12:23 AM Patrick Begou <patrick.be...@univ-grenoble-alpes.fr> wrote: > > Hi James, > > this is an interesting discussion for me. This raises 2 questions for my > understanding of the concept: > > 1) do you mean snapshots are not taken in account for quotas ? A user > can remove files after a snapshot to retrieve some storage ?
That's correct-ish. CephFS size accounting is all done in terms of file size, but as disk allocations are all done at the OSD level, the MDS has no idea what they look like. (This also means if you create a sparse file and write 1 byte at the 1TB offset, the MDS accounting will say you have used 1TB.) Snapshots are not accounted at all, because their space too is only known to individual OSDs on a particular object, not at any central point. > 2) in a cephfs context using subvolumes per user require a mount point > for each user on the clients ? Or many users folders with different > quota can coexist in the subvolume ? Subvolumes are intended to align to a particular application or user. Trying to use a single subvolume for multiple users would in some ways work fine, but in others be extremely weird. The real answer to why subvolumes are special (or indeed *exist*) is that they are CephFS' answer to the need for APIs that work well with projects like OpenStack Manila or Kubernetes CSI expectations. From the MDS' perspective, subvolumes are special in exactly two ways: 1) you cannot hard link across a subvolume boundary, 2) you cannot snapshot inside of a subvolume, only at the root. But the mgr/volumes interface has lots of other expectations of them, and they get other bits set up around them to support the integrations they exist for. (They're configured with separate rados namespaces; a cephx key is generated for each subvolume specific to it and that does not have snapshot permissions; probably other stuff.) And the ways in which they are special on the MDS are entirely because it let us resolve other problems. -Greg > > Patrick (Ceph beginner not using subvolumes yet) > > Le 20/06/2025 à 10:24, Alexander Patrakov a écrit : > > Hello James, > > > > The special thing about subvolumes is that you can only take snapshots at > > their root but not inside. This is a security feature: give a subdirectory > > of a subvolume to a user, set the quota on the subvolume, and you can be > > sure that the user will not be able to circumvent the quota by creating new > > snapshots in the area he has access to. > > > > On Fri, Jun 20, 2025 at 3:54 PM James Tocknell <james.tockn...@mq.edu.au> > > wrote: > > > >> Hi All > >> > >> Possibly a dumb question, but given with cephfs you can apply quotas to > >> arbitrary directories, snapshot arbitrary directories, assign arbitrary > >> directories to pools etc. what is special/different about subvolumes, and > >> why would I want to use them (and more importantly, not want to use them)? > >> I guess the only thing I can see could be around where the extended > >> attributes are assigned (with the two-level directory structure enabling > >> setting a quota while not allowing read access to the directory above, > >> based on looking at the cephfs CSI?), but I feel I'm missing something > >> obvious. > >> > >> The specific usecase I'm looking at is evaluating using just quotas vs > >> subvolumes for team shares (to ensure that each team only uses the amount > >> of storage they require), but in general I'm trying to understand > >> subvolumes. > >> > >> Regards > >> James > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@ceph.io > >> To unsubscribe send an email to ceph-users-le...@ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
