Hi, Konstantin, currently we work on IGFS failovers and reliability, so IGFS should behave correctly in any expected use case. It's not quite clear to me what does "when an _optional_ secondary file system is plugged in" mean. Can you please explain this use case in more detail?
On Thu, Nov 26, 2015 at 5:36 AM, Konstantin Boudnik <c...@apache.org> wrote: > On Wed, Nov 25, 2015 at 04:54PM, Ivan V. wrote: > > Hi, Konstantin, > > "TRASH" (the name comes from > > org.apache.ignite.internal.processors.igfs.IgfsFileInfo#TRASH_ID Java > > constant) notion is only applicable to primary (IGFS) file system. This > is > > a special "folder" that does not have file system path. When IGFS is > > running over a secondary Fs, TRASH also exists in the primary IGFS, but > > does not exist in the secondary Fs. > > In secondary Fs deletion is performed just through the ordinary Fs API. > So, > > we *do not* employ any assumption regarding the TRASH existence and > > behavior in the secondary Fs. > > > > As Dmitriy mentioned above, TRASH in primary Fs is needed for performance > > reasons: with it we delete file with only 1 transaction in Meta cache: we > > do not do any transactions in Data cache. > > (Similar technique is frequently applied frequently in real Fs deletion, > > like mv foo /tmp/ && rm -r /tmp/foo/ .) > > I understand. However, I am wary about the potentially funny and > inconsistent > cases when an _optional_ secondary file system is plugged in. > > Cos > > > Currently we have fix of > https://issues.apache.org/jira/browse/IGNITE-1679 > > that makes PURGE events enabled for all files. > > I still not quite realize how this functionality will be used by > customers, > > but now it is repaired: once merged, you will be able to use it. > > > > On Tue, Nov 24, 2015 at 2:52 AM, Dmitriy Setrakyan < > dsetrak...@apache.org> > > wrote: > > > > > Cos, > > > > > > The main reason soft delete was added is performance. Without > soft-delete, > > > the delete operation would have to wait until a file is fully deleted > from > > > a folder, which may take time. > > > > > > As far as secondary FS handling it, IGFS does not require a secondary > FS, > > > so we should account for cases when IGFS is running stand-alone. > > > > > > Thoughts? > > > > > > D. > > > > > > On Mon, Nov 23, 2015 at 11:00 PM, Konstantin Boudnik <c...@apache.org> > > > wrote: > > > > > > > Let me ask a different question: what's the point of having the > concept > > > of > > > > TRASH? > > > > > > > > Here's an example why I think the 'soft' delete would only complicate > > > > thing. > > > > Suppose IGFS is sitting on top of HDFS and both have 'Trash' enabled. > > > Now, > > > > the file is getting soft-deleted from IGFS and is moved to TRASH > folder. > > > > But > > > > in HDFS it is also a move to a place which doesn't have any special > > > meaning > > > > for HDFS. > > > > > > > > Even worst, if IFGS TRASH is linked to HDFS .Trash. HDFS has it's own > > > > policy > > > > on how to clean that up, which is likely to be different from that on > > > IGFS. > > > > Often enough, HDFS .Trash is simply disabled. This discrepancy is > going > > > to > > > > create a situation when a file should still be in TRASH, but the > > > secondary > > > > FS > > > > has already purged it. > > > > > > > > And what if yet another secondary file system like S3 has yet another > > > > policy > > > > around their own trash, which they don't even have, I believe? > > > > > > > > Where I am going with this is pretty straight forward: let's drop the > > > > soft-delete support and let the secondary FS to deal with it. If > there's > > > no > > > > secondary FS configured - the content of deleted file will have to > > > > retrieved > > > > by other means. > > > > > > > > Thoughts? > > > > Cos > > > > > > > > On Fri, Nov 20, 2015 at 07:15PM, Ivan V. wrote: > > > > > Hi, dev, > > > > > need opinions on the question discussed in > > > > > https://issues.apache.org/jira/browse/IGNITE-1679 (IGFS: Purge > event > > > is > > > > > inconsistent). > > > > > In short: in Igfs we have "soft" delete that moves the deleted > file or > > > > > folder to special "TRASH" folder. > > > > > Special async worker walks inside TRASH and removes the items > > > > permanently. > > > > > When an item is completely removed, an event of type > > > > > org.apache.ignite.events.EventType#EVT_IGFS_FILE_PURGED is fired. > > > > > But such events are now fired only for files, and only in case if > such > > > > file > > > > > was deleted itself, but not a part of a folder sub-tree. It's quite > > > > obvious > > > > > that such behavior is not quite consistent, so we should either > get rid > > > > of > > > > > PURGE events at all, or make them consistent. > > > > > In the latter case it would be good to have answer to the question: > > > what > > > > > are real use cases when we may need the purge events ? (Now they > seem > > > to > > > > > be used in tests only). > > > > > If we don't have such real use cases, are there any objections to > get > > > rid > > > > > of the purge events at all? > > > > > Thanks in advance. > > > > > > > >
