Igniters, We revealed concurrency problem in IGFS and I would like to discuss possible solutions to it.
Consider the following file system structure: root |-- A | |-- B | | |-- C | |-- D ... two concurrent operations in different threads: T1: move(/A/B, /A/D); T2: move(/A/D, /A/B/C); ... and how IGFS handles it now: T1: verify that "/A/B" and "/A/D" exist, they are not child-parent to each other, etc. -> OK. T2: do the same for "A/D" and "A/B/C" -> OK. T1: get IDs of "/A", "/A/B" and "/A/D" to lock them later inside tx. T2: get IDs of "/A", "/A/D", "/A/B" and "/A/B/C" to lock them later inside tx. T1: Start pessimistic tx, lock IDs of "/A", "/A/B", "/A/D", perform move -> OK. root |-- A | |-- D | | |-- B | | | |-- C T2: Start pessimistic tx, lock IDs of "/A", "/A/D", "/A/B" and "/A/B/C" (*directory structure already changed at this time!*), perform move -> OK. root |-- A B |-- D | |-- C | | |-- B (loop!) File system is corrupted. Folders B, C and D are not reacheable from root. To fix this now we additionaly check if directory structure is still valid *inside transaction*. It works, no more corruptions. But it requres taking locks on the whole paths *including root*. So move, delete and mkdirs opeartions *can no longer be concurrent*. Probably there is a way to relax this while still ensuring consistency, but I do not see how. One idea is to store real path inside each entry. This way we will be able to ensure that it is still at a valid location without blocking parents, so concurrnecy will be restored. But we will have to propagate strucutral changes to children. E.g. move of a folder with 100 items will lead to update of >100 cache entries. Not so good. Any other ideas? Vladimir.
