I don't understand what Ignite does with processing filesystem directories. This sounds like a question for whoever is supporting your OS. It is generally a very bad idea to write to files which belong to Ignite. I suspect you're struggling with OS file locking semantics and concurrent file access. If this is what you are doing with your Ignite client, you likely need to operate similar to what you did for Hazelcast, which like Ignite is downstream of your concurrent file processing and locking.
On Sat, Nov 9, 2024 at 8:16 PM Alex Roytman <roytm...@gmail.com> wrote: > Hello Everyone, > > I am processing files (in a complex way) in a large number of file system > directories. For the most part, at any given time different directories > will be touched but the same directory may be written to from different > threads or nodes (and it may take some time to complete processing) so I > need to do it under a lock. > > I would like to ask if there is a significant cost in creating, using and > destroying a lock and if resources will be cleanly released. > In my Hazelcast based implementation I hash all directory paths being used > to 256 locks because if I create/use/destroy a lock for each directory, the > locks are never fully deallocated upon destruction and fairly quickly eat > up all resources. > > 256 (configurable) locks seem to provide sufficient concurrency even for > processing that may take 100 of seconds > > Would you expect ignite 3 the same behavior as ignite 2 in this respect > or it may be different > > Would you recommend I should continue using this strategy with Ignite? > > Thank you, > Alex >
