Pavel, it definitely makes sense. Do you have a design in mind? D.
On Sat, Jun 30, 2018, 07:24 Pavel Kovalenko <jokse...@gmail.com> wrote: > Igniters, > > I would like to start a discussion about designing a new feature because I > think it's time to start making steps towards it. > I noticed, that some of our users have tried to store large homogenous > entries (> 1, 10, 100 Mb/Gb/Tb) to our caches, but without big success. > > IGFS project has the possibility to do it, but as for me it has one big > disadvantage - it's in-memory only, so users have a strict size limit of > their data and have data loss problem. > > Our durable memory has a possibility to persist a data that doesn't fit to > RAM to disk, but page structure of it is not supposed to store large pieces > of data. > > There are a lot of projects of distributed file systems like HDFS, > GlusterFS, etc. But all of them concentrate to implement high-grade file > protocol, rather than user-friendly API which leads to high entry threshold > to start implementing something over it. > We shouldn't go in this way. Our main goal should be providing to user easy > and fast way to use file storage and processing here and now. > > If take HDFS as closest possible by functionality project, we have one big > advantage against it. We can use our caches as files metadata storage and > have the infinite possibility to scale it, while HDFS is bounded by > Namenode capacity and has big problems with keeping a large number of files > in the system. > > We achieved very good experience with persistence when we developed our > durable memory, and we can couple together it and experience with services, > binary protocol, I/O and start to design a new IEP. > > Use cases and features of the project: > 1) Storing XML, JSON, BLOB, CLOB, images, videos, text, etc without > overhead and data loss possibility. > 2) Easy, pluggable, fast and distributed file processing, transformation > and analysis. (E.g. ImageMagick processor for images transformation, > LuceneIndex for texts, whatever, it's bounded only by your imagination). > 3) Scalability out of the box. > 4) User-friendly API and minimal steps to start using this storage in > production. > > I repeated again, this project is not supposed to be a high-grade > distributed file system with full file protocol support. > This project should primarily focus on target users, which would like to > use it without complex preparation. > > As for example, a user can deploy Ignite with such storage and web-server > with REST API as Ignite service and get scalable, performant image server > out of the box which can be accessed using any programming language. > > As a far target goal, we should focus on storing and processing a very > large amount of the data like movies, streaming, which is the big trend > today. > > I would like to say special thanks to our community members Alexey Stelmak > and Dmitriy Govorukhin which significantly helped me to put together all > pieces of that puzzle. > > So, I want to hear your opinions about this proposal. >
