Hello, as I've read Martin Kleppmans book "Designing Data Intensive Applications" I'm thinking about using Kafka to distribute a temporal document store I'm engineering called SirixDB[1].
SirixDB allows the creation of databases and therein resources, which are currently binary tree-representations of XML or JSON documents. SirixDB uses a huge durable and persistent (in the functional sense) tree, which also indexes revisions and shares unchanged structures among revisions. The main entry point is a UberPage, which is swapped atomically. Thus, a transaction log is only needed as an in-memory buffer (which only needs to be persisted on memory pressure if a transaction adds data without committing, which should be an anti-pattern ;-)). As I read that Kafka is now also able to read a partition of a topic from followers I'm thinking about if it would be possible to exchange Kafkas persistent layer as I think we don't need an additional persisted "transaction log" for consistency. Basically I thought that database-names might be topics and resources partitions, whereas a message would consist of a transaction log entry. Is it possible to replace the log-segments with another "storage system" as for instance SirixDB? What's not possible is to truncate old stuff as of now, because SirixDB also versions database pages (thus they also share records between revisions) and it would be too tricky I guess. But I'd have some ideas... I know that some data stores use Kafka as a distributed transaction log, but in my case I would like to use it without having to first store data in Kafka and then basically in my indexed log a second time. I'd rather replace the Kafka internal log segment storage if that's possible. For the other use case Apache BookKeeper might be even better. Goetz Graefe, Caetano Sauer and Theo Haedter describe a similar approach, which doesn't have to use a persisted transaction log for consistency[2]. What do you think? Kind regards and a happy holiday season Johannes [1] https://sirix.io [2] http://www.vldb.org/pvldb/vol11/p2249-sauer.pdf
