Hello,

as I've read Martin Kleppmans book "Designing Data Intensive Applications"
I'm thinking about using Kafka to distribute a temporal document store I'm
engineering called SirixDB[1].

SirixDB allows the creation of databases and therein resources, which are
currently binary tree-representations of XML or JSON documents. SirixDB
uses a huge durable and persistent (in the functional sense) tree, which
also indexes revisions and shares unchanged structures among revisions. The
main entry point is a UberPage, which is swapped atomically. Thus, a
transaction log is only needed as an in-memory buffer (which only needs to
be persisted on memory pressure if a transaction adds data without
committing, which should be an anti-pattern ;-)).

As I read that Kafka is now also able to read a partition of a topic from
followers I'm thinking about if it would be possible to exchange Kafkas
persistent layer as I think we don't need an additional persisted
"transaction log" for consistency.

Basically I thought that database-names might be topics and resources
partitions, whereas a message would consist of a transaction log entry.

Is it possible to replace the log-segments with another "storage system" as
for instance SirixDB? What's not possible is to truncate old stuff as of
now, because SirixDB also versions database pages (thus they also share
records between revisions) and it would be too tricky I guess. But I'd have
some ideas...

I know that some data stores use Kafka as a distributed transaction log,
but in my case I would like to use it without having to first store data in
Kafka and then basically in my indexed log a second time. I'd rather
replace the Kafka internal log segment storage if that's possible. For the
other use case Apache BookKeeper might be even better.

Goetz Graefe, Caetano Sauer and Theo Haedter describe a similar approach,
which doesn't have to use a persisted transaction log for consistency[2].


What do you think?

Kind regards and a happy holiday season
Johannes

[1] https://sirix.io
[2] http://www.vldb.org/pvldb/vol11/p2249-sauer.pdf

Reply via email to