Hello, Evgeni. > It seems like this solution will have an unpredictable delay for > synchronization for handling events.
It’s true that CDC solution doesn’t have strict boundaries for notification delay because of asynchronous nature. But, I assume that we will introduce a WAL rollout timeout for CDC cases Please, take a look at the ticket [1]. The same approach is used by Oracle and other databases that implement CDC. Anyway, I treat notification delay and split of the event and its consumption as an advantage of CDC, not downside :) > Why can't we just implement a Debezium connector for Ignite, for example? I think we can. But, AFAIK debezium connectors developed for other databases uses CDC implementations similar to proposed. [1] https://issues.apache.org/jira/browse/IGNITE-13582?src=confmacro > 14 окт. 2020 г., в 15:36, Evgenii Zhuravlev <e.zhuravlev...@gmail.com> > написал(а): > > Hi, > >> On the segment archiving, utility iterates it using existing WALIterator >> Wait and respond to some specific events or data changes. > It seems like this solution will have an unpredictable delay for > synchronization for handling events. > > Why can't we just implement a Debezium connector for Ignite, for example? > https://debezium.io/documentation/reference/1.3/index.html. It is a pretty > popular product that uses Kafka underneath. > > Evgenii > > > ср, 14 окт. 2020 г. в 05:00, Pavel Kovalenko <jokse...@gmail.com>: > >> This tool is also can be used to store snapshots in an external warehouse. >> >> >> ср, 14 окт. 2020 г. в 14:57, Pavel Kovalenko <jokse...@gmail.com>: >> >>> Hi Nikolay, >>> >>> The idea is good. But what do you think to integrate these ideas into >>> WAL-G project? >>> https://github.com/wal-g/wal-g >>> It's a well-known tool that is already used to stream WAL for PostgreSQL, >>> MySQL, and MongoDB. >>> The advantages are integration with S3, GCP, Azure out of the box, >>> encryption, and compression. >>> >>> >>> ср, 14 окт. 2020 г. в 14:21, Nikolay Izhikov <nizhi...@apache.org>: >>> >>>> Hello, Igniters. >>>> >>>> I want to start a discussion of the new feature [1] >>>> >>>> CDC - capture data change. The feature allows the consumer to receive >>>> online notifications about data record changes. >>>> >>>> It can be used in the following scenarios: >>>> * Export data into some warehouse, full-text search, or >>>> distributed log system. >>>> * Online statistics and analytics. >>>> * Wait and respond to some specific events or data changes. >>>> >>>> Propose to implement new IgniteCDC application as follows: >>>> * Run on the server node host. >>>> * Watches for the appearance of the WAL archive segments. >>>> * Iterates it using existing WALIterator and notifies consumer >> of >>>> each record from the segment. >>>> >>>> IgniteCDC features: >>>> * Independence from the server node process (JVM) - issues and >>>> failures of the consumer will not lead to server node instability. >>>> * Notification guarantees and failover - i.e. CDC track and save >>>> the pointer to the last consumed record. Continue notification from this >>>> pointer in case of restart. >>>> * Resilience for the consumer - it's not an issue when a >> consumer >>>> temporarily consumes slower than data appear. >>>> >>>> WDYT? >>>> >>>> [1] >>>> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+CDC+-+Capture+Data+Change >>> >>> >>
