timoninmaxim commented on code in PR #11497: URL: https://github.com/apache/ignite/pull/11497#discussion_r1981675471
########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -60,6 +74,32 @@ image:../../assets/images/integrations/CDC-ignite2igniteClient.svg[] | `MappingsCount` | Count of mappings events applied to destination cluster |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/thin/IgniteToIgniteClientCdcStreamer.java[Ignite2IgniteClientCdcStreamer] bean configuration to the source ignite configuration, and use the resulting configuration with `ignite-cdc.sh` to start CDC client on the source Ignite instance. + +NOTE: Make sure both source and destination clusters are up. Review Comment: For what? ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -30,7 +30,21 @@ NOTE: All implementations of the cross-cluster replication support replication o NOTE: To use SQL queries on the destination cluster over CDC-replicated data, set the same `VALUE_TYPE` in link:sql-reference/ddl#create-table[CREATE TABLE] on both source and destination clusters for each table. -== Ignite to Java Thin Client CDC streamer +==== Installation + +. Build `cdc-ext` module with maven: ++ +```console + $~/src/ignite-extensions/> mvn clean package -DskipTests + $~/src/ignite-extensions/> ls modules/cdc-ext/target | grep zip +ignite-cdc-ext.zip Review Comment: no such file. ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -90,19 +130,76 @@ image:../../assets/images/integrations/CDC-ignite2ignite.svg[] | `MappingsCount` | Count of mappings events applied to destination cluster |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/IgniteToIgniteCdcStreamer.java[Ignite2IgniteCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<!--IgniteToIgniteCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.IgniteToIgniteCdcStreamer"> + <property name="destinationIgniteConfiguration"> + <bean class="org.apache.ignite.configuration.IgniteConfiguration"> + <property name="igniteInstanceName" value="cluster-cdc-client" /> + <property name="clientMode" value="true" /> + <property name="localHost" value="127.0.0.1" /> + <property name="discoverySpi" ref="destination.TcpDiscoverySpi"/> + </bean> + </property> + + <property name="caches"> + <list> + <value>cache1</value> + <value>cache2</value> + </list> + </property> + + <property name="onlyPrimary" value="false"/> + <property name="maxBatchSize" value="1024"/> +</bean> + +<!--Destination TcpDiscoverySpi for CDC streamer--> +<bean id="destination.TcpDiscoverySpi" class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> + <property name="ipFinder"> + <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> + <property name="addresses" value="127.0.0.1:47600..47610" /> + </bean> + </property> + + <property name="localPort" value="47601" /> + <property name="joinTimeout" value="10000" /> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. Review Comment: misprint in "configuation" ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -165,33 +262,69 @@ section of the official Kafka documentation. | `MarkersCount` | Count of metadata markers sent to Kafka. |=== -=== `kafka-to-ignite.sh` application +=== Configuration example -This application should be started near the destination cluster. -`kafka-to-ignite.sh` will read CDC events from Kafka topic and then apply them to destination cluster. +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/IgniteToKafkaCdcStreamer.java[Ignite2KafkaCdcStreamer] bean configuration to the source ignite configuration, and use the resulting configuration with `ignite-cdc.sh` to start CDC client on the source Ignite instance. -IMPORTANT: `kafka-to-ignite.sh` implements the fail-fast approach. It just fails in case of any error. The restart procedure should be configured with the OS tools. +```xml +<!--Kafka properties for CDC streamer--> +<util:properties id="kafkaProperties" location="file:/config/path/kafka.properties"/> -Count of instances of the application does not corellate to the count of destination server nodes. -It should be just enough to process source cluster load. -Each instance of application will process configured subset of topic partitions to spread the load. -`KafkaConsumer` for each partition will be created to ensure fair reads. +<!--IgniteToKafkaCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.kafka.IgniteToKafkaCdcStreamer"> + <property name="topic" value="dc1_to_dc2"/> + <property name="metadataTopic" value="metadata_from_dc1"/> + <property name="kafkaPartitions" value="16"/> + <property name="caches"> + <list> + <value>terminator</value> + </list> + </property> + <property name="maxBatchSize" value="1024"/> + <property name="onlyPrimary" value="false"/> + <property name="kafkaProperties" ref="kafkaProperties"/> +</bean> +``` -==== Installation +Use the following steps to fully configure the replication with Apache Kafka: + +1. Pre-activate source cluster before starting the CDC client. You can use `command.sh` for that. -. Build `cdc-ext` module with maven: -+ -```console - $~/src/ignite-extensions/> mvn clean package -DskipTests - $~/src/ignite-extensions/> ls modules/cdc-ext/target | grep zip -ignite-cdc-ext.zip +``` +./control.sh --set-state ACTIVE --host localhost:server_connector_port --yes ``` -. Unpack `ignite-cdc-ext.zip` archive to `$IGNITE_HOME` folder. +2. Use the resulting configuration with `ignite-cdc.sh` to start CDC client on the source Ignite instance. Review Comment: Smth wrong with enumeration <img width="819" alt="Снимок экрана 2025-03-05 в 18 46 18" src="https://github.com/user-attachments/assets/cbc4a81f-4a7f-4519-9e15-f5c74f83b82e"; /> ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -90,19 +130,76 @@ image:../../assets/images/integrations/CDC-ignite2ignite.svg[] | `MappingsCount` | Count of mappings events applied to destination cluster |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/IgniteToIgniteCdcStreamer.java[Ignite2IgniteCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<!--IgniteToIgniteCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.IgniteToIgniteCdcStreamer"> + <property name="destinationIgniteConfiguration"> + <bean class="org.apache.ignite.configuration.IgniteConfiguration"> + <property name="igniteInstanceName" value="cluster-cdc-client" /> + <property name="clientMode" value="true" /> + <property name="localHost" value="127.0.0.1" /> + <property name="discoverySpi" ref="destination.TcpDiscoverySpi"/> + </bean> + </property> + + <property name="caches"> + <list> + <value>cache1</value> + <value>cache2</value> + </list> + </property> + + <property name="onlyPrimary" value="false"/> + <property name="maxBatchSize" value="1024"/> +</bean> + +<!--Destination TcpDiscoverySpi for CDC streamer--> +<bean id="destination.TcpDiscoverySpi" class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> + <property name="ipFinder"> + <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> + <property name="addresses" value="127.0.0.1:47600..47610" /> + </bean> + </property> + + <property name="localPort" value="47601" /> + <property name="joinTimeout" value="10000" /> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. + +NOTE: Make sure both source and destination clusters are up. + == CDC replication using Kafka This way to replicate changes between clusters requires setting up two applications: . `ignite-cdc.sh` with `org.apache.ignite.cdc.kafka.IgniteToKafkaCdcStreamer` that will capture changes from source cluster and write it to Kafka topic. . `kafka-to-ignite.sh` that will read changes from Kafka topic and then write them to destination cluster. +NOTE: Please, enable `ignite-cdc-ext` to be able to run `kafka-to-ignite.sh`. Review Comment: We already mention it in Overview? ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -90,19 +130,76 @@ image:../../assets/images/integrations/CDC-ignite2ignite.svg[] | `MappingsCount` | Count of mappings events applied to destination cluster |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/IgniteToIgniteCdcStreamer.java[Ignite2IgniteCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<!--IgniteToIgniteCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.IgniteToIgniteCdcStreamer"> + <property name="destinationIgniteConfiguration"> + <bean class="org.apache.ignite.configuration.IgniteConfiguration"> + <property name="igniteInstanceName" value="cluster-cdc-client" /> + <property name="clientMode" value="true" /> + <property name="localHost" value="127.0.0.1" /> + <property name="discoverySpi" ref="destination.TcpDiscoverySpi"/> + </bean> + </property> + + <property name="caches"> + <list> + <value>cache1</value> + <value>cache2</value> + </list> + </property> + + <property name="onlyPrimary" value="false"/> + <property name="maxBatchSize" value="1024"/> +</bean> + +<!--Destination TcpDiscoverySpi for CDC streamer--> +<bean id="destination.TcpDiscoverySpi" class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> + <property name="ipFinder"> + <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> + <property name="addresses" value="127.0.0.1:47600..47610" /> + </bean> + </property> + + <property name="localPort" value="47601" /> + <property name="joinTimeout" value="10000" /> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. + +NOTE: Make sure both source and destination clusters are up. + == CDC replication using Kafka This way to replicate changes between clusters requires setting up two applications: . `ignite-cdc.sh` with `org.apache.ignite.cdc.kafka.IgniteToKafkaCdcStreamer` that will capture changes from source cluster and write it to Kafka topic. . `kafka-to-ignite.sh` that will read changes from Kafka topic and then write them to destination cluster. +NOTE: Please, enable `ignite-cdc-ext` to be able to run `kafka-to-ignite.sh`. + NOTE: Instances of `ignite-cdc.sh` with configured streamer should be started on each server node of source cluster to capture all changes. IMPORTANT: CDC trough Kafka requires _metadata topic with the only one partition_ for sequential ordering guarantees. image:../../assets/images/integrations/CDC-ignite2kafka.svg[] +==== Kafka Installation Review Comment: Let's use '===' here ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -263,6 +396,85 @@ NOTE: link:https://kafka.apache.org/documentation/#consumerconfigs_request.timeo `kafka-to-ignite.sh` uses the same logging configuration as the Ignite node does. The only difference is that the log is written in the "kafka-ignite-streamer.log" file. +=== Configuration example Review Comment: You already have "Configuration example" in this hierarchy ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -90,19 +130,76 @@ image:../../assets/images/integrations/CDC-ignite2ignite.svg[] | `MappingsCount` | Count of mappings events applied to destination cluster |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/IgniteToIgniteCdcStreamer.java[Ignite2IgniteCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<!--IgniteToIgniteCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.IgniteToIgniteCdcStreamer"> + <property name="destinationIgniteConfiguration"> + <bean class="org.apache.ignite.configuration.IgniteConfiguration"> + <property name="igniteInstanceName" value="cluster-cdc-client" /> + <property name="clientMode" value="true" /> + <property name="localHost" value="127.0.0.1" /> + <property name="discoverySpi" ref="destination.TcpDiscoverySpi"/> + </bean> + </property> + + <property name="caches"> + <list> + <value>cache1</value> + <value>cache2</value> + </list> + </property> + + <property name="onlyPrimary" value="false"/> + <property name="maxBatchSize" value="1024"/> +</bean> + +<!--Destination TcpDiscoverySpi for CDC streamer--> +<bean id="destination.TcpDiscoverySpi" class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> + <property name="ipFinder"> + <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> + <property name="addresses" value="127.0.0.1:47600..47610" /> + </bean> + </property> + + <property name="localPort" value="47601" /> + <property name="joinTimeout" value="10000" /> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. + +NOTE: Make sure both source and destination clusters are up. Review Comment: For what? ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -30,7 +30,21 @@ NOTE: All implementations of the cross-cluster replication support replication o NOTE: To use SQL queries on the destination cluster over CDC-replicated data, set the same `VALUE_TYPE` in link:sql-reference/ddl#create-table[CREATE TABLE] on both source and destination clusters for each table. -== Ignite to Java Thin Client CDC streamer +==== Installation Review Comment: 4 indents makes this header small. Looks like we need 2 indents. ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -30,7 +30,21 @@ NOTE: All implementations of the cross-cluster replication support replication o NOTE: To use SQL queries on the destination cluster over CDC-replicated data, set the same `VALUE_TYPE` in link:sql-reference/ddl#create-table[CREATE TABLE] on both source and destination clusters for each table. -== Ignite to Java Thin Client CDC streamer +==== Installation + +. Build `cdc-ext` module with maven: ++ +```console + $~/src/ignite-extensions/> mvn clean package -DskipTests + $~/src/ignite-extensions/> ls modules/cdc-ext/target | grep zip +ignite-cdc-ext.zip +``` + +. Unpack `ignite-cdc-ext.zip` archive to `$IGNITE_HOME` folder. Review Comment: We discuss that the command for unpack should be showed explicitly. ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -90,19 +130,76 @@ image:../../assets/images/integrations/CDC-ignite2ignite.svg[] | `MappingsCount` | Count of mappings events applied to destination cluster |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/IgniteToIgniteCdcStreamer.java[Ignite2IgniteCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<!--IgniteToIgniteCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.IgniteToIgniteCdcStreamer"> + <property name="destinationIgniteConfiguration"> + <bean class="org.apache.ignite.configuration.IgniteConfiguration"> + <property name="igniteInstanceName" value="cluster-cdc-client" /> + <property name="clientMode" value="true" /> + <property name="localHost" value="127.0.0.1" /> + <property name="discoverySpi" ref="destination.TcpDiscoverySpi"/> + </bean> + </property> + + <property name="caches"> + <list> + <value>cache1</value> + <value>cache2</value> + </list> + </property> + + <property name="onlyPrimary" value="false"/> + <property name="maxBatchSize" value="1024"/> +</bean> + +<!--Destination TcpDiscoverySpi for CDC streamer--> +<bean id="destination.TcpDiscoverySpi" class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> + <property name="ipFinder"> + <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> + <property name="addresses" value="127.0.0.1:47600..47610" /> + </bean> + </property> + + <property name="localPort" value="47601" /> + <property name="joinTimeout" value="10000" /> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. Review Comment: Why do you use a note here, while in other blocks mention it in a sentence before configuration example? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@ignite.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
