maksaska commented on code in PR #11497: URL: https://github.com/apache/ignite/pull/11497#discussion_r1981912018
########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -90,19 +130,76 @@ image:../../assets/images/integrations/CDC-ignite2ignite.svg[] | `MappingsCount` | Count of mappings events applied to destination cluster |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/IgniteToIgniteCdcStreamer.java[Ignite2IgniteCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<!--IgniteToIgniteCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.IgniteToIgniteCdcStreamer"> + <property name="destinationIgniteConfiguration"> + <bean class="org.apache.ignite.configuration.IgniteConfiguration"> + <property name="igniteInstanceName" value="cluster-cdc-client" /> + <property name="clientMode" value="true" /> + <property name="localHost" value="127.0.0.1" /> + <property name="discoverySpi" ref="destination.TcpDiscoverySpi"/> + </bean> + </property> + + <property name="caches"> + <list> + <value>cache1</value> + <value>cache2</value> + </list> + </property> + + <property name="onlyPrimary" value="false"/> + <property name="maxBatchSize" value="1024"/> +</bean> + +<!--Destination TcpDiscoverySpi for CDC streamer--> +<bean id="destination.TcpDiscoverySpi" class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> + <property name="ipFinder"> + <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> + <property name="addresses" value="127.0.0.1:47600..47610" /> + </bean> + </property> + + <property name="localPort" value="47601" /> + <property name="joinTimeout" value="10000" /> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. + +NOTE: Make sure both source and destination clusters are up. + == CDC replication using Kafka This way to replicate changes between clusters requires setting up two applications: . `ignite-cdc.sh` with `org.apache.ignite.cdc.kafka.IgniteToKafkaCdcStreamer` that will capture changes from source cluster and write it to Kafka topic. . `kafka-to-ignite.sh` that will read changes from Kafka topic and then write them to destination cluster. +NOTE: Please, enable `ignite-cdc-ext` to be able to run `kafka-to-ignite.sh`. Review Comment: Yep. Removed it ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -165,33 +262,69 @@ section of the official Kafka documentation. | `MarkersCount` | Count of metadata markers sent to Kafka. |=== -=== `kafka-to-ignite.sh` application +=== Configuration example -This application should be started near the destination cluster. -`kafka-to-ignite.sh` will read CDC events from Kafka topic and then apply them to destination cluster. +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/IgniteToKafkaCdcStreamer.java[Ignite2KafkaCdcStreamer] bean configuration to the source ignite configuration, and use the resulting configuration with `ignite-cdc.sh` to start CDC client on the source Ignite instance. -IMPORTANT: `kafka-to-ignite.sh` implements the fail-fast approach. It just fails in case of any error. The restart procedure should be configured with the OS tools. +```xml +<!--Kafka properties for CDC streamer--> +<util:properties id="kafkaProperties" location="file:/config/path/kafka.properties"/> -Count of instances of the application does not corellate to the count of destination server nodes. -It should be just enough to process source cluster load. -Each instance of application will process configured subset of topic partitions to spread the load. -`KafkaConsumer` for each partition will be created to ensure fair reads. +<!--IgniteToKafkaCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.kafka.IgniteToKafkaCdcStreamer"> + <property name="topic" value="dc1_to_dc2"/> + <property name="metadataTopic" value="metadata_from_dc1"/> + <property name="kafkaPartitions" value="16"/> + <property name="caches"> + <list> + <value>terminator</value> + </list> + </property> + <property name="maxBatchSize" value="1024"/> + <property name="onlyPrimary" value="false"/> + <property name="kafkaProperties" ref="kafkaProperties"/> +</bean> +``` -==== Installation +Use the following steps to fully configure the replication with Apache Kafka: + +1. Pre-activate source cluster before starting the CDC client. You can use `command.sh` for that. -. Build `cdc-ext` module with maven: -+ -```console - $~/src/ignite-extensions/> mvn clean package -DskipTests - $~/src/ignite-extensions/> ls modules/cdc-ext/target | grep zip -ignite-cdc-ext.zip +``` +./control.sh --set-state ACTIVE --host localhost:server_connector_port --yes ``` -. Unpack `ignite-cdc-ext.zip` archive to `$IGNITE_HOME` folder. +2. Use the resulting configuration with `ignite-cdc.sh` to start CDC client on the source Ignite instance. Review Comment: Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@ignite.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
