timoninmaxim commented on code in PR #11497: URL: https://github.com/apache/ignite/pull/11497#discussion_r1966495161
########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -60,6 +60,34 @@ image:../../assets/images/integrations/CDC-ignite2igniteClient.svg[] | `MappingsCount` | Count of mappings events applied to destination cluster |=== +=== Configuration example Review Comment: Let's also fix name of the head. "Ignite to Java Thin Client CDC streamer" - looks like "Ignite to **Ignite**" missed ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -60,6 +60,34 @@ image:../../assets/images/integrations/CDC-ignite2igniteClient.svg[] | `MappingsCount` | Count of mappings events applied to destination cluster |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/thin/IgniteToIgniteClientCdcStreamer.java[Ignite2IgniteClientCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<bean id="cdc.streamer" class="org.apache.ignite.cdc.thin.IgniteToIgniteClientCdcStreamer"> + <property name="destinationClientConfiguration"> + <bean class="org.apache.ignite.configuration.ClientConfiguration"> + <property name="addresses" value="127.0.0.1:10800" /> + </bean> + </property> + + <property name="caches"> + <list> + <value>cache1</value> + <value>cache2</value> + </list> + </property> + + <property name="onlyPrimary" value="false"/> + <property name="maxBatchSize" value="1024"/> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. + +NOTE: Make sure both source and destination clusters are up. Review Comment: Any notes should be near text before config ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -165,14 +237,64 @@ section of the official Kafka documentation. | `MarkersCount` | Count of metadata markers sent to Kafka. |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/IgniteToKafkaCdcStreamer.java[Ignite2KafkaCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<!--Kafka properties for CDC streamer--> +<util:properties id="kafkaProperties" location="file:/config/path/kafka.properties"/> + +<!--IgniteToKafkaCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.kafka.IgniteToKafkaCdcStreamer"> + <property name="topic" value="dc1_to_dc2"/> + <property name="metadataTopic" value="metadata_from_dc1"/> + <property name="kafkaPartitions" value="16"/> + <property name="caches"> + <list> + <value>terminator</value> + </list> + </property> + <property name="maxBatchSize" value="1024"/> + <property name="onlyPrimary" value="false"/> + <property name="kafkaProperties" ref="kafkaProperties"/> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. + +NOTE: You need to preactivate source cluster before starting the CDC client. You can use `command.sh` for that Review Comment: Why this requirement only for Kafka streamer? What about in-memory clusters? ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -60,6 +60,34 @@ image:../../assets/images/integrations/CDC-ignite2igniteClient.svg[] | `MappingsCount` | Count of mappings events applied to destination cluster |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/thin/IgniteToIgniteClientCdcStreamer.java[Ignite2IgniteClientCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<bean id="cdc.streamer" class="org.apache.ignite.cdc.thin.IgniteToIgniteClientCdcStreamer"> + <property name="destinationClientConfiguration"> + <bean class="org.apache.ignite.configuration.ClientConfiguration"> + <property name="addresses" value="127.0.0.1:10800" /> + </bean> + </property> + + <property name="caches"> + <list> + <value>cache1</value> + <value>cache2</value> + </list> + </property> + + <property name="onlyPrimary" value="false"/> + <property name="maxBatchSize" value="1024"/> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. Review Comment: Please merge this sentence and first sentence in this block. ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -165,14 +237,64 @@ section of the official Kafka documentation. | `MarkersCount` | Count of metadata markers sent to Kafka. |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/IgniteToKafkaCdcStreamer.java[Ignite2KafkaCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<!--Kafka properties for CDC streamer--> +<util:properties id="kafkaProperties" location="file:/config/path/kafka.properties"/> + +<!--IgniteToKafkaCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.kafka.IgniteToKafkaCdcStreamer"> + <property name="topic" value="dc1_to_dc2"/> + <property name="metadataTopic" value="metadata_from_dc1"/> + <property name="kafkaPartitions" value="16"/> + <property name="caches"> + <list> + <value>terminator</value> + </list> + </property> + <property name="maxBatchSize" value="1024"/> + <property name="onlyPrimary" value="false"/> + <property name="kafkaProperties" ref="kafkaProperties"/> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. + +NOTE: You need to preactivate source cluster before starting the CDC client. You can use `command.sh` for that + +``` +./control.sh --set-state ACTIVE --host localhost:server_connector_port --yes +``` + +You can specify Kafka producer properties in the separate file `kafka.properties` + +```xml +bootstrap.servers=xxx.x.x.x:9092 +request.timeout.ms=10000 +``` + +NOTE: Configure Kafka topics beforehand. CDC clients will fail on trying to connect, if Kafka topics were not started. Review Comment: To many notes actually. It should be replaced with step-by-step instruction instead ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -193,6 +315,17 @@ Now, you have additional binary `$IGNITE_HOME/bin/kafka-to-ignite.sh` and `$IGNI NOTE: Please, enable `ignite-cdc-ext` to be able to run `kafka-to-ignite.sh`. +==== Kafka Installation Review Comment: Mention about this prerequisite in beginning of the example, not in the end. ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -339,3 +553,132 @@ Configuration is done via Ignite node plugin: </bean> </property> ``` + +== Common CDC strategies Review Comment: Replication strategies. CDC is just an instrument for implementing replication. There are different types/strategy for replication. ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -339,3 +553,132 @@ Configuration is done via Ignite node plugin: </bean> </property> ``` + +== Common CDC strategies + +There are basically two strategies to choose from when it comes to CDC replication with Apache Ignite, and a handful of options to configure the transport for the data. + +* Strategies include `Active-Passive` and `Active-Active` replication regimes. You can read it as 'from one cluster to another only' and 'from one cluster to another and vice versa'. + +The former strategy implies that only one cluster would be actively used for data consumption from the outside user application, while the other one consumes data through CDC. + +The latter approach allows users to use both cluster simultaneously, meanwhile the CDC clients transfer the updates between them. For that strategy to work, the user should configure conflict resolver. + +* To make things work you have two independent tools to configure: Review Comment: Why do you use this hierarchy? ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -339,3 +553,132 @@ Configuration is done via Ignite node plugin: </bean> </property> ``` + +== Common CDC strategies + +There are basically two strategies to choose from when it comes to CDC replication with Apache Ignite, and a handful of options to configure the transport for the data. + +* Strategies include `Active-Passive` and `Active-Active` replication regimes. You can read it as 'from one cluster to another only' and 'from one cluster to another and vice versa'. Review Comment: Split it on 2 paragraphs ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -165,14 +237,64 @@ section of the official Kafka documentation. | `MarkersCount` | Count of metadata markers sent to Kafka. |=== +=== Configuration example + +Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/IgniteToKafkaCdcStreamer.java[Ignite2KafkaCdcStreamer] bean configuration to the source ignite configuration. + +```xml +<!--Kafka properties for CDC streamer--> +<util:properties id="kafkaProperties" location="file:/config/path/kafka.properties"/> + +<!--IgniteToKafkaCdcStreamer--> +<bean id="cdc.streamer" class="org.apache.ignite.cdc.kafka.IgniteToKafkaCdcStreamer"> + <property name="topic" value="dc1_to_dc2"/> + <property name="metadataTopic" value="metadata_from_dc1"/> + <property name="kafkaPartitions" value="16"/> + <property name="caches"> + <list> + <value>terminator</value> + </list> + </property> + <property name="maxBatchSize" value="1024"/> + <property name="onlyPrimary" value="false"/> + <property name="kafkaProperties" ref="kafkaProperties"/> +</bean> +``` + +NOTE: Use the resulting configuation with `ignite-cdc.sh` to start CDC client on the source Ignite instance. + +NOTE: You need to preactivate source cluster before starting the CDC client. You can use `command.sh` for that + +``` +./control.sh --set-state ACTIVE --host localhost:server_connector_port --yes +``` + +You can specify Kafka producer properties in the separate file `kafka.properties` + +```xml +bootstrap.servers=xxx.x.x.x:9092 +request.timeout.ms=10000 +``` + +NOTE: Configure Kafka topics beforehand. CDC clients will fail on trying to connect, if Kafka topics were not started. + +In this CDC scenario we use Kafka with Zookeeper. For that to work, set two Kafka topics prior to CDC start-up. We used 'dc1_to_dc2' and 'metadata_from_dc1' naming respectively. You can use the following commands: + +``` +./kafka-topics.sh --create --partitions 16 --replication-factor 1 --topic dc1_to_dc2 --bootstrap-server localhost:9092 +./kafka-topics.sh --create --partitions 1 --replication-factor 1 --topic metadata_from_dc1 --bootstrap-server localhost:9092 +``` + +NOTE: For Active-Passive replication regime with Kafka two topics would suffice. For Active-Active replication regime you need another two topics to transfer data from the second cluster. Review Comment: regime -> mode? or even can be just ommitted at all. ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -339,3 +553,132 @@ Configuration is done via Ignite node plugin: </bean> </property> ``` + +== Common CDC strategies + +There are basically two strategies to choose from when it comes to CDC replication with Apache Ignite, and a handful of options to configure the transport for the data. + +* Strategies include `Active-Passive` and `Active-Active` replication regimes. You can read it as 'from one cluster to another only' and 'from one cluster to another and vice versa'. + +The former strategy implies that only one cluster would be actively used for data consumption from the outside user application, while the other one consumes data through CDC. + +The latter approach allows users to use both cluster simultaneously, meanwhile the CDC clients transfer the updates between them. For that strategy to work, the user should configure conflict resolver. + +* To make things work you have two independent tools to configure: + +** Thin/Thick clients for connection to destination clusters - responsible for 'put' operations on the destination cluster. + +** Ignite/Kafka driven middleman data transport - You can pass data through Kafka topics, or you can use Ignite out of the box solution. + +Basically, you can combine them any way you want to meet your project goals. + +=== CDC example manager Review Comment: Please make step-by-step instructions for every example. ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -339,3 +553,132 @@ Configuration is done via Ignite node plugin: </bean> </property> ``` + +== Common CDC strategies + +There are basically two strategies to choose from when it comes to CDC replication with Apache Ignite, and a handful of options to configure the transport for the data. + +* Strategies include `Active-Passive` and `Active-Active` replication regimes. You can read it as 'from one cluster to another only' and 'from one cluster to another and vice versa'. + +The former strategy implies that only one cluster would be actively used for data consumption from the outside user application, while the other one consumes data through CDC. + +The latter approach allows users to use both cluster simultaneously, meanwhile the CDC clients transfer the updates between them. For that strategy to work, the user should configure conflict resolver. + +* To make things work you have two independent tools to configure: + +** Thin/Thick clients for connection to destination clusters - responsible for 'put' operations on the destination cluster. + +** Ignite/Kafka driven middleman data transport - You can pass data through Kafka topics, or you can use Ignite out of the box solution. + +Basically, you can combine them any way you want to meet your project goals. + +=== CDC example manager + +`ignite-cdc-ext` ships with CDC example manager `cdc-start-up.sh` alongside `kafka-to-ignite.sh`. You can find it under '$IGNITE_HOME/examples/config/cdc-start-up/' directory. + +You can use this script to start all kinds of replication strategies without any additional configuration. + +The script will use predefined configuration '.xml' files from '$IGNITE_HOME/examples/config/cdc-start-up/' directory. Feel free to examine them as you try the manager out. + +NOTE: Use `--help` to explore the manager capabilities + +NOTE: Please, enable `ignite-rest-http` and `ignite-json` to be able to run `cdc-start-up.sh` with `--check-cdc`. + +Examples for reference: + +* Help message + +``` +./cdc-start-up.sh --help +``` + +* Start Ignite node with specified properties: + +To start an Ignite cluster node, use `--ignite` or `-i` command with `cdc-start-up.sh`. You also need to specify properties holder directory. + +There are currently 2 examples for 2 clusters, that you can run simultaneously. You can find them under `$IGNITE_HOME/examples/config/cdc-start-up/cluster-1` and `$IGNITE_HOME/examples/config/cdc-start-up/cluster-2` as `ignite-cdc.properties`. These files contains all independent settings that you can tinker for your needs. + +NOTE: All properties files are preconfigured to work out of the box. Review Comment: Too many notes ########## docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc: ########## @@ -339,3 +553,132 @@ Configuration is done via Ignite node plugin: </bean> </property> ``` + +== Common CDC strategies + +There are basically two strategies to choose from when it comes to CDC replication with Apache Ignite, and a handful of options to configure the transport for the data. + +* Strategies include `Active-Passive` and `Active-Active` replication regimes. You can read it as 'from one cluster to another only' and 'from one cluster to another and vice versa'. + +The former strategy implies that only one cluster would be actively used for data consumption from the outside user application, while the other one consumes data through CDC. + +The latter approach allows users to use both cluster simultaneously, meanwhile the CDC clients transfer the updates between them. For that strategy to work, the user should configure conflict resolver. + +* To make things work you have two independent tools to configure: + +** Thin/Thick clients for connection to destination clusters - responsible for 'put' operations on the destination cluster. + +** Ignite/Kafka driven middleman data transport - You can pass data through Kafka topics, or you can use Ignite out of the box solution. + +Basically, you can combine them any way you want to meet your project goals. + +=== CDC example manager + +`ignite-cdc-ext` ships with CDC example manager `cdc-start-up.sh` alongside `kafka-to-ignite.sh`. You can find it under '$IGNITE_HOME/examples/config/cdc-start-up/' directory. + +You can use this script to start all kinds of replication strategies without any additional configuration. + +The script will use predefined configuration '.xml' files from '$IGNITE_HOME/examples/config/cdc-start-up/' directory. Feel free to examine them as you try the manager out. + +NOTE: Use `--help` to explore the manager capabilities + +NOTE: Please, enable `ignite-rest-http` and `ignite-json` to be able to run `cdc-start-up.sh` with `--check-cdc`. + +Examples for reference: + +* Help message + +``` +./cdc-start-up.sh --help +``` + +* Start Ignite node with specified properties: + +To start an Ignite cluster node, use `--ignite` or `-i` command with `cdc-start-up.sh`. You also need to specify properties holder directory. + +There are currently 2 examples for 2 clusters, that you can run simultaneously. You can find them under `$IGNITE_HOME/examples/config/cdc-start-up/cluster-1` and `$IGNITE_HOME/examples/config/cdc-start-up/cluster-2` as `ignite-cdc.properties`. These files contains all independent settings that you can tinker for your needs. Review Comment: Which examples do you mean? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@ignite.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
