jmckenzie-dev commented on code in PR #330:
URL: https://github.com/apache/cassandra-sidecar/pull/330#discussion_r3283939411
##########
scripts/build-shaded-dtest-jar-local.sh:
##########
@@ -35,6 +35,9 @@ echo "${JAVA_HOME}"
# The container that runs the script has jdk11 installed only.
# Setting the env var to build with jdk11.
export CASSANDRA_USE_JDK11=true
+# Trunk (5.1-SNAPSHOT) compiles 2700+ source files; without an explicit heap
+# limit ant hits the JVM default (~2GB) and gets OOM-killed on large executors.
+export ANT_OPTS="${ANT_OPTS:-} -Xmx4g"
Review Comment:
Is this related to this CDC dockerfile demo setup out of curiosity? I'm fine
either way, just wondered how they connected.
##########
server/build.gradle:
##########
@@ -159,6 +159,9 @@ dependencies {
implementation
"org.apache.kafka:kafka-clients:${project.kafkaClientVersion}"
implementation "com.esotericsoftware:kryo-shaded:${kryoVersion}"
+ // Confluent Avro serializer — used when
value.serializer=KafkaAvroSerializer (confluent mode)
+ implementation 'io.confluent:kafka-avro-serializer:7.6.0'
Review Comment:
Is this only needed for this demo? Or do we need this for execution in the
sidecar broadly for the avro serializer support in CDC? If the former, I'm a
little on the fence on including this non-trivial dependency in the broader
project vs. having a separate gradle subproject for the demo so it doesn't
pollute the primary project namespaces w/stuff we only need for the demo.
##########
docker/cdc-demo/scripts/start.sh:
##########
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Builds the sidecar and starts the CDC demo stack.
+#
+# Usage (from anywhere in the repo):
+# ./scripts/start.sh # build + start in confluent mode
(default)
+# ./scripts/start.sh --bytearray # build + start in bytearray mode
+# ./scripts/start.sh --clean # wipe all data volumes before
starting
+# ./scripts/start.sh --skip-build # reuse existing
cassandra-sidecar:dev image
+# ./scripts/start.sh --clean --skip-build
+set -euo pipefail
+
+# ANSI color codes
+BOLD='\033[1m'
+GREEN='\033[0;32m'
+CYAN='\033[0;36m'
+YELLOW='\033[0;33m'
+UNDERLINE='\033[4m'
+RESET='\033[0m'
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+DEMO_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+CLEAN=false
+SKIP_BUILD=false
+SERIALIZER_MODE=confluent
+
+for arg in "$@"; do
+ case "$arg" in
+ --clean) CLEAN=true ;;
+ --skip-build) SKIP_BUILD=true ;;
+ --confluent) SERIALIZER_MODE=confluent ;;
+ --bytearray) SERIALIZER_MODE=bytearray ;;
+ *) echo "Unknown argument: $arg" >&2; exit 1 ;;
+ esac
+done
+
+# ── Stop existing stack
───────────────────────────────────────────────────────
+if $CLEAN; then
+ bash "$SCRIPT_DIR/stop.sh" --clean
+else
+ bash "$SCRIPT_DIR/stop.sh"
+fi
+
+# ── Build
─────────────────────────────────────────────────────────────────────
+if $SKIP_BUILD; then
+ if ! docker image inspect cassandra-sidecar:dev > /dev/null 2>&1; then
+ echo "ERROR: --skip-build specified but cassandra-sidecar:dev image
not found." >&2
+ echo " Run without --skip-build to build the image first." >&2
+ exit 1
+ fi
+ printf "${YELLOW}Skipping build — reusing existing cassandra-sidecar:dev
image.${RESET}\n"
+else
+ printf "${BOLD}==> Building sidecar distribution (./gradlew
installDist)...${RESET}\n"
+ "$REPO_ROOT/gradlew" -p "$REPO_ROOT" installDist \
+ -x test -x integrationTest -x containerTest \
+ --parallel --quiet
+
+ printf "${BOLD}==> Building sidecar Docker image...${RESET}\n"
+ DOCKER_BUILDKIT=1 docker build \
+ -f "$REPO_ROOT/docker/cdc-demo/Dockerfile.sidecar" \
+ -t cassandra-sidecar:dev \
+ "$REPO_ROOT"
+fi
+
+# ── Start stack
───────────────────────────────────────────────────────────────
+printf "${BOLD}==> Starting stack (serializer-mode:
${SERIALIZER_MODE})...${RESET}\n"
+cd "$DEMO_DIR"
+export SERIALIZER_MODE
+docker compose up -d
+
+# ── Wait for sidecar ─────────────────────────────────────────────────────────
+echo ""
+echo "Waiting for sidecar to be ready (follow progress: docker compose logs -f
cassandra-init sidecar)..."
+until curl -sf http://localhost:9043/api/v1/__health > /dev/null 2>&1; do
Review Comment:
This can infinite loop in the error scenario; a timeout would be friendly. 5
minutes or something - whatever makes sense.
##########
docker/cdc-demo/scripts/start.sh:
##########
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Builds the sidecar and starts the CDC demo stack.
+#
+# Usage (from anywhere in the repo):
+# ./scripts/start.sh # build + start in confluent mode
(default)
+# ./scripts/start.sh --bytearray # build + start in bytearray mode
+# ./scripts/start.sh --clean # wipe all data volumes before
starting
+# ./scripts/start.sh --skip-build # reuse existing
cassandra-sidecar:dev image
+# ./scripts/start.sh --clean --skip-build
+set -euo pipefail
+
+# ANSI color codes
+BOLD='\033[1m'
+GREEN='\033[0;32m'
+CYAN='\033[0;36m'
+YELLOW='\033[0;33m'
+UNDERLINE='\033[4m'
+RESET='\033[0m'
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+DEMO_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+CLEAN=false
+SKIP_BUILD=false
+SERIALIZER_MODE=confluent
+
+for arg in "$@"; do
Review Comment:
No `--help` - could we add that?
##########
docker/cdc-demo/scripts/start.sh:
##########
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Builds the sidecar and starts the CDC demo stack.
+#
+# Usage (from anywhere in the repo):
+# ./scripts/start.sh # build + start in confluent mode
(default)
+# ./scripts/start.sh --bytearray # build + start in bytearray mode
+# ./scripts/start.sh --clean # wipe all data volumes before
starting
+# ./scripts/start.sh --skip-build # reuse existing
cassandra-sidecar:dev image
+# ./scripts/start.sh --clean --skip-build
+set -euo pipefail
+
+# ANSI color codes
+BOLD='\033[1m'
+GREEN='\033[0;32m'
+CYAN='\033[0;36m'
+YELLOW='\033[0;33m'
+UNDERLINE='\033[4m'
+RESET='\033[0m'
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+DEMO_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+CLEAN=false
+SKIP_BUILD=false
+SERIALIZER_MODE=confluent
+
+for arg in "$@"; do
+ case "$arg" in
+ --clean) CLEAN=true ;;
+ --skip-build) SKIP_BUILD=true ;;
+ --confluent) SERIALIZER_MODE=confluent ;;
+ --bytearray) SERIALIZER_MODE=bytearray ;;
+ *) echo "Unknown argument: $arg" >&2; exit 1 ;;
+ esac
+done
+
+# ── Stop existing stack
───────────────────────────────────────────────────────
+if $CLEAN; then
+ bash "$SCRIPT_DIR/stop.sh" --clean
+else
+ bash "$SCRIPT_DIR/stop.sh"
+fi
+
+# ── Build
─────────────────────────────────────────────────────────────────────
+if $SKIP_BUILD; then
+ if ! docker image inspect cassandra-sidecar:dev > /dev/null 2>&1; then
+ echo "ERROR: --skip-build specified but cassandra-sidecar:dev image
not found." >&2
+ echo " Run without --skip-build to build the image first." >&2
+ exit 1
+ fi
+ printf "${YELLOW}Skipping build — reusing existing cassandra-sidecar:dev
image.${RESET}\n"
+else
+ printf "${BOLD}==> Building sidecar distribution (./gradlew
installDist)...${RESET}\n"
+ "$REPO_ROOT/gradlew" -p "$REPO_ROOT" installDist \
+ -x test -x integrationTest -x containerTest \
+ --parallel --quiet
+
+ printf "${BOLD}==> Building sidecar Docker image...${RESET}\n"
+ DOCKER_BUILDKIT=1 docker build \
+ -f "$REPO_ROOT/docker/cdc-demo/Dockerfile.sidecar" \
+ -t cassandra-sidecar:dev \
+ "$REPO_ROOT"
+fi
+
+# ── Start stack
───────────────────────────────────────────────────────────────
+printf "${BOLD}==> Starting stack (serializer-mode:
${SERIALIZER_MODE})...${RESET}\n"
+cd "$DEMO_DIR"
+export SERIALIZER_MODE
+docker compose up -d
+
+# ── Wait for sidecar ─────────────────────────────────────────────────────────
+echo ""
+echo "Waiting for sidecar to be ready (follow progress: docker compose logs -f
cassandra-init sidecar)..."
+until curl -sf http://localhost:9043/api/v1/__health > /dev/null 2>&1; do
+ sleep 5
+done
+
+echo "Sidecar is up. Waiting for CDC iterators to start..."
+# timeout(1) is GNU coreutils — not available by default on macOS.
+if command -v timeout > /dev/null 2>&1; then
+ timeout 120 bash -c \
+ 'docker compose logs -f sidecar 2>&1 | grep -m 1 "CDC iterators
started successfully"' \
+ > /dev/null || echo "Warning: timed out waiting for CDC iterators —
check: docker compose logs sidecar"
+else
+ docker compose logs -f sidecar 2>&1 | grep -m 1 "CDC iterators started
successfully" > /dev/null || true
Review Comment:
Do we have any other options on macOS instead of just "follow forever"? :)
##########
docker/cdc-demo/README.md:
##########
@@ -0,0 +1,287 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+# CDC Demo — Docker Compose Setup
+
+End-to-end demo that boots Cassandra, Cassandra Sidecar, Kafka, and Confluent
+Schema Registry. Writes to a CDC-enabled Cassandra table are captured by the
+sidecar, serialized as Avro (with schemas registered in Schema Registry), and
+published to a Kafka topic.
+
+## Architecture
+
+```
+┌──────────────┐ cdc_raw/commitlog ┌──────────────────┐
+│ Cassandra │ ─────────────────────►│ Cassandra │──► Kafka topic
+│ (port 9042) │ (shared volume) │ Sidecar │ (cdc-mutations)
+└──────────────┘ │ (port 9043) │
+ └──────────────────┘
+ │ KafkaAvroSerializer
+ ▼
+ ┌──────────────────┐
+ │ Schema Registry │
+ │ (port 8081) │
+ └──────────────────┘
+
+ ┌──────────────────┐
+ │ Kafka UI │
+ │ (port 8080) │
+ └──────────────────┘
+```
+
+**Services:**
+| Service | Image | Role |
+|---|---|---|
+| `kafka` | `confluentinc/cp-kafka:7.6.0` | KRaft broker (no ZooKeeper) |
+| `schema-registry` | `confluentinc/cp-schema-registry:7.6.0` | Avro schema
store |
+| `cassandra` | `cassandra:5.0` | CDC-enabled Cassandra node |
+| `cassandra-init` | `cassandra:5.0` | One-shot: seeds sidecar schema +
configs |
+| `sidecar` | `cassandra-sidecar:dev` | Reads commit logs, publishes to Kafka |
+| `kafka-ui` | `ghcr.io/kafbat/kafka-ui:v1.6.1` | Browse topics + decoded Avro
messages |
Review Comment:
The file has 1.5.0 and we have 1.6.1 here - update this or that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
