Hi,
We have an in-house cluster of Kafka brokers and ZooKeepers.
Kafka version kafka_2.12-2.7.0
ZooKeeper version apache-zookeeper-3.6.2-bin
The topic is published to Google BigQuery.
We would like to use a schema registry as opposed to sending schema with
payload for each message that adds to the volume of traffic.
As I understand from the available literature, this schema registry acts as
a a kind of API to decouple consumers from producers and provide a common
interface for topic metadata.
Currently I am looking at the open source community version for creating
and maintaining this schema registry. We tried wepay provider from
Confluent (which I assumed it seamlessly allow one to publish kafka topic
(source Json) to Google BigQuery but still getting the error:
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask
due to unrecoverable exception.
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:614)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
at
org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
com.wepay.kafka.connect.bigquery.exception.ConversionConnectException:
Top-level Kafka Connect schema must be of type 'struct'
Appreciate any advice.
Thanks
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
