Hi,
To stream data from kafka topic to Google BigQuery, i see some references
to
wepay-kafka-connect-bigquery-2.1.0
However, this sounds like being depreciated.
Currently we can stream directory through Spark Structured Streaming with
writeStream. \
outputMode('append'). \
option("truncate", "false"). \
foreachBatch(SendToBigQuery). \
This will use Spark-BigQuery API which isd pretty efficient .
I was looking at the Kafka connector for BigQuery and it appears that some
documents may not be the latest. For example
wepay-kafka-connect-bigquery-2.1.0
For example in the past Aerospike (NoSql database vendor) provided
connector for us and we just used the command
$KAFKA_HOME/bin/connect-standalone.sh \
/opt/aerospike-kafka-connect-sink/etc/connect-standalone.properties \
/opt/aerospike-kafka-connect-sink/etc/aerospike-sink.properties
The templates for those two property files were provided (plus jars) and we
updated them accordingly to get the stream going directly from Kafka to the
Aerospike table.
Can anyone point out to the latest version of kafka connector to Bigquery
documents?
Many thanks,
Mich
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
