HiApache Kafka users email distribution list.
I'm trying to post to connect-offsets topic a message with a lsn from the
past. I dump the connect offsets topic with the following command:
./kafka-console-consumer.sh --bootstrap-server <conn string>
--consumer.config ../config/consumer.properties --property print.key=true
--new-consumer --topic connect-offsets
(config-consumer.properties has a group.id and SSL configuration
information).
Then I'll extract a message, and keeping all the key & value fields the
same with the exception of using an old LSN, I turn off Kafka Connect and
produce to the connect-offsets topic with a connection formed with the
following command:
[root@billing-kafka001 bin]# ./kafka-console-producer.sh --broker-list <conn
string> --topic connect-offsets --property "key.separator=," --property
"parse.key=true" --property "compression.codec=1" --producer.config
../config/producer.properties
(producer.properties has only ssl configuration settings).
The messages look like this:
["postgres-events-connector",{"server":"staging-billing"}]{"last_snapshot_record":true,"lsn":1275604312,"txId":2125,"ts_usec":1520019207386333000,"snapshot":true}
When Kafka Connect is restarted, however, the offset source_info reported
is that of the last commit issued by the producer, not my reset point.
Am I obtaining the key in the wrong fashion? Should txId be reset to the
value of commit I obtained the LSN from (I tested this and my results were
no better)? Am I using the correct compression codec for the topic?
Any assistance would be greatly appreciated!!
Thanks in advance for any assistance anyone can provide,
Joseph Hammerman
