Hi Marina,
We hit a similar problem with our S3 connectors. We added a level of
indirection, a JSON validating microservice, before publishing to the Kafka
topic. The microservice published non-JSON formatted messages to a separate
Kafka topic called error-jsons and we flushed those messages using
Kafka's own consumer is here:
2
3
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
test --from-beginning
Source: https://kafka.apache.org/quickstart
On Oct 17, 2017 8:25 AM, "Dhawan Gajendran"
wrote:
Kafka comes with a cli consumer which you cou
Kafka comes with a cli consumer which you could use to retrieve messages
from your topic.
https://github.com/edenhill/kafkacat/blob/master/README.md
Please keep in mind this is more for testing purposes than for production.
On Oct 17, 2017 8:20 AM, "Jaswinder Singh" wrote:
> Hello,
>
> My name
r)
>
> Pardon. I don't quite understand the above setup. Can you describe in more
> detail ?
>
> Which version of connector are you using ?
>
> Cheers
>
> On Mon, Oct 16, 2017 at 11:30 PM, Dhawan Gajendran <
> dhawan.gajend...@datavisor.com> wrote:
>
> >
Hi All,
We had done a POC with kafka s3 connect confluent platform and have been
trying to move the setup to production. In the current setup we have 7
kafka topics and each topic drains the messages to a s3 bucket using s3
confluent connect.
My setup:
There are 8 workers distributed across 8 ubu
