Hi,
It may be not for your case, but I have implemented an example about kafka
transaction:
https://medium.com/@mykidong/kafka-transaction-56f022af1b0c
, in this example, offsets are saved to external db.
- Kidong
2019년 10월 31일 (목) 오전 11:39, Sergi Vladykin 님이 작성:
> Ok, so what is the advi
Hi,
I have written a tool like mirror maker:
https://github.com/mykidong/kafka-spiegel/blob/master/README.md
You can try this.
Cheers,
Kidong Lee
2019년 5월 4일 (토) 오후 8:38, ASHOK MACHERLA 님이 작성:
>
>
> Sent from Outlook
>
> On 30-Apr-2019 3:37 PM, ASHOK MACHERLA wrote
-view-event");
// consul agent host and port.
String agentHost = "localhost";
int agentPort = 8500;
AvroDeserializeService avroDeserializeService = new
ConsulAvroDeserializeService(topicAndPathProps, agentHost, agentPort);
Schema schema = avroDeserializeService.getSchema("item-
You can send avro record to kafka and consume it without schema registry.
In my approach, avro schema file avsc must be in the classpath on both
producer and consumer side.
On producer side, first write value avro serializer and set the properties
of key.serializer and value.serializer to kafka p
and Consul. This functionality
> > should actually be pluggable with Connect by implementing a custom
> > `Converter`, e.g. the SR comes with AvroConverter which acts as the glue.
> > Converter classes can be specified with the `key.converter` and
> > `value.converter` confi
Hi,
I have written a simple Kafka ETL which consumes avro encoded data from
Kafka and save them to Parquet on HDFS:
https://github.com/mykidong/kafka-etl-consumer
It is implemented with Kafka Consumer API and Parquet Writer API.
- Kidong Lee.
izer which I'm
> not
> > aware of any existing ones. In that case, you'd have to write your
> > own,
> > and your AvroSerializer is probably a good thing to template from.
> > Then
> > you would just use the HDFSSink Connector again and chang
uet on hdfs using
Avro schema which is located in the classpath, for instance,
/META-INF/avro/xxx.avsc.
Any idea to write parquet sink?
- Kidong Lee.
uet on hdfs using
Avro schema which is located in the classpath, for instance,
/META-INF/avro/xxx.avsc.
Any idea to write parquet sink?
- Kidong Lee.
