Re: Kafka Connect - Customize REST request headers

Thank you for the explanation, Chris. In case it helps, what I'm looking for is similar to KIP 577[1]. My specific example involves a hard-coded key/value pair that needs to be used for pod-to-pod as I can connect to any worker without that specific header, but workers cannot communicate amon

Re: Kafka Connect - Customize REST request headers

Hi Yeikel, Neat question! And thanks for the link to the RestClient code; very helpful. I don't believe there's a way to configure Kafka Connect to add these headers to forwarded requests right now. You may be able to do some kind of out-of-band proxy magic to intercept forwarded requests and ins

RE: Kafka Connect - AvroConverter - Unions with Enums Problem

On 2020/05/02 17:22:43 Nagendra Korrapati wrote: > Hello > > Sorry to write this in this mailing list. This is more specific to my problem in using Kafka Connect > > I have Union with Enum in it. ToConnectSchema converted it into Schema.STRUCT type with fields and the enum field as STRING. > > The

Re: Kafka Connect Schema from JSON

In case this is helpful, I wrote a (WIP) version that does what you say, but using JSONSchema instead of avro schema. https://github.com/ottomata/kafka-connect-jsonschema On Thu, Aug 29, 2019 at 12:48 PM Josef Hak wrote: > Hi, > please is it possible to create kafka connect Schema using json, e

Re: Kafka connect task assignment Improvement ( New Feature )

Manjeet, this sounds like a problem that exists outside Connect's purview. Connect has nothing to do with resource management. Ryanne On Mon, Jul 29, 2019, 2:05 PM Manjeet Duhan wrote: > Hi , > > This is Manjeet here working in operative media . I have been working on > confluent kafka for almo

Re: Kafka connect task assignment Improvement ( New Feature )

Dude don't put a lot of details about operative media and names. From: Manjeet Duhan Sent: Monday, July 29, 2019 11:28 PM To: dev@kafka.apache.org Cc: Praveen Manvi Subject: Kafka connect task assignment Improvement ( New Feature ) Hi , This is Manjeet here

RE: kafka connect stops consuming data when kafka broker goes down

, 2019 5:17 PM To: dev@kafka.apache.org Cc: Basil Brito, Aldan (Nokia - IN/Bangalore) Subject: Re: kafka connect stops consuming data when kafka broker goes down It’s not totally clear, but this may be https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-7941 For which there is a fix

Re: kafka connect stops consuming data when kafka broker goes down

It’s not totally clear, but this may be https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-7941 For which there is a fix that is very nearly approved: https://github.com/apache/kafka/pull/6283 Paul > On Jun 5, 2019, at 1:26 AM, Srinivas, Kaushik (Nokia - IN/Bangalore) > wrote:

RE: kafka connect stops consuming data when kafka broker goes down

Hello, Anyone has any information on this issue. Created a critical ticket for the same, since this is a major stability issue for connect framework. https://issues.apache.org/jira/browse/KAFKA-8485?filter=-2 Thanks. Kaushik, NOKIA From: Srinivas, Kaushik (Nokia - IN/Bangalore) Sent: Monday, Jun

RE: Kafka Connect

Yeah, I tried it. But it works only on CDC mode. And I need connector which executes queries. -Message d'origine- De : Sagar [mailto:sagarmeansoc...@gmail.com] Envoyé : mardi 4 juin 2019 15:29 À : dev@kafka.apache.org Objet : Re: Kafka Connect Hi meirkhan, You can give debezium a

Re: Kafka Connect

Hi meirkhan, You can give debezium a shot. It has a connector for mongodb On Tue, 4 Jun 2019 at 6:56 PM, wrote: > Hello Kafka! > > Here is my case. I need the Kafka Connector to Mongodb which imitates JDBC > Source Connector. > So that data is loaded to Kafka by periodically executing a query.

Re: Kafka Connect task re-balance repeatedly

Thanks Ewen. Will take a look at the config and if there are any findings, will come back here later. -- Sent from my iPhone On Mar 22, 2018, at 7:39 PM, Ewen Cheslack-Postava wrote: The log is showing that the Connect worker is trying to make sure it has read the entire log and gets to offset

Re: Kafka Connect task re-balance repeatedly

The log is showing that the Connect worker is trying to make sure it has read the entire log and gets to offset 119, but some other worker says it has read to offset 169. The two are in inconsistent states, so the one that seems to be behind will not start work with potentially outdated configurati

Re: Kafka Connect Source questions

On Thu, Oct 12, 2017 at 1:59 AM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Hi, > > > > I had a look at the Connect Source Worker code and have two questions: > When a Source Task commits offsets, does it perform compaction / > optimisation before sending off? E.g. I read from 1 s

Re: Kafka Connect suggestion before creating new KIP

On Mon, Jul 24, 2017 at 1:50 PM, Florian Hussonnois wrote: > Hi all, > > Here is two suggestions for the Kafka Connect API that I prefer to present > you first before creating KIPs. > > 1/ > Currently, in Kafka Connect we distinguish two configuration types: the > worgker config and the connector

Re: Kafka Connect Parquet Support?

Theoretically yes, we would want to support this in Confluent's S3 connector. One of the stumbling blocks is that apparently the Parquet code is somewhat tied to HDFS currently which causes problems when you're not using the HDFS S3 connectivity. See, e.g., https://github.com/confluentinc/kafka-con

Re: kafka connect

In case you do not find Kafka Connector for listening to TCP sockets, you can use NiFi for this task. NiFi has a processor to listen to TCP messages. On Tue, Jul 4, 2017 at 1:26 PM, Clay Teahouse wrote: > Hello All, > > I have a few questions regarding kafka connect. I'd appreciate your > replie

Re: kafka connect

If you are looking for connectors there is a good curated list of connectors here https://www.confluent.io/product/connectors/ I don't see tcp on the list but the general naming scheme for open source connectors on github is to call them “kafka-con

Re: kafka connect questions

Thanks Per for the feedback. My comments below. 1) There is a load balancer in front of the cluster. 2) Kafka does run in a cluster mode. I was referring to the producer that doesn't. Furthermore, the data sources are not heterogeneous and we cannot install anything on them. All we can do have th

Re: kafka connect questions

Hi Clay, I agree with Gwen in thinking that you might want to take a second look at streaming protobuf data to Kafka and then having connectors read that from Kafka. To address the issues in order: 1. You say you have hundreds of machines sending data, but if you run a connector that is not tied

Re: kafka connect questions

Well I guess there is one: https://github.com/dhanuka84/kafka-connect-tcp Maybe you can use or build on top of that. On 05/07/17 05:45, Gwen Shapira wrote: I don't remember seeing one. There is no reason not to write one (let us know if you do, so we can put it on the connector hub!). Few thing

Re: kafka connect questions

Hello Gwen, Thanks for the reply. My comments/answers inline. 1. Connectors that listen on sockets typically run in stand-alone mode, so they can tied to a specific machine (in distributed mode, connectors can move around). [Clay:] Even if the connectors move around, they can still listen to a sp

Re: kafka connect questions

I don't remember seeing one. There is no reason not to write one (let us know if you do, so we can put it on the connector hub!). Few things: 1. Connectors that listen on sockets typically run in stand-alone mode, so they can tied to a specific machine (in distributed mode, connectors can move aro

Re: Kafka Connect Parquet Support?

Hi Clayton, It seems like an interesting improvement. Given that Parquet is columnar, you would expect some space savings. I guess the big question is, would each batch of records become a single parquet file? And how does this integrate with the existing logic, which might assume that each rec

Re: Kafka Connect: To much restarting with a SourceConnector with dynamic set of tasks

Thanks a lot for responding, Randall! See my comments below. Regards, Per Steffensen On 22/05/17 22:36, Randall Hauch wrote: You're not doing anything wrong, but I suspect you're requesting task reconfiguration more frequently than was originally envisioned, which means that the current implem

Re: Kafka Connect: To much restarting with a SourceConnector with dynamic set of tasks

You're not doing anything wrong, but I suspect you're requesting task reconfiguration more frequently than was originally envisioned, which means that the current implementation is not as optimal for your case. I'm not sure how much effort is required to implement this new behavior. The logic for

RE: Kafka Connect and Partitions

ance. I think this puzzle is now > solved :) > Best wishes, > David > > -----Original Message- > From: Randall Hauch [mailto:rha...@gmail.com] > Sent: 28 April 2017 16:08 > To: dev@kafka.apache.org > Subject: Re: Kafka Connect and Partitions > > The source connector

Re: Kafka Connect and Partitions

ow > solved :) > Best wishes, > David > > -----Original Message- > From: Randall Hauch [mailto:rha...@gmail.com] > Sent: 28 April 2017 16:08 > To: dev@kafka.apache.org > Subject: Re: Kafka Connect and Partitions > > The source connector creates SourceRecord obje

RE: Kafka Connect and Partitions

r guidance. I think this puzzle is now solved :) Best wishes, David -Original Message- From: Randall Hauch [mailto:rha...@gmail.com] Sent: 28 April 2017 16:08 To: dev@kafka.apache.org Subject: Re: Kafka Connect and Partitions The source connector creates SourceRecord object and can se

RE: Kafka Connect and Partitions

these are related in order to figure out why the default partitioner is writing all data to the same (Kafka) partition. Still puzzled, David -Original Message- From: Randall Hauch [mailto:rha...@gmail.com] Sent: 28 April 2017 16:08 To: dev@kafka.apache.org Subject: Re: Kafka Connect and Pa

Re: Kafka Connect and Partitions

is > mapped to the same Kafka partition. > > Could you explain how the SourceRecord is used to derive the partition key > please. Can you see from the above summary why the partition key is null? > It defeats me :( > > Have a good weekend, thanks, > > David > > -Original Messag

RE: Kafka Connect and Partitions

SourceRecord is used to derive the partition key please. Can you see from the above summary why the partition key is null? It defeats me :( Have a good weekend, thanks, David -Original Message- From: Gwen Shapira [mailto:g...@confluent.io] Sent: 27 April 2017 17:44 To: dev@kafka.apac

Re: Kafka Connect and Partitions

1] Kafka Offset/Partition [90634/31] > ... > [41:30,584] INFO ### Topic [topic11] Kafka Offset/Partition [90639/31] > [41:30,584] INFO ### Topic [topic7] Kafka Offset/Partition [91011/31] > ... > [41:30,584] INFO ### Topic [topic7] Kafka Offset/Partition [91019/31] > [41:30,584

Re. Kafka Connect and Partitions

62883] has processed [0] events [41:31,178] INFO WorkerSinkTask{id=topicsToFolders-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSinkTask:261) -Original Message- From: Gwen Shapira [mailto:g...@confluent.io] Sent: 26 April 2017 18:29 To: dev@kafka.apache.org<mailto:dev@kafka.apache.org> Subje

RE: Kafka Connect and Partitions

records [41:31,178] INFO ### Writing [0] events [41:31,178] INFO ### Sink Task [518962883] has processed [0] events [41:31,178] INFO WorkerSinkTask{id=topicsToFolders-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSinkTask:261) -Original Message- From: Gwen Shapira [m

Re: Kafka Connect and Partitions

Hi, I'll need a bit more detail to help :) Are you writing a connector or trying to use an existing one? If existing, which connector? Is it source or sink? Here are few things I'd look at when debugging: * Is the connector reading from the topic you think you are reading? * Do you actually have

Re: Kafka Connect / Access to OffsetStorageReader from SourceConnector

Hi Florian, Just curious, what 'shared storage' you guys use to keep the files before ingested into Kafka? In our case, we could not figure out such a nice distributed+shared file system that is NOT HDFS alike and runs before Kafka. So we use individual harddisks on connector machines and keep of

Re: Kafka Connect / Access to OffsetStorageReader from SourceConnector

Hey Florian, It seems reasonable to me to let the connector track task progress through offsets. I recall there have been other use cases for communication between tasks and connectors (perhaps Ewen or someone else will jump in here and mention them), so I'm not sure if there if this could fall un

Re: Kafka Connect / Access to OffsetStorageReader from SourceConnector

Hi Jason, Yes, this is the idea. The connector assigns a subset of files to each task. A task stores the size of file, the bytes offset and the bytes size of the last sent record as a source offsets. A file is finished when recordBytesOffsets + recordBytesSize = fileBytesSize. The connector shou

Re: Kafka Connect / Access to OffsetStorageReader from SourceConnector

Hey Florian, Can you explain a bit more how having access to the offset storage from the connector helps in your use case? I guess you are planning to use offsets to be able to tell when a task has finished a file? Thanks, Jason On Fri, Feb 17, 2017 at 4:45 AM, Florian Hussonnois wrote: > Hi K

RE: Kafka Connect key.converter and value.converter properties for Avro encoding

on on the commit log that didn't give rise to events. I've never encountered null events in my previous experience with Kafka. Regards, David -Original Message- From: Ewen Cheslack-Postava [mailto:e...@confluent.io] Sent: 08 November 2016 04:38 To: dev@kafka.apache.org Subject:

Re: Kafka Connect key.converter and value.converter properties for Avro encoding

rue nulls in Kafka. > > Thanks again, > David > > > > > > > > -Original Message- > From: Ewen Cheslack-Postava [mailto:e...@confluent.io] > Sent: 07 November 2016 04:35 > To: dev@kafka.apache.org > Subject: Re: Kafka Connect key.converter and val

RE: Kafka Connect key.converter and value.converter properties for Avro encoding

vember 2016 04:35 To: dev@kafka.apache.org Subject: Re: Kafka Connect key.converter and value.converter properties for Avro encoding You won't be accepting/returning SpecificRecords directly when working with Connect's API. Connect intentionally uses an interface that is different from

Re: Kafka Connect key.converter and value.converter properties for Avro encoding

ly be achieved via a corresponding > SpecificDatumReader. > > Does this look a reasonable approach? > > Many thanks if you've read this far! > > Regards, > David > > > -Original Message- > From: Gwen Shapira [mailto:g...@confluent.io] > Sent: 02 November 2

RE: Kafka Connect key.converter and value.converter properties for Avro encoding

e achieved via a corresponding SpecificDatumReader. Does this look a reasonable approach? Many thanks if you've read this far! Regards, David -Original Message- From: Gwen Shapira [mailto:g...@confluent.io] Sent: 02 November 2016 21:18 To: dev@kafka.apache.org Subject: Re: Kafka Conn

Re: Kafka Connect key.converter and value.converter properties for Avro encoding

Both the Confluent Avro Converter and the Confluent Avro Serializer use the Schema Registry. The reason is, as Tommy Becker mentioned below, to avoid storing the entire schema in each record (which the JSON serializer in Apache Kafka does). It has few other benefits schema validation and such. If

Re: Kafka Connect key.converter and value.converter properties for Avro encoding

Although I can't speak to details of the Confluent packaging, anytime you're using Avro you need the schemas for the records you're working with. In an Avro data file the schema is included in the file itself. But when you're encoding individual records like in Kafka, most people instead encode

Re: Kafka Connect Transformers

Added wiki access. Enjoy :) On Fri, Jul 1, 2016 at 11:24 AM, Nisarg Shah wrote: > Need to submit a KIP for https://issues.apache.org/jira/browse/KAFKA-3209. > Please provide wiki write access to ‘snisarg’. > > Thanks, > Nisarg Shah. > >> On Jun 28, 2016, at 6:27 PM, Nisarg Shah wrote: >> >> Nee

Re: Kafka Connect Transformers

Need to submit a KIP for https://issues.apache.org/jira/browse/KAFKA-3209. Please provide wiki write access to ‘snisarg’. Thanks, Nisarg Shah. > On Jun 28, 2016, at 6:27 PM, Nisarg Shah wrote: > > Need permissions to edit the wiki. Username is ‘snisarg’. > > Thanks, > Nisarg. > >> On Jun 2

Re: Kafka Connect Transformers

Need permissions to edit the wiki. Username is ‘snisarg’. Thanks, Nisarg. > On Jun 28, 2016, at 09:08, Nisarg Shah wrote: > > Hello, > > I need to create a page so that I can write a Kafka Improvement Proposal for > the below. My username is ‘snisarg’. > > Thanks, > Nisarg > >> On Jun 19,

Re: Kafka Connect Transformers

Hello, I need to create a page so that I can write a Kafka Improvement Proposal for the below. My username is ‘snisarg’. Thanks, Nisarg > On Jun 19, 2016, at 10:43 PM, Nisarg Shah wrote: > > Hello, > > I am looking to do https://issues.apache.org/jira/browse/KAFKA-3209 >

Re: Kafka Connect connector monitoring tasks via offset storage reader

On May 24, 2016 at 4:28:41 PM, Liquan Pei (liquan...@gmail.com) wrote: Hi Randall,  This is interesting. Essentially we can track the progress by getting data  from offset storage. What are the use cases you have in mind that uses the  offsets of source partitions? I can imagine that comparing

Re: Kafka Connect connector monitoring tasks via offset storage reader

Hi Randall, This is interesting. Essentially we can track the progress by getting data from offset storage. What are the use cases you have in mind that uses the offsets of source partitions? I can imagine that comparing the source offsets for the new data and already delivered data and make some

Re: Kafka Connect ++ Kafka Streams

Replies from the dev list weren't getting to my original sender account. I could create a pull request but it has some patchy code in ConnectEmbedded and dependencies which shouldn't be there (connect-runtime) as it was mostly intended for imagining how the integrated connect and streams topology w

Re: Kafka Connect ++ Kafka Streams

I am thinking maybe we can even consider pulling the project as a whole into examples instead of adding the connector and streams implementation separately into Kafka Connect and Kafka Streams if Michal is interested to filing a PR: currently the examples folder only contains consumer / producer de

Re: Kafka Connect ++ Kafka Streams

Michal -- This is really cool. Mind submitting a pull request? Also, would you like your IRC connector to be featured on the Kafka Connector Hub ? On Fri, Mar 25, 2016 at 9:08 AM, Michal Hariš wrote: > So I had a go and hacked it up here: ConnectEmbedded.java > <

Re: Kafka Connect ++ Kafka Streams

So I had a go and hacked it up here: ConnectEmbedded.java And this is how the wikipedia demo looks with it: hello-kafka-streams

Re: Kafka Connect ++ Kafka Streams

On Thu, Mar 24, 2016 at 4:46 AM, Michal Hariš wrote: > Hello Kafka people! > > Great to see Kafka Streams coming along, the design validates (and in many > way supersedes) my own findings from working with various stream processing > systems/frameworks and eventually ending-up using just a small

Re: KAFKA Connect - Source Connector for Mainframe REST Services

Thanks. Yes. This is resolved.  Please also check the below mail i sent earlier.  Hi , I just found that the pom.xml of Kafka-Connect-jdbc is missing these entriesAlso , both the libraries Common-Config and Common-Utils are not found in the confluent maven repo. Please upload and update the pom.x

Re: KAFKA Connect - Source Connector for Mainframe REST Services

I think the relevant questions here were addressed in this Github issue: https://github.com/confluentinc/kafka-connect-jdbc/issues/30 -Ewen On Tue, Dec 15, 2015 at 12:51 PM, saravanan tirugnanum wrote: > Also please share some example of JDBC Source Connector running in > distributed mode and

Re: KAFKA Connect - Source Connector for Mainframe REST Services

Also please share some example of JDBC Source Connector running in distributed mode and assigning tasks across different workers. RegardsSaravanan On Tuesday, 15 December 2015 1:23 PM, saravanan tirugnanum wrote: Hi  I am working on designing and building a SourceConnector to run in a