Re: Kafka Connect on Kubernetes: Statefulset vs Deployment

mzi/strimzi-kafka-operator/issues/4676 > > Thanks > > -Original Message- > From: Vignesh > Sent: 16 June 2025 01:34 > To: users@kafka.apache.org > Subject: Re: Kafka Connect on Kubernetes: Statefulset vs Deployment > > [You don't often get email from

RE: Kafka Connect on Kubernetes: Statefulset vs Deployment

mziPodSet some time ago because of this issue. https://github.com/strimzi/strimzi-kafka-operator/pull/8090 https://github.com/strimzi/strimzi-kafka-operator/issues/4676 Thanks -Original Message- From: Vignesh Sent: 16 June 2025 01:34 To: users@kafka.apache.org Subject: Re: Ka

Re: Kafka Connect on Kubernetes: Statefulset vs Deployment

Kafka Connect is a stateless component by design. It relies on external Kafka topics to persist its state, including connector configurations, offsets, and status updates. In a distributed Kafka Connect cluster, this state is managed through the following configurable topics: - config.stora

Re: Kafka Connect on Kubernetes: Statefulset vs Deployment

I created the docker+kube stuff for our kafka-connect at my current job. I use standard deployment. kafka-connect doesn't care of hostname or IP. The sole trick is to inject the connector configuration at runtime (if you want). -- Raph On 14/06/2025 2:12 pm, Prateek Kohli wrote: > Hi All, > >

Re: Kafka Connect on Kubernetes: Statefulset vs Deployment

something wrong? From: Federico Valeri Sent: Saturday, June 14, 2025 7:33:29 PM To: users@kafka.apache.org Subject: Re: Kafka Connect on Kubernetes: Statefulset vs Deployment [You don't often get email from fedeval...@gmail.com. Learn why this is importa

Re: Kafka Connect on Kubernetes: Statefulset vs Deployment

Hi Prateek. In a Kafka Connect cluster, the advertised address represents the identity of the worker node. Connectors and tasks are scheduled to the individual worker nodes based on their identity. If you use a Kubernetes Deployment, when you roll the cluster, new Pods with new IPs will be be cre

Re: Kafka Connect

Hi, I think SMTs (KIP-66) could work for your case. https://kafka.apache.org/documentation.html#connect_transforms Regards, OSB On Fri, Nov 15, 2024, 03:03 Surbhi Mungre wrote: > Can Kafka Connect be used to read messages from one Kafka Cluster, apply > some basic transformation and write mes

Re: Kafka Connect

We have similar usecase and we use flink for transformation. Flink reads from kafka , does the transformation and writes back to kafka. Thanks, Prince > On Nov 14, 2024, at 8:22 PM, Neeraj Vaidya > wrote: > > I don’t think KStreams is a good option just by itself for inter site > replication

Re: Kafka Connect

I don’t think KStreams is a good option just by itself for inter site replication. How about using a replication technology like MM2 to first replicate to a topic in the destination cluster and then run KStreams client there in the destination cluster to consume, transform and then produce to y

Re: Kafka Connect Limits

Thank you for having a look at this. I agree that the only way to really gauge load is to look at lag. But the connector tasks should not crash and die because of load. I will raise this with SF. On Wed, Jul 3, 2024 at 7:14 PM Greg Harris wrote: > Hey Burton, > > Thanks for your question and bug

Re: Kafka Connect Limits

Hey Burton, Thanks for your question and bug report. The exception you included does not indicate that your connectors are overloaded. The primary way of diagnosing an overloaded connector is the consumer lag metric, and if you're seeing acceptable lag, that should indicate that your connectors a

Re: [Kafka Connect] Dead letter queues for source connectors?

Hey Greg, Thinking more, I do like the idea of a source-side equivalent of the ErrantRecordReporter interface! However, I also suspect we may have to reason more carefully about what users could do with this kind of information in a DLQ topic. Yes, it's an option to reset the connector (or a copy

Re: [Kafka Connect] Dead letter queues for source connectors?

Hey Chris, That's a cool idea! That can certainly be applied for failures other than poll(), and could be useful when combined with the Offsets modification API. Perhaps failures inside of poll() can be handled by an extra mechanism, similar to the ErrantRecordReporter, which allows reporting aff

Re: [Kafka Connect] Dead letter queues for source connectors?

Hi Greg, This was my understanding as well--if we can't turn a record into a byte array on the source side, it's difficult to know exactly what to write to a DLQ topic. One idea I've toyed with recently is that we could write the source partition and offset for the failed record (assuming, hopefu

Re: [Kafka Connect] Dead letter queues for source connectors?

Hi Yeikel, Thanks for your question. It certainly isn't clear from the original KIP-298, the attached discussion, or the follow-up KIP-610 as to why the situation is asymmetric. The reason as I understand it is: Source connectors are responsible for importing data to Kafka. If an error occurs dur

Re: Kafka Connect - Customize REST request headers

Thank you for the explanation, Chris. In case it helps, what I'm looking for is similar to KIP 577[1]. My specific example involves a hard-coded key/value pair that needs to be used for pod-to-pod as I can connect to any worker without that specific header, but workers cannot communicate amon

Re: Kafka Connect - Customize REST request headers

Hi Yeikel, Neat question! And thanks for the link to the RestClient code; very helpful. I don't believe there's a way to configure Kafka Connect to add these headers to forwarded requests right now. You may be able to do some kind of out-of-band proxy magic to intercept forwarded requests and ins

Re: Kafka connect Graceful stop of task failed

Hello Greg Thanks a *lot* for your help on this. Indeed the empty poll is not the issue for us. As mentioned, our setup is a poll every 24 hours. So the `stop()` being stuck due to the `poll()` is hitting us hard. I did a trace today on my dev environment, I can indeed see this waiting log entry

Re: Kafka connect Graceful stop of task failed

Hey Robson, Thanks for opening an issue on the JDBC repo, I think this is certainly relevant feedback for the connector developers. I commented on the issue with a potential regression that I saw, you can try downgrading your connector to see if the behavior improves. I also know that kafka-connec

Re: Kafka connect Graceful stop of task failed

How can I stop getting these updates ? On Mon, Aug 21, 2023 at 9:01 AM Robson Hermes wrote: > This email was sent from an external source so please treat with caution. > > No, it stops them also. > The problem is precisely what Greg described, now the stop signal comes > from the same thread. So

Re: Kafka connect Graceful stop of task failed

No, it stops them also. The problem is precisely what Greg described, now the stop signal comes from the same thread. So any source task which is running in a blocking way will not process the stop signal until the current poll finishes. So would need to patch source jdbc connector. On Mon, 21 Aug

Re: Kafka connect Graceful stop of task failed

I think when you delete connector it removes the task and workers continues to run. When you stop it actually stops the worker. Both different things. Point to be noted is Worker has connector. So connector should be removed before stopping the worker. Though I am not expert in this. On Mon, 21 A

Re: Kafka connect Graceful stop of task failed

Hello Sunil I'm not calling a stop, I'm straight deleting the connectors with the DELETE. Stopping the connector is done internally during deletion. Regards Robson On Mon, 21 Aug 2023 at 15:36, sunil chaudhari wrote: > You have to remove connectors first using delete api > and then stop the co

Re: Kafka connect Graceful stop of task failed

You have to remove connectors first using delete api and then stop the connector On Thu, 17 Aug 2023 at 2:51 AM, Robson Hermes wrote: > Hello > > I'm using kafka connect 7.4.0 to read data from Postgres views and write to > another Postgres tables. So using JDBC source and sink connectors. > All

Re: Kafka connect Graceful stop of task failed

Hello Greg (sorry about the duplicate e-mail, forgot to cc users mailing list) Thanks a lot for your detailed reply. I'm using JDBC Source connectors from kafka-connect-jdbc . Indeed the `poll()` implementation is blocked, so it only processes a

Re: Kafka connect Graceful stop of task failed

Hi Robson, Thank you for the detailed bug report. I believe the behavior that you're describing is caused by this flaw: https://issues.apache.org/jira/browse/KAFKA-15090 which is still under discussion. Since the above flaw was introduced in 3.0, source connectors need to return from poll() befor

RE: Kafka Connect Rest Extension Question

ge- From: mil...@votecgroup.com [mailto:mil...@votecgroup.com] Sent: 01 August 2023 11:56 To: users@kafka.apache.org Subject: RE: Kafka Connect Rest Extension Question Hi Team, Greetings, We actually reached out to you for Oracle/ IT / SAP / Infor / Microsoft "VOTEC IT SERVICE PARTNERSH

RE: Kafka Connect Rest Extension Question

Hi Team, Greetings, We actually reached out to you for Oracle/ IT / SAP / Infor / Microsoft "VOTEC IT SERVICE PARTNERSHIP" "IT SERVICE OUTSOURCING" " "PARTNER SERVICE SUBCONTRACTING" We have very attractive newly introduce reasonably price PARTNER IT SERVICE ODC SUBCONTRACTING MODEL in USA,

RE: Kafka Connect Rest Extension Question

inal Message- From: Greg Harris [mailto:greg.har...@aiven.io.INVALID] Sent: 31 July 2023 23:42 To: users@kafka.apache.org Subject: Re: Kafka Connect Rest Extension Question Hello Yang Hyung Wook, In your post I do not see anything obviously wrong, so you may need to do some more debugging. 1. Are

Re: Kafka Connect Rest Extension Question

Hello Yang Hyung Wook, In your post I do not see anything obviously wrong, so you may need to do some more debugging. 1. Are you using the same jar for both the classpath and plugin.path tests? If not, do they both contain the service loader manifest file? You can test this with https://docs.orac

Re: Kafka Connect exactly-once semantic and very large transactions

Hi Chris, thanks for your response! Yes, we are looking also on other means how to enable exactly-once semantics for existing data (e.g. using incremental snapshot which snapshots the data incrementally and in smaller chunks), but first we would like to fully understand all the implications and

Re: Kafka Connect exactly-once semantic and very large transactions

Hi Vojta, >From my limited understanding of the Debezium snapshot process, I believe that you're correct that producing the entire snapshot in a transaction is the way to provide exactly-once semantics during that phase. If there's a way to recover in-progress snapshots and skip over already-produ

Re: Kafka connect process listens to an unknown port

Hey Jorge, I looked into it, and can reproduce the second LISTEN port in a vanilla Kafka Connect cluster without any connectors running. Using jstack, I see that there are two threads that appear to be waiting in the corresponding accept methods: "RMI TCP Accept-0" #15 daemon prio=5 os_prio=31 c

Re: Kafka Connect Startup Hook

In Strimzi, we use a Java agent to register the tracer ( https://github.com/strimzi/strimzi-kafka-operator/tree/main/tracing-agent/ if you wanna check the source code). Jakub On Mon, Mar 20, 2023 at 9:18 AM Jan Baudisch (extern) < jan.baudisch.ext...@bdess.com> wrote: > Hello, > > can someone pl

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Ah, it definitely seems like KIP-710 will address the issue we've been bitten by most.We'll eagerly await the kafka-3.5.0 release and then see if enabling 'dedicated.mode.enable.internal.rest' is possible with Strimzi. Thanks for the help and patience! :-)

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Frank, > I don't think forcing the API users to introduce the nonce is desirable. I agree. That is why the nonce is a workaround, and not a proper solution. It's something to alleviate the symptoms in the short-term until a bugfix & upgrade can fix it. > Have you had any ideas on how this can be

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

So we've just hit this issue again just with the MM2 connector and trying to add a new mirrored topic.We're running MirrorMaker 2 in Strimzi. i.e. "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector"We have 6 worker nodes.We changed the config to add a new mirror topic. i.e

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

I don't think forcing the API users to introduce the nonce is desirable.For us, it would mean reaching out to the Strimzi project to try to get that implemented on their side, which I would imagine would be a proposal which would meet some resistance. Have you had any ideas on how this can be im

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Frank, The configs are being compared after ConfigProviders have been resolved. This is happening both as a Connector config (by ClusterConfigState::connectorConfig) and as task configs (by ClusterConfigState::taskConfig). This means that two configurations that have different direct contents (the

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

I'm still having trouble understanding how the configs could match in the code you highlighted when we change connector and/or task config values when no keys are being pruned by the connector implementations in question.Would capturing a new generation value within the config itself on every s

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Frank, > I'm operating on the assumption that the connectors in question get stuck in an inconsistent state > Another thought... if an API exists to list all connectors in such a state, then at least some monitoring/alerting could be put in place, right? There is two different inconsistencies rel

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Another thought... if an API exists to list all connectors in such a state, then at least some monitoring/alerting could be put in place, right?

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

So I've been looking into the codebase to familiarize myself with it.I'm operating on the assumption that the connectors in question get stuck in an inconsistent state which causes them to prune the new task configs from those which are "broadcast" to the workers.I see on KafkaConfigBackingSto

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Frank, I don't think that the fix needs to necessarily follow the #12450 PR, we can choose to start from scratch now that we know more about the issue. If that PR is useful as a starting point, we can also include it, that is up to you. Greg On Mon, Feb 6, 2023 at 10:21 AM Frank Grimes wrote:

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Hi Greg, I actually just found the following comment on this PR for  https://issues.apache.org/jira/browse/KAFKA-13809:  https://github.com/apache/kafka/pull/12450 > we get the same behavior (KAFKA-9228 notwithstanding) by passing the original >properties through to tasks transparently It seems

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Frank, I think you're right that the KAFKA-9228 ticket doesn't capture every possible reconfiguration that might result in a dropped restart. The ticket calls out the FileStream connectors, which generate their configurations by dropping unknown config values, which is relatively uncommon. This me

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Hi Greg, The "long-term inconsistency" we have observed is not with no tasks at all, but instead with all the previously running tasks remaining in a running state but with a previous config. If I'm understanding the original bug report correctly, the scope of the problem was thought to only af

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Frank, I realized I didn't respond to the title directly, sorry about that. The reason that `ClusterConfigState::inconsistentConnectors` is not used, is that the effect of an inconsistent connector is applied via `ClusterConfigState::tasks`. If a connector is inconsistent, then the tasks method wi

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

Frank, The inconsistentConnectors method is related to an extremely specific inconsistency that can happen when a worker writes some task configurations, and then disconnects without writing a following "commit tasks record" to the config topic. This is a hold-over from the early days of connect f

Re: Kafka Connect - offset.storage.topic reuse across clusters

Connectors overwriting each other's offsets is the primary concern. If you have a guarantee that there will only ever be one connector with a given name running at once on any of the Connect clusters that use the same offsets topic, and you want offsets to be shared for all source connectors on any

Re: Kafka Connect - offset.storage.topic reuse across clusters

Hi Robin, I'm interested in a use case in which I need to be able to have a connect cluster fail, and then bring up a new cluster with the same offset topics and connectors. By new cluster I mean a cluster with a new `group.id`. I am aware I could just use the same group id as before but I would l

Re: Kafka Connect - offset.storage.topic reuse across clusters

Hi Jordan, Is there a good reason for wanting to do this? I can think of multiple reasons why you shouldn't do this even if technically it works in some cases. Or it's just curiosity as to whether you can/should? thanks, Robin. -- Robin Moffatt | Principal Developer Advocate | ro...@confluent

Re: Kafka connect JMS sink connector performance

Hi Tilak, > While using the connector we observed that, for each call to the `put` method in our `SinkTask` implementation, there is some delay between calls which is in the range of [200ms to 1sec]. Yes, that makes sense! As you should have known, the sinkTask is also a Kafka consumer, which try

Re: Kafka connect JMS sink connector performance

Hello All, Can anyone please provide any feedback on the issue. To add further information, we are using kafka version 2.7.1 Thanks & Regards, Tilak Sasmal On Tue, Dec 14, 2021 at 11:46 AM Tilak Sasmal wrote: > Hello All, > > We are currently using Kafka Connect and a JMS sink connector (custo

Re: Kafka Connect Dist. Worker Does not join group

Hello, I wanted to follow up to see if anyone has ever experienced such behavior with Kafka Connect? Any guidance would be greatly appreciated! Thanks, Upesh Upesh Desai | Senior Software Developer | ude...@itrsgroup.com www.itrsgroup.com From: Upesh Desai Date: Thursday, April 22, 2021 at 4:1

Re: Kafka Connect: producer idempotence after increasing producer.max.in.flight.requests.per.connection

Hi Yong Gang, Idempotence is not guaranteed if you increase max in flight requests. Because if 5 requests are dispatched, but 2 requests fail and are retried, the messages in those 2 requests will arrive after the messages in the other three requests, so ordering will be changed. Assuming everyth

Re: Kafka Connect Distributed Mode Issues

Yeah, looks like it's an issue with the plugin. I don't have any experience of it, sorry. On Tue, 6 Apr. 2021, 12:32 am Himanshu Shukla, wrote: > bootstrap.servers=b-1:9092,b-2:9092 > group.id=connect-cluster > key.converter=org.apache.kafka.connect.json.JsonConverter > value.converter=org.apach

Re: Kafka Connect Distributed Mode Issues

bootstrap.servers=b-1:9092,b-2:9092 group.id=connect-cluster key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=true offset.storage.topic=connect-offsets-2 offset.stor

Re: Kafka Connect Distributed Mode Issues

Hi Himanshu, Have you adjusted your consumer properties as the error message suggested? Alternatively reduce your your consumer.max.poll.records in the worker config. Basically, the sink you're using is spending too much time processing in the poll loop, so either tweak the properties as mention

Re: Kafka Connect Distributed Mode Issues

Did anyone face it before? The connector URL is giving 500 request time out. On Thu, Apr 1, 2021 at 9:55 AM Himanshu Shukla wrote: > Hi, > I am using kafka-connect-file-pulse connector and scanning around 20K > files. After the scan step, the whole connect cluster is becoming > unresponsive. I c

Re: Kafka Connect Distributed Mode Doubt

Thanks for the clarifications, Robin. On Tue, Mar 23, 2021 at 2:51 PM Robin Moffatt wrote: > 1. Kafka Connect standalone workers have their connectors configured based > on properties file(s) passed on the command line at startup. You cannot use > REST to add or remove them > 2. Correct, Standal

Re: Kafka Connect Distributed Mode Doubt

1. Kafka Connect standalone workers have their connectors configured based on properties file(s) passed on the command line at startup. You cannot use REST to add or remove them 2. Correct, Standalone workers are isolated instances that cannot share load with other workers 3. Correct, Distributed w

Re: Kafka connect replication using MirrorMaker 2.0

Hi Daniel, it is probably hard to figure out how to sync topics across two kafka connect cluster. In general, if implementing a solution requires strong technical pre-requisite (e.g. kafka connect will be aware of each other offset), It may be better to go with simpler solution first, for examp

Re: Kafka connect replication using MirrorMaker 2.0

Hello and thank you for the reply! My problem is not with consumption of messages, because as you said, MirrorMaker2 knows how to deal with the consumer offsets. Rather my problem is with source connectors and the topic connect-offsets. Because Kafka connect manages where it stopped reading f

Re: Kafka connect replication using MirrorMaker 2.0

Hi Daniel, MirrorMaker2 creates its own "offsets" topic to track the process of consumption. just my 2 cents - If you already have two Kafka connect clusters in two different sites, it sounds practical to: (1) use "cluster" mode, instead of "dedicated" mode of MirrorMaker2 (2) add one "MirrorMak

Re: Kafka Connect Connector Tasks Uneven Division

Hi Robin Request you to please reply. Regards and Thanks Deepak Raghav On Wed, Jun 10, 2020 at 11:57 AM Deepak Raghav wrote: > Hi Robin > > Can you please reply. > > I just want to add one more thing, that yesterday I tried with > connect.protocal=eager. Task distribution was balanced after

Re: Kafka Connect Connector Tasks Uneven Division

Hi Robin Can you please reply. I just want to add one more thing, that yesterday I tried with connect.protocal=eager. Task distribution was balanced after that. Regards and Thanks Deepak Raghav On Tue, Jun 9, 2020 at 2:37 PM Deepak Raghav wrote: > Hi Robin > > Thanks for your reply and acc

Re: Kafka Connect Connector Tasks Uneven Division

Hi Robin Thanks for your reply and accept my apology for the delayed response. As you suggested that we should have a separate worker cluster based on workload pattern. But as you said, task allocation is nondeterministic, so same things can happen in the new cluster. Please let me know if my un

Re: Kafka Connect Connector Tasks Uneven Division

The KIP for the current rebalancing protocol is probably a good reference: https://cwiki.apache.org/confluence/display/KAFKA/KIP-415:+Incremental+Cooperative+Rebalancing+in+Kafka+Connect -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 26 May 2020 at 14:25, D

Re: Kafka Connect Connector Tasks Uneven Division

Hi Robin Thanks for the clarification. As you suggested, that task allocation between the workers is nondeterministic. I have shared the same information within in my team but there are some other parties, with whom I need to share this information as explanation for the issue raised by them and

Re: Kafka Connect Connector Tasks Uneven Division

I don't think you're right to assert that this is "expected behaviour": > the tasks are divided in below pattern when they are first time registered Kafka Connect task allocation is non-determanistic. I'm still not clear if you're solving for a theoretical problem or an actual one. If this is a

Re: Kafka Connect Connector Tasks Uneven Division

Hi Robin I had gone though the link you provided, It is not helpful in my case. Apart from this, *I am not getting why the tasks are divided in *below pattern* when they are *first time registered*, which is expected behavior. I*s there any parameter which we can pass in worker property file which

Re: Kafka Connect Connector Tasks Uneven Division

Thanks for the clarification. If this is an actual problem that you're encountering and need a solution to then since the task allocation is not deterministic it sounds like you need to deploy separate worker clusters based on the workload patterns that you are seeing and machine resources availabl

Re: Kafka Connect Connector Tasks Uneven Division

Hi Robin Replying to your query i.e One thing I'd ask at this point is though if it makes any difference where the tasks execute? It actually makes difference to us, we have 16 connectors and as I stated tasks division earlier, first 8 connector' task are assigned to first worker process and ano

Re: Kafka Connect Connector Tasks Uneven Division

OK, I understand better now. You can read more about the guts of the rebalancing protocol that Kafka Connect uses as of Apache Kafka 2.3 an onwards here: https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka/ One thing I'd ask at this point is though if it makes any differenc

Re: Kafka Connect Connector Tasks Uneven Division

Hi Robin Thanks for your reply. We are having two worker on different IP. The example which I gave you it was just a example. We are using kafka version 2.3.1. Let me tell you again with a simple example. Suppose, we have two EC2 node, N1 and N2 having worker process W1 and W2 running in distri

Re: Kafka Connect Connector Tasks Uneven Division

So you're running two workers on the same machine (10.0.0.4), is that correct? Normally you'd run one worker per machine unless there was a particular reason otherwise. What version of Apache Kafka are you using? I'm not clear from your question if the distribution of tasks is presenting a problem

Re: Kafka Connect Connector Tasks Uneven Division

Hi Please, can anybody help me with this? Regards and Thanks Deepak Raghav On Tue, May 19, 2020 at 1:37 PM Deepak Raghav wrote: > Hi Team > > We have two worker node in a cluster and 2 connector with having 10 tasks > each. > > Now, suppose if we have two kafka connect process W1(Port 8080)

Re: Kafka Connect SMT to insert key into message

Hi, I think you're right. There's a ValueToKey transformation, but not a KeyToValue. I think that would be a better fit than adding to InsertField because you always use the concrete transformations InsertField$Key and InsertField$Value, and they have the same configuration. Probably a fairly simp

Re: Kafka Connect SMT to insert key into message

So looking at the code of InsertField, it looks like there isn't an obvious way, unless there's some way to chaining SMTs to achieve it. Question then is, is it worth adding it to the InsertField SMT? The change looks reasonably straightforward, and I'm happy to do a PR if it fits with the aims of

Re: Kafka Connect ignores the bootstrap.servers parameter

Hi Goran, Thanks for your reply. We've tried this and got interesting results. First, we found out that there is a port forwarding between the client and the server. So we put the local IP (192.168.x.x) into the 'listeners' parameter, and the external IP (98.1.96.147) into the 'advertised.listene

Re: Kafka Connect ignores the bootstrap.servers parameter

Client connects to IP address defined by bootstrap.servers and gets metada that contain IP address where kafka is available. That is configured in server.properties: # Hostname and port the broker will advertise to producers and consumers. If not set, # it uses the value for "listeners" if conf

Re: Kafka Connect ignores the bootstrap.servers parameter

Hi Suresh, Thanks for your quick response. All the configuration files, except for the connect-standalone.properties are unmodified and use all the default parameters. Attaching them anyway just in case. This setup works perfectly on my laptop with VirtualBox (ZooKeeper and the broker on VM, the

Re: Kafka connect not loading classes from uber jar

There has been similar issue raised at confluent too: https://github.com/confluentinc/cp-docker-images/issues/638 I am however not using docker, but it looks like same issue where kafka connect startup throws lot of these warning messages before starting up. Any idea how to avoid all these warnin

Re: Kafka connect issue with mysql - Debezium CDC

Hi, I was able to resolve this issue by changing the name of server to docker container name. [image: image.png] Thanks, Asmath On Wed, Dec 25, 2019 at 8:30 PM KhajaAsmath Mohammed < mdkhajaasm...@gmail.com> wrote: > Hi, > > I am trying to do POC for kafka CDC with database and ingest it into

Re: kafka connect incoming port

Kafka Connect workers listen on the port defined in *rest.port* in the worker configuration - typically 8083. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Mon, 25 Nov 2019 at 18:41, Johnson, Paul wrote: > Hello, > > I am working on setting up a Kafka Connect

Re: kafka connect 409 with connect-distributed

Hi Kafka users, I've been trying to investigate this a bit further, in the documentation for the Connect REST API, found this paragraph: "*Note that if you try to modify, update or delete a resource under > connector which may require the request to be forwarded to the leader, > Connect will retu

Re: Kafka Connect issues when running in Docker

Do you get any output from *docker logs*? And does it work if you don't use authentication? How about if you try one of the dockerised Kafka Connect examples here? https://github.com/confluentinc/demo-scene/tree/master/kafka-connect-zero-to-hero -- Robin Moffatt | Senior Developer Advocate | ro

Re: Kafka Connect JDBC Sink - SQLServerException: Column in table is of a type that is invalid for use as a key column in an index

Just checked the constraint, and the maximum is 900 for varchar before SQL Server 2016 On 2019/06/28 14:55:01, sendoh wrote: > > > On 2019/06/28 14:50:37, sendoh wrote: > > I encounter the same issue as > > https://github.com/confluentinc/kafka-connect-jdbc/issues/379 > > > > I think I

Re: Kafka Connect JDBC Sink - SQLServerException: Column in table is of a type that is invalid for use as a key column in an index

On 2019/06/28 14:50:37, sendoh wrote: > I encounter the same issue as > https://github.com/confluentinc/kafka-connect-jdbc/issues/379 > > I think I could contribute to implement a custom `buildCreateTableStatement` > in SqlServerDatabaseDialect that checks if primary key is a string, set i

Re: Kafka Connect - HDFS or FileStream

Appreciate if somebody has experience on the above and respond to the same. Logically it should work if we have a property set hdfs.url = file:///home/user/data On Wed, May 15, 2019 at 5:06 AM Vinay Jain wrote: > Redirecting to a file will not work, we would not be able to create a > different

Re: Kafka Connect - HDFS or FileStream

Redirecting to a file will not work, we would not be able to create a different filenames after some time or filesize. HDFS connect sink already has those options , also i could use some minor transformations while using the connect On Tue, May 14, 2019 at 7:28 AM Hans Jespersen wrote: > Can you

Re: Kafka Connect - HDFS or FileStream

Can you just use kafka-console-consumer and just redirect the output into a file? -hans On Mon, May 13, 2019 at 1:55 PM Vinay Jain wrote: > Hi > > The data needs to be transferred to some other system in other network, and > due to some security reasons, the other systems cannot be exposed . S

Re: Kafka Connect - HDFS or FileStream

Hi The data needs to be transferred to some other system in other network, and due to some security reasons, the other systems cannot be exposed . So the available mechanism is file based integration. Is there a production ready Kafka connect adapter which can create files in local directory. Reg

Re: Kafka Connect - HDFS or FileStream

Can you explain more about why you're writing a file with the data? Presumably, this is for another application to consume; could it not take the data from Kafka directly, whether with a native client or over the REST proxy? Oftentimes local files are unnecessary 'duck tape' for integration that ca

Re: Kafka - Connect for logs processing

Hi Hans, Thanks for quick response. I am gonna look into it. Thanks Pulkit On Fri, Mar 15, 2019 at 11:39 AM Hans Jespersen wrote: > Take a look at kafka-connect-spooldir and see if it meets your needs. > > https://www.confluent.io/connector/kafka-connect-spooldir/ > > This connector can monitor

Re: Kafka - Connect for logs processing

Take a look at kafka-connect-spooldir and see if it meets your needs. https://www.confluent.io/connector/kafka-connect-spooldir/ This connector can monitor a directory and pick up any new files that are created. Great for picking up batch files, parsing them, and publishing each line as if it w

RE: Kafka connect FieldPartitioner with scheduled rotation

it is a defect or a genuine requirement. Thanks, -Manu From: Pere Urbón Bayes Sent: Monday, February 04, 2019 11:41 PM To: Manu Jacob Cc: users@kafka.apache.org Subject: Re: Kafka connect FieldPartitioner with scheduled rotation EXTERNAL Hi Manu, if we take master of the connector, I

Re: Kafka connect FieldPartitioner with scheduled rotation

utureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > [2019-02-03 17:59:01,262]

RE: Kafka connect FieldPartitioner with scheduled rotation

.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2019-02-03 17:59:01,262] ERROR Task is being killed and will not recover until manually restarted

  1   2   3   >