Hello Artem,
Thanks for the KIP.
I have the same question as Roger on concurrent writes, and an additional
one on consumer behavior. Typically, transactions will timeout if not
committed within some time interval. With the proposed changes in this KIP,
consumers cannot consume past the ongoing tr
Many congrats Chris!
On Tue, Jul 26, 2022 at 4:02 AM Bruno Cadonna wrote:
> Congratulations Chris! Well deserved!
>
> Best,
> Bruno
>
> On 26.07.22 05:24, Kumud Kumar Srivatsava Tirupati wrote:
> > Congratulations Chris!
> >
> > On Tue, 26 Jul, 2022, 7:11 AM deng ziming,
> wrote:
> >
> >> Congr
hey folks,
Thanks a lot for the KIP and the discussions. Here are a couple of thoughts
(apologies if I missed some point in this thread).
1. it seems like we are changing the behavior of /connector-plugins by now
returning the list of transforms, converters as well? This would break
backward comp
+1 (non-binding). Thanks for the KIP, Knowles! and appreciate the
follow-ups!
On Thu, Nov 11, 2021 at 2:55 PM John Roesler wrote:
> Thanks, Knowles!
>
> I'm +1 (binding)
>
> -John
>
> On Wed, 2021-11-10 at 12:42 -0500, Christopher Shannon
> wrote:
> > +1 (non-binding). This looks good to me and
One more nit: the RetryWithToleranceOperator class is not a public
interface. So we do not have to call the changes in them out in the Public
Interfaces section.
On Tue, Nov 16, 2021 at 10:42 AM Arjun Satish
wrote:
> Chris' point about upgrades is valid. An existing configuration
> > > On Tue, Nov 2, 2021 at 10:00 AM Knowles Atchison Jr <
> > katchiso...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Third time's the charm.
> > > >
> > > > I've added a getter for the RetryWithTo
nds of the connector developer. I suppose that is for the best, in a
> vacuum only the worker should have a say in how it handles message
> production.
>
> Additional thoughts and feedback are welcome.
>
> Knowles
>
> On Thu, Oct 28, 2021 at 10:54 AM Arjun Satish
> wrote:
ritten, but
> looking at SourceTask more closely, in commitRecord(SourceRecord,
> RecordMetadata), the RecordMetadata is set to null in the event of a
> filtered transformation so the framework is already doing this in a certain
> regard.
>
> Knowles
>
> On Thu, Oct 28, 202
need some kind of callback from inside the connector with the
> Source Record to successfully ack back to my source system.
>
> I have updated the KIP regarding the callback being executed in a different
> thread than poll().
>
> Knowles
>
> On Thu, Oct 28, 2021 at 2:02 AM Ar
Hi Knowles,
Thanks for the KIP!
Could you please call out some use-cases on what the source connectors
would do when they hit such exceptions? I'm wondering if we would need to
do anything other than skipping such records, writing some log messages,
and/or writing some error context to a DLQ?
On
Arjun Satish created KAFKA-10387:
Summary: Cannot include SMT configs with source connector that
include topic.creation.* properties
Key: KAFKA-10387
URL: https://issues.apache.org/jira/browse/KAFKA-10387
Arjun Satish created KAFKA-10340:
Summary: Source connectors should report error when trying to
producer records to non-existent topics instead of hanging forever
Key: KAFKA-10340
URL: https://issues.apache.org
;firehose", the task needs to pause all partitions?
On Tue, May 19, 2020 at 9:26 AM Arjun Satish wrote:
> Can we get a couple of examples that shows utility of waiting on the
> Future<>? Also, in preCommit() we would report back on the incomplete
> offsets. So that feedbac
wrote:
> Thanks, Aakash, for updating the KIP.
>
> On Tue, May 19, 2020 at 2:18 AM Arjun Satish
> wrote:
>
> > Hi Randall,
> >
> > Thanks for the explanation! Excellent point about guaranteeing offsets in
> > the async case.
> >
> > If we can g
g non-errant records
> to
> > > the
> > > > > > sink. However, this could also be addressed by allowing for batch
> > > > > reporting
> > > > > > of errant records instead of accepting a single record at a time;
> > the
&g
Thanks for all the feedback, folks.
re: having a callback as a parameter, I agree that at this point, it might
not add much value to the proposal.
re: synchronous vs asynchronous, is the motivation performance/higher
throughput? Taking a step back, calling report(..) in the new interface
does a c
what
> behavior it should expect? Is there any case when the reporter will be null
> or be a no-op, and if so what should the sink task do? Should it simply
> wrap and throw a ConnectException? And if there is a reporter, won't
> Connect treat this sink record as "processed"
#x27;ve agreed upon a number of other changes, but I don't see
> any
> > > of
> > > > > those changes on the KIP. Aakash, can you please update the KIP
> > quickly
> > > > so
> > > > > we can make sure the other parts are the KIP are acceptable
; > make the upgrade path easy to follow. Maybe you'd like to add a note to
> > > your compatibility section in this KIP as well.
> > >
> > > Regards,
> > > Konstantine
> > >
> > > On Sat, May 16, 2020 at 10:13 AM Aakash Shah
> wrote:
>
ike to come to a consensus
> soon
> > due to the AK 2.6 deadlines; I will then shortly update the KIP and
> start a
> > vote.
> >
> > Thanks,
> > Aakash
> >
> > On Fri, May 15, 2020 at 2:24 PM Randall Hauch wrote:
> >
> >> On Fr
Couple of thoughts:
1. If we add new parameters to put(..), and new connectors implement only
this method, it makes them backward incompatible with older workers. I
think newer connectors may only choose to only implement the latest method,
and we are passing the compatibility problems back to the
gt; Konstantine
> > > >
> > > >
> > > > On Wed, Aug 14, 2019 at 12:13 AM Cyrus Vafadari >
> > > > wrote:
> > > >
> > > >> I am excited to see this implemented +1 nonbinding
> > > >>
> > >
Thanks for the KIP, Almog.
+1 (non-binding)
On Mon, Sep 16, 2019 at 4:45 PM Randall Hauch wrote:
> Thanks for the nice improvement, Almog!
>
> +1 (binding)
>
> Randall
>
> On Thu, Aug 15, 2019 at 11:59 AM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Thanks Almog!
> > Nicely
therwise this looks good!
>
> Best regards,
>
> Randall
>
> On Mon, Sep 16, 2019 at 4:15 AM Arjun Satish
> wrote:
>
> > Good catch, Randall. Yes, we should be able to set the level of any
> logger
> > given its name. If this is an ancestor, then the levels of all
Sep 12, 2019 at 9:58 AM Gwen Shapira wrote:
> >
> > > The new REST API for logger management looks great to me.
> > >
> > >
> > > On Thu, Sep 12, 2019 at 8:36 AM Arjun Satish
> > > wrote:
> > > >
> > > > Bumping this
Bumping this thread.
If there are no further comments, please add your votes here:
https://www.mail-archive.com/dev@kafka.apache.org/msg100313.html
Thanks in advance,
Arjun
On Fri, Sep 6, 2019 at 4:22 PM Arjun Satish wrote:
> Thanks a lot, Jason! Answers inline. I'll also modify th
ot be persistent and only apply to the worker that received
the request.
> Thanks,
> Jason
>
> On Fri, Aug 30, 2019 at 1:25 AM Arjun Satish
> wrote:
>
> > OK. I didn't realize the plan was to deprecate and remove the JMX
> endpoint.
> > KIP-412 says brokers w
s
> of
> > an admin endpoint now so that we're not left with odd compatibility
> baggage
> > in the future.
>
> Hi Jason,
>
> I agree... I think Connect needs a REST admin API. There will probably be
> a lot of other stuff that we'll want to add to it.
>
>
JMX-based API? Is there literally
> > anyone in the world that wants to use JMX if they don't have to? I
> thought
> > one of the major motivations of KIP-412 was how much of a pain JMX is.
> >
> > Thanks,
> > Jason
> >
> > On Mon, Aug 19, 2019 at 5:28
he restrictions around log4j is that this is
> information is significant and IMO needs to be included in the KIP.
>
> Speaking of its relevance to KIP-412, I think a reference would be nice
> too.
>
> Konstantine
>
>
>
> On Thu, Aug 15, 2019 at 4:00 PM Arjun Satish
>
> On Tue, Aug 6, 2019 at 9:34 AM Cyrus Vafadari
> wrote:
> >
> >> This looks like a useful feature, the strategy makes sense, and the KIP
> is
> >> thorough and nicely written. Thanks!
> >>
> >> Cyrus
> >>
> >> On Thu, Aug 1, 2019, 12:40 PM C
Hey everyone,
I'd like to start a vote for KIP-495 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect).
This change will make Connect easier to debug in production environment.
Based on the discussion, I updated the KIP to reflect how Connect w
one with
> > care - I will add that documentation to the serialization config.
> >
> > Note that there would not be an issue on the _serialization_ side of
> > things as Jackson respects BigDecimal.
> >
> > Almog
> >
> > On Tue, Aug 6, 2019 at 11:23 PM
hey Almog, nice work! couple of thoughts (hope I'm not late since you
started the voting thread already):
can you please add some examples to show the changes that you are
proposing. makes me think that for a given decimal number, we will have two
encodings: “asHex” and “asNumber”.
should we call
will use that new utility. Is the
> "Example Usage" section (which involves invoking the utility with a
> namespace of "kafka.connect") actually meant to be part of the proposed
> changes to public interface?
>
> Cheers,
>
> Chris
>
> On Mon, Jul 22, 20
Hi everyone.
I'd like to propose the following KIP to implement changing log levels on
the fly in Connect workers:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect
Would like to hear your thoughts on this.
Thanks very much,
Arjun
.
Thanks very much and apologies for the confusion!
Best,
On Mon, May 6, 2019 at 10:33 AM Paul Davidson
wrote:
> Thanks Arjun. I've updated the KIP using your suggestion - just a few
> slight changes.
>
> On Fri, May 3, 2019 at 4:48 PM Arjun Satish
> wrote:
>
> > Ma
>
> Thanks,
>
> Paul
>
> On Thu, May 2, 2019 at 5:36 PM Arjun Satish
> wrote:
>
> > Paul,
> >
> > You might want to make a note on the KIP regarding the impact on quotas.
> >
> > Thanks,
> >
> > On Thu, May 2, 2019 at 9:48 AM Paul D
yanne, Arjun, Magesh)
>
> Regards,
>
> Paul
>
> On Wed, May 1, 2019 at 10:07 PM Arjun Satish
> wrote:
>
> > Good point, Gwen. We always set a non empty value for client id:
> >
> >
> https://github.com/apache/kafka/blob/2.2.0/clients/src/main/java/
e very difficult to enforce?
> So setting the client name will only change something if there's already a
> quota for that client?
>
> On the other hand, I fully support switching to "easy-to-wildcard" template
> for the client id.
>
> On Wed, May 1, 2019 at 8:50
I just realized that setting the client.id on the will now trigger any
quota restrictions (
https://kafka.apache.org/documentation/#design_quotasconfig) on the broker.
It seems like this PR will enforce quota policies that will either require
admins to set limits for each task (since the chosen for
Thanks, Paul! This is very useful.
+1 (non-binding)
Best,
Arjun
On Fri, Apr 12, 2019 at 4:13 PM Ryanne Dolan wrote:
> +1 (non binding)
>
> Thanks
> Ryanne
>
> On Fri, Apr 12, 2019, 11:11 AM Paul Davidson
> wrote:
>
> > Just a reminder that KIP-411 is open for voting. No votes received yet!
>
[
https://issues.apache.org/jira/browse/KAFKA-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish resolved KAFKA-7877.
-
Resolution: Fixed
Updated KIP.
> Connect DLQ not used in SinkTask
[
https://issues.apache.org/jira/browse/KAFKA-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish resolved KAFKA-7999.
-
Resolution: Fixed
PR: https://github.com/apache/kafka/pull/6326
> Flaky T
Arjun Satish created KAFKA-7909:
---
Summary: Coordinator changes cause Connect integration test to fail
Key: KAFKA-7909
URL: https://issues.apache.org/jira/browse/KAFKA-7909
Project: Kafka
Issue
Arjun Satish created KAFKA-7772:
---
Summary: Dynamically adjust log level in Connect workers
Key: KAFKA-7772
URL: https://issues.apache.org/jira/browse/KAFKA-7772
Project: Kafka
Issue Type
Arjun Satish created KAFKA-7503:
---
Summary: Integration Tests for Connect
Key: KAFKA-7503
URL: https://issues.apache.org/jira/browse/KAFKA-7503
Project: Kafka
Issue Type: Task
Arjun Satish created KAFKA-7228:
---
Summary: DeadLetterQueue throws a NullPointerException
Key: KAFKA-7228
URL: https://issues.apache.org/jira/browse/KAFKA-7228
Project: Kafka
Issue Type: Task
Arjun Satish created KAFKA-7003:
---
Summary: Add headers with error context in messages written to the
Connect DeadLetterQueue topic
Key: KAFKA-7003
URL: https://issues.apache.org/jira/browse/KAFKA-7003
Arjun Satish created KAFKA-7002:
---
Summary: Allow replication factor to be set via a configuration
property for the Connect DLQ topic
Key: KAFKA-7002
URL: https://issues.apache.org/jira/browse/KAFKA-7002
Arjun Satish created KAFKA-7001:
---
Summary: Rename `errors.allowed.max` in Connect to
`errors.tolerance`
Key: KAFKA-7001
URL: https://issues.apache.org/jira/browse/KAFKA-7001
Project: Kafka
Arjun Satish created KAFKA-6981:
---
Summary: Missing Connector Config
(errors.deadletterqueue.topic.name) kills Connect Clusters
Key: KAFKA-6981
URL: https://issues.apache.org/jira/browse/KAFKA-6981
KIP in the section here, and let me know
what you think:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
298%3A+Error+Handling+in+Connect#KIP-298:ErrorHandlinginConnect-
DeadLetterQueue(forSinkConnectorsonly)
Thanks very much,
On Tue, May 22, 2018 at 2:37 PM, Arjun Satish
wrote:
>
> +1. Thanks for the KIP!
>
> On Mon, May 21, 2018 at 1:34 PM, Arjun Satish
> wrote:
>
> > All,
> >
> > Thanks so much for your feedback on this KIP. I've made some small
> > modifications today. I'll wait till midnight today (PDT) to close this
section
to reflect this:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%3A+Error+Handling+in+Connect#KIP-298:ErrorHandlinginConnect-ProposedChanges
Thanks,
On Mon, May 21, 2018 at 3:12 PM, Arjun Satish
wrote:
> Hey Jason,
>
> This KIP does take serialization errors to be retri
want the option to skip over processing
> failures, but retry indefinitely if the downstream system is unavailable.
> Is that use case supported?
>
> Thanks,
> Jason
>
>
>
> On Mon, May 21, 2018 at 12:39 PM, Arjun Satish
> wrote:
>
> > Thanks a lot, Ewen! I
> >> Guozhang
> > >>
> > >> On Fri, May 18, 2018 at 3:36 PM, Gwen Shapira
> > wrote:
> > >>
> > >>> +1
> > >>>
> > >>> Thank you! Error handling in Connect will be a huge improvement.
> > >>>
OK. Let's simplify tolerance to simply have NONE or ALL values. For
extensions, we can open a KIP and implement in later versions.
Thanks a lot!
On Mon, May 21, 2018 at 1:18 PM, Ewen Cheslack-Postava
wrote:
> On Mon, May 21, 2018 at 12:39 PM Arjun Satish
> wrote:
>
> >
r to get it past one bad message, then reverting back
> to -1 or 0.
>
> -Ewen
>
> On Mon, May 21, 2018 at 11:01 AM Arjun Satish
> wrote:
>
> > Hey Jason,
> >
> > Thanks for your comments. Please find answers inline:
> >
> > On Mon, May 21, 2018 at
Hey Jason,
Thanks for your comments. Please find answers inline:
On Mon, May 21, 2018 at 9:52 AM, Jason Gustafson wrote:
> Hi Arjun,
>
> Thanks for the KIP. Just a few comments/questions:
>
> 1. The proposal allows users to configure the number of retries. I usually
> find it easier as a user t
ht direction.
> >
> > No further comments from my side.
> >
> > Thanks Arjun!
> >
> > - Konstantine
> >
> > On Fri, May 18, 2018 at 1:07 AM, Arjun Satish
> > wrote:
> >
> > > Ewen,
> > >
> > > Thanks a lot for your
> I think it's moving to the right direction.
>
> No further comments from my side.
>
> Thanks Arjun!
>
> - Konstantine
>
> On Fri, May 18, 2018 at 1:07 AM, Arjun Satish
> wrote:
>
> > Ewen,
> >
> > Thanks a lot for your comments!
> >
&g
ead of delivering the raw
> data, we potentially lose raw data & schema info because we're rendering it
> as JSON. Not sure that's a good idea...
>
> I think that last item might be the biggest concern to me -- DLQ formats
> and control over content & reproces
t; "For errors in a source connector, the process is similar, but care needs
> to be taken while writing back to the source." and sounds like it's
> suggested that Connect will write records back to the source, which can't
> be correct.
>
> Finally, a nit: "
All,
Many thanks for all the feedback on KIP-298. Highly appreciate the time and
effort you all put into it.
I've updated the KIP accordingly, and would like to start to start a vote
on it.
KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
298%3A+Error+Handling+in+Connect
JIRA: https:/
te records can
easily creep in, and can be notoriously hard to detect and clean up.
> Thanks,
> Matt
>
> On Tue, May 15, 2018 at 8:46 PM, Arjun Satish
> wrote:
>
> > Matt,
> >
> > Thanks so much for your comments. Really appreciate it!
> >
> > 1. Good po
ons.
>
> Thanks,
> Magesh
>
> On Wed, May 16, 2018 at 6:56 PM, Arjun Satish
> wrote:
>
> > Hi Konstantine,
> >
> > Thanks a lot for your feedback. I have made the necessary changes to the
> > KIP.
> >
> > Best,
> >
> > On Wed, Ma
illed'
>
> - If you intend to add a link to a PR additionally to the jira ticket, it'd
> be handy to add it to the KIP header (along with state, thread, jira, etc).
> Now it's a bit hidden in the text and it's not clear that the KIP includes
> a link to a P
TimeoutExceptions (which extend RetriableExceptions)
are bubbled back to the application, and need to be handled as per
application requirements.
Best,
On Tue, May 15, 2018 at 8:30 PM, Arjun Satish
wrote:
> Magesh,
>
> Thanks for the feedback! Really appreciate your comments.
>
> 1
t; for each of them? I would think most people might want the behavior applied
> to all the connectors.
>
> Let me know your thoughts :).
>
> Thanks
> Magesh
>
> On Tue, May 8, 2018 at 11:59 PM, Arjun Satish
> wrote:
>
> > All,
> >
> > I'
nfig that temporarily whitelisted that Exception as
> retry-worthy
> and continued attempting to make progress while the other team worked
> on mitigating the problem.
>
> Thanks for the KIP!
>
> On Wed, May 9, 2018 at 2:59 AM, Arjun Satish
> wrote:
>
> > All,
All,
I'd like to start a discussion on adding ways to handle and report record
processing errors in Connect. Please find a KIP here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%3A+Error+Handling+in+Connect
Any feedback will be highly appreciated.
Thanks very much,
Arjun
Thank you!
On Mon, May 7, 2018 at 5:41 PM, Matthias J. Sax
wrote:
> Done :)
>
> On 5/7/18 2:30 PM, Arjun Satish wrote:
> > Could someone please grant me the permissions to submit a KIP? My
> username
> > on apache.org is wicknicks.
> >
> > Thanks very much,
> > Arjun
> >
>
>
Could someone please grant me the permissions to submit a KIP? My username
on apache.org is wicknicks.
Thanks very much,
Arjun
Arjun Satish created KAFKA-6833:
---
Summary: KafkaProducer throws "Invalid partition given with
record" exception
Key: KAFKA-6833
URL: https://issues.apache.org/jira/browse/KAFKA-6833
Proj
Arjun Satish created KAFKA-6511:
---
Summary: Connect header parser incorrectly parses arrays
Key: KAFKA-6511
URL: https://issues.apache.org/jira/browse/KAFKA-6511
Project: Kafka
Issue Type: Bug
Hi,
Could you please add me to the contributors list for Apache Kafka?
My Apache username is wicknicks, and the profile page is located here (
https://issues.apache.org/jira/secure/ViewProfile.jspa?name=wicknicks).
Thanks very much,
Arjun
77 matches
Mail list logo
