[jira] [Resolved] (KAFKA-10244) An new java interface to replace 'kafka.common.MessageReader'

2023-03-25 Thread Chia-Ping Tsai (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-10244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-10244.

Fix Version/s: 3.5.0
   Resolution: Fixed

> An new java interface to replace 'kafka.common.MessageReader'
> -
>
> Key: KAFKA-10244
> URL: https://issues.apache.org/jira/browse/KAFKA-10244
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
>  Labels: need-kip, needs-kip
> Fix For: 3.5.0
>
>
> inspired by 
> https://github.com/apache/kafka/commit/caa806cd82fb9fa88510c81de53e69ac9846311d.
> kafka.common.MessageReader is a pure scala trait and we should offer a java 
> replacement to users.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1707

2023-03-25 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 2040803 lines...]
[2023-03-25T17:45:20.590Z] Caused by:
[2023-03-25T17:45:20.590Z] java.lang.NullPointerException
[2023-03-25T17:45:20.590Z] 
[2023-03-25T17:45:20.590Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 180 > PauseResumeIntegrationTest > 
shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean) > 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean)[2]
 STARTED
[2023-03-25T17:46:21.082Z] 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean)[2]
 failed, log available in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/build/reports/testOutput/org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean)[2].test.stdout
[2023-03-25T17:46:21.082Z] 
[2023-03-25T17:46:21.082Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 180 > PauseResumeIntegrationTest > 
shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean) > 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean)[2]
 FAILED
[2023-03-25T17:46:21.082Z] java.lang.AssertionError: Assertion failed with 
an exception after 6 ms
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:395)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:353)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:726)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:699)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.awaitOutput(PauseResumeIntegrationTest.java:416)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(PauseResumeIntegrationTest.java:218)
[2023-03-25T17:46:21.082Z] 
[2023-03-25T17:46:21.082Z] Caused by:
[2023-03-25T17:46:21.082Z] 
org.apache.kafka.common.errors.RecordDeserializationException: Error 
deserializing key/value for partition output-stream-1-0 at offset 0. If needed, 
please seek past the record to continue consumption.
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.clients.consumer.internals.CompletedFetch.parseRecord(CompletedFetch.java:312)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:264)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.clients.consumer.internals.AbstractFetch.fetchRecords(AbstractFetch.java:340)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.clients.consumer.internals.AbstractFetch.collectFetch(AbstractFetch.java:306)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1300)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1251)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1224)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.readRecords(IntegrationTestUtils.java:1301)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.readKeyValues(IntegrationTestUtils.java:1259)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinKeyValueRecordsReceived$1(IntegrationTestUtils.java:728)
[2023-03-25T17:46:21.082Z] at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:385)
[2023-03-25T17:46:21.082Z] ... 5 more
[2023-03-25T17:46:21.082Z] 
[2023-03-25T17:46:21.082Z] Caused by:
[2023-03-25T17:46:21.082Z] java.lang.NullPointerException
[2023-03-25T17:46:21.082Z] 
[2023-03-25T17:46:21.082Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 180 > PauseResumeIntegrationTest > 
shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies(boolean) > 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies(boolean)[1]
 STARTED
[2023-03-25T17:47:30.934Z] 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeAllKafkaStreamsWithNam

[jira] [Created] (KAFKA-14848) KafkaConsumer incorrectly passes locally-scoped serializers to FetchConfig

2023-03-25 Thread Kirk True (Jira) 
Kirk True created KAFKA-14848:
-

 Summary: KafkaConsumer incorrectly passes locally-scoped 
serializers to FetchConfig
 Key: KAFKA-14848
 URL: https://issues.apache.org/jira/browse/KAFKA-14848
 Project: Kafka
  Issue Type: Bug
  Components: clients, consumer
Affects Versions: 3.5.0
Reporter: Kirk True
Assignee: Kirk True


[~rayokota] found some {{{}NullPointerException{}}}s that originate because of 
a recently introduced error in the {{KafkaConsumer}} constructor. The code was 
changed to  pass the serializers variables into the {{FetchConfig}} 
constructor. However, this code change incorrectly used the locally-scoped 
variables, not the instance-scoped variables. Since the locally-scoped 
variables could be {{{}null{}}}, this results in the {{FetchConfig}} storing 
{{null}} references, leading to downstream breakage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

2023-03-25 Thread Sagar 
Hi John,

Thanks for taking. look at the KIP!

The point about stream time not advancing in case of infrequent updates is
an interesting one. I can imagine if the upstream producer to a Kafka
Streams application is a Source Connector which isn't sending records
frequently(due to the nature of the data ingestion for example), then the
downstream stream processing can land into the issues you described above.

Which also brings me to the second point you made about how this would be
used by downstream consumers. IIUC, you are referring to the consumers of
the newly added topic i.e the heartbeat topic. In my mind, the heartbeat
topic is an internal topic (similar to offsets/config/status topic in
connect), the main purpose of which is to trick the framework to produce
records to the offsets topic and advance the offsets. Since every connector
could have a different definition of offsets(LSN, BinLogID etc for
example), that logic to determine what the heartbeat records should be
would have to reside in the actual connector.

Now that I think of it, it could very well be consumed by downstream
consumers/ Streams or Flink Applications and be further used for some
decision making. A very crude example could be let's say if the heartbeat
records sent to the new heartbeat topic include timestamps, then the
downstream streams application can use that timestamp to close any time
windows. Having said that, it still appears to me that it's outside the
scope of the Connect framework and is something which is difficult to
generalise because of the variety of Sources and the definitions of offsets.

But, I would still be more than happy to add this example if you think it
can be useful in getting a better understanding of the idea and also its
utility beyond connect. Please let me know!

Thanks!
Sagar.


On Fri, Mar 24, 2023 at 7:22 PM John Roesler  wrote:

> Thanks for the KIP, Sagar!
>
> At first glance, this seems like a very useful feature.
>
> A common pain point in Streams is when upstream producers don't send
> regular updates and stream time cannot advance. This causes
> stream-time-driven operations to appear to hang, like time windows not
> closing, suppressions not firing, etc.
>
> From your KIP, I have a good idea of how the feature would be integrated
> into connect, and it sounds good to me. I don't quite see how downstream
> clients, such as a downstream Streams or Flink application, or users of the
> Consumer would make use of this feature. Could you add some examples of
> that nature?
>
> Thank you,
> -John
>
> On Fri, Mar 24, 2023, at 05:23, Sagar wrote:
> > Hi All,
> >
> > Bumping the thread again.
> >
> > Sagar.
> >
> >
> > On Fri, Mar 10, 2023 at 4:42 PM Sagar  wrote:
> >
> >> Hi All,
> >>
> >> Bumping this discussion thread again.
> >>
> >> Thanks!
> >> Sagar.
> >>
> >> On Thu, Mar 2, 2023 at 3:44 PM Sagar  wrote:
> >>
> >>> Hi All,
> >>>
> >>> I wanted to create a discussion thread for KIP-910:
> >>>
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-910%3A+Update+Source+offsets+for+Source+Connectors+without+producing+records
> >>>
> >>> Thanks!
> >>> Sagar.
> >>>
> >>
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1708

2023-03-25 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 3893124 lines...]
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:699)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.awaitOutput(PauseResumeIntegrationTest.java:416)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(PauseResumeIntegrationTest.java:218)
[2023-03-26T01:50:24.688Z] 
[2023-03-26T01:50:24.688Z] Caused by:
[2023-03-26T01:50:24.688Z] 
org.apache.kafka.common.errors.RecordDeserializationException: Error 
deserializing key/value for partition output-stream-1-0 at offset 0. If needed, 
please seek past the record to continue consumption.
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.clients.consumer.internals.CompletedFetch.parseRecord(CompletedFetch.java:312)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:264)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.clients.consumer.internals.AbstractFetch.fetchRecords(AbstractFetch.java:340)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.clients.consumer.internals.AbstractFetch.collectFetch(AbstractFetch.java:306)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1300)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1251)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1224)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.readRecords(IntegrationTestUtils.java:1301)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.readKeyValues(IntegrationTestUtils.java:1259)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinKeyValueRecordsReceived$1(IntegrationTestUtils.java:728)
[2023-03-26T01:50:24.688Z] at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:385)
[2023-03-26T01:50:24.688Z] ... 5 more
[2023-03-26T01:50:24.688Z] 
[2023-03-26T01:50:24.688Z] Caused by:
[2023-03-26T01:50:24.688Z] java.lang.NullPointerException
[2023-03-26T01:50:24.688Z] 
[2023-03-26T01:50:24.688Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 180 > PauseResumeIntegrationTest > 
shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean) > 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean)[2]
 STARTED
[2023-03-26T01:51:28.166Z] 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean)[2]
 failed, log available in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/build/reports/testOutput/org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean)[2].test.stdout
[2023-03-26T01:51:28.166Z] 
[2023-03-26T01:51:28.166Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 180 > PauseResumeIntegrationTest > 
shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean) > 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(boolean)[2]
 FAILED
[2023-03-26T01:51:28.166Z] java.lang.AssertionError: Assertion failed with 
an exception after 6 ms
[2023-03-26T01:51:28.166Z] at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:395)
[2023-03-26T01:51:28.166Z] at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:353)
[2023-03-26T01:51:28.166Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:726)
[2023-03-26T01:51:28.166Z] at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:699)
[2023-03-26T01:51:28.166Z] at 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.awaitOutput(PauseResumeIntegrationTest.java:416)
[2023-03-26T01:51:28.166Z] at 
org.apache.kafka.streams.integration.PauseResumeIntegrationTest.shouldPauseAndResumeKafkaStreamsWithNamedTopologies(PauseResumeIntegrationTest.java:218)
[2023-03-26T01:51:28.166Z] 
[2023-03-26T01:51:28.166Z] Caused by:
[2023-03-26T01:51:28.166Z]