Re: [VOTE] KIP-349 Priorities for Source Topics

also: https://cwiki.apache.org/confluence/display/KAFKA/KIP-353%3A+Improve+Kafka+Streams+Timestamp+Synchronization On 20.08.2018 15:01, Thomas Becker wrote: I agree with Jan. A strategy interface for choosing processing order is nice, and would hopefully be a step towards getting this in stre

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Generally, I think positive configs (`allow` instead of `suppress`) are easier to understand. Ismael On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax wrote: > Thanks for the summary! > > We might want to add a diagram/table to the docs when we add this > feature (with whatever config name we ch

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Hi Jason, Thanks for the feedback. 1. I chose to return null here because according to the documentation it may return null data, therefore the users of this methods are perpared for getting a null. Thinking of it though it may be better to throw an exception by default because it'd indicate a pro

Re: Current Kafka Steams and KSQL Performance Metrics / Benchmarks?

Thanks Guozhang! I am asking primarily because I have seen Flink & Spark Streaming users boasting of millions of records / second being processed and was interested to learn where Kafka Streams / KSQL stands. This would also help a lot in capability planning for teams looking to use Kafka Streams.

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

I think the KIP needs the rejected alternatives section to have more detail. For example, another option would be something like the following, which works great as long as one overrides one of the methods, but pretty bad if one doesn't. :) default T deserialize(String topic, byte[] data) { re

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Also, we may consider deprecating the deserialize method that does not take headers. Yes, it's a convenience, but it also adds confusion. Ismael On Thu, Aug 23, 2018 at 6:48 AM Ismael Juma wrote: > I think the KIP needs the rejected alternatives section to have more > detail. For example, anoth

Re: [DISCUSS] KIP-358: Migrate Streams API to Duration instead of long ms times

Hello, Mathias. Thanks for your feedback. > Thus, it might make sense to keep old and just add new ones? As far as I understand, we will keep old methods anyway to prevent public API backward compatibility. I agree with you, methods that used internally shouldn't be deprecated. > End users ca

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

We recently had a pretty serious Kafka outage caused by a bug in one of our consumers that caused it to create new topics in an infinite loop AKA a topic bomb! Having consumers restricted from creating topics

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Thanks for sharing Andrew! Ismael On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto wrote: > We recently had a pretty serious Kafka outage > < > https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary > > > caused by a bug in one of our consumers that caused it to crea

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Andrew, one question: you are relying on auto topic creation for the producer and that's why you can't just disable it? On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma wrote: > Thanks for sharing Andrew! > > Ismael > > On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto wrote: > >> We recently had a pretty

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Yup :) On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma wrote: > Andrew, one question: you are relying on auto topic creation for the > producer and that's why you can't just disable it? > > On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma wrote: > > > Thanks for sharing Andrew! > > > > Ismael > > > > O

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Ah, I just realized that as proposed this is only for the Java consumer client, correct? Would it be possible to make this a broker config, like the current one? Something like: auto.create.topics.enable=true # allow both producer and consumer to create auto.create.topics.enable=consumer # allow

[jira] [Resolved] (KAFKA-6309) add support for getting topic defaults from AdminClient

[ https://issues.apache.org/jira/browse/KAFKA-6309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6309. -- Resolution: Fixed Closing as per above comment. We can use "describeConfigs(brokerResource, new Descri

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Yeah, the reason why we want to deprecate the auto create functionality is that it happens when a metadata request is done instead of when a write operation happens. So, there's no reason to differentiate between the two. Ismael On Thu, Aug 23, 2018 at 8:16 AM Andrew Otto wrote: > Ah, I just re

Build failed in Jenkins: kafka-trunk-jdk8 #2919

See Changes: [ismael] MINOR: Eliminate warnings from KafkaProducerTest (#5548) -- [...truncated 870.50 KB...] kafka.security.auth.ZkAuthorizationTest > testDeleteRecursiv

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

It seems nobody has any objections against the change. That's for the KIP improvement. I'll go ahead and merge the PR. -Matthias On 8/21/18 2:44 PM, John Roesler wrote: > Hello again, all, > > I belatedly had a better idea for adding grace period to the Windows class > hierarchy (TimeWindows,

[jira] [Created] (KAFKA-7331) Kafka does not detect broker loss in the event of a network partition within the cluster

Kevin Li created KAFKA-7331: --- Summary: Kafka does not detect broker loss in the event of a network partition within the cluster Key: KAFKA-7331 URL: https://issues.apache.org/jira/browse/KAFKA-7331 Project:

[jira] [Created] (KAFKA-7332) Improve error message when trying to produce message without key for compacted topic

Patrik Kleindl created KAFKA-7332: - Summary: Improve error message when trying to produce message without key for compacted topic Key: KAFKA-7332 URL: https://issues.apache.org/jira/browse/KAFKA-7332

[jira] [Created] (KAFKA-7333) Protocol changes for KIP-320

Jason Gustafson created KAFKA-7333: -- Summary: Protocol changes for KIP-320 Key: KAFKA-7333 URL: https://issues.apache.org/jira/browse/KAFKA-7333 Project: Kafka Issue Type: Bug Re

[DISCUSS] KIP-363: Make FunctionConversions private

Hi, As pointed out in this comment #5539 (comment) the object FunctionConversions is only of internal use and therefore should be private to the lib only so that we can do changes without going through KIP like this one. KIP: https

Build failed in Jenkins: kafka-trunk-jdk10 #429

See Changes: [wangguoz] KAFKA-7316: Fix Streams Scala filter recursive call #5538 -- [...truncated 1.98 MB...] org.apache.kafka.streams.processor.internals.assignment.Stic

Re: Build failed in Jenkins: kafka-trunk-jdk10 #429

I'm looking into this one. On Thu, 23 Aug 2018 at 20:19 Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See < > https://builds.apache.org/job/kafka-trunk-jdk10/429/display/redirect?page=changes > > > > Changes: > > [wangguoz] KAFKA-7316: Fix Streams Scala filter recursive call #5538 >

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Hi Ismael, Regarding the deprecation of the 2 parameter method: should we do this with the Serializer interface as well? I've updated the "Rejected Alternatives" with a few. I've added this circular reference one too but actually there's a way (pretty heavyweight) by adding a guard class that pre

[jira] [Created] (KAFKA-7334) Suggest changing config for state.dir in case of FileNotFoundException

Ted Yu created KAFKA-7334: - Summary: Suggest changing config for state.dir in case of FileNotFoundException Key: KAFKA-7334 URL: https://issues.apache.org/jira/browse/KAFKA-7334 Project: Kafka Issue

Re: [DISCUSS] KIP-357: Add support to list ACLs per principal

+1 (binding) Thanks, Harsha On Wed, Aug 22, 2018, at 9:15 AM, Manikumar wrote: > Hi Viktor, > We already have a method in Authorizer interface to get acls for a given > principal. > We will use this method to fetch acls and filter the results for requested > Resources. > Authorizer { >def get

Re: [DISCUSS] KIP-363: Make FunctionConversions private

+1 In the Motivation section, you can quote the comment from pull request so that reader doesn't have to click through. Cheers On Thu, Aug 23, 2018 at 12:13 PM Joan Goyeau wrote: > Hi, > > As pointed out in this comment #5539 (comment) >

Re: Build failed in Jenkins: kafka-trunk-jdk10 #429

I ran streams unit tests as of commit 4156ea0a9bcca67d209fd3b43d2268c9abd5a0b5 . All tests passed locally. FYI On Thu, Aug 23, 2018 at 12:23 PM Joan Goyeau wrote: > I'm looking into this one. > > On Thu, 23 Aug 2018 at 20:19 Apache Jenkins Server < > jenk...@builds.apache.org> wrote: > > > See

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Hey Viktor, Thinking about it a little more, I wonder if we should just not provide a default method for serialize(topic, data) and deserialize(topic, data). Implementing these methods is a trivial burden for users and it feels like there's no good solution which allows both methods to have defaul

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

To clarify, what I am suggesting is to only remove the default implementation for these methods. So users would be required to implement serialize(topic, data) and deserialize(topic, data). -Jason On Thu, Aug 23, 2018 at 1:48 PM, Jason Gustafson wrote: > Hey Viktor, > > Thinking about it a litt

Jenkins build is back to normal : kafka-trunk-jdk8 #2920

See

[jira] [Created] (KAFKA-7335) Store clusterId locally to ensure broker joins the right cluster

Jason Gustafson created KAFKA-7335: -- Summary: Store clusterId locally to ensure broker joins the right cluster Key: KAFKA-7335 URL: https://issues.apache.org/jira/browse/KAFKA-7335 Project: Kafka

Metadata Update Corner Case

While updating a small Kafka client application to use Kafka 1.1.1 and the KafkaProducer (rather than the deprecated Producer) a maintenance related unit test started failing. The scenario is out of the ordinary, but, the resulting problem may still be of interest: * Two Kafka nodes * O

Re: [VOTE] KIP-291: Have separate queues for control requests and data requests

Hi, Lucas, Sorry for the delay. The new proposal looks good to me overall. A few minor comments below. 1. It's possible that listener.name.for.controller is set, but set to the same value as inter.broker.listener.name. In that case, should we have a single network thread and the request handling

Build failed in Jenkins: kafka-trunk-jdk10 #430

See Changes: [wangguoz] MINOR: Return correct instance of SessionWindowSerde (#5546) -- [...truncated 1.53 MB...] kafka.coordinator.transaction.TransactionCoordinatorTest

Re: [VOTE] KIP-346 - Improve LogCleaner behavior on error

Hi, Stan, Thanks for the KIP. Looks good to me overall. Just one comment below. uncleanable-partitions-count is per logDir, but uncleanable-bytes is not. Should we make them consistent? Jun On Wed, Aug 22, 2018 at 4:15 AM, Stanislav Kozlovski wrote: > Hi everybody, > > @Jason - I've updated

Re: [DISCUSS] KIP-280: Enhanced log compaction

Hi, Luis, Thanks for the reply. A few more comments below. 1. About the topic level configuration. It seems that it's useful for the new configs to be at the topic level. Currently, the following configs related to compaction are already at the topic level. min.cleanable.dirty.ratio min.compacti

Re: [DISCUSS] KIP-363: Make FunctionConversions private

+1. On Thu, Aug 23, 2018 at 12:47 PM, Ted Yu wrote: > +1 > > In the Motivation section, you can quote the comment from pull request so > that reader doesn't have to click through. > > Cheers > > On Thu, Aug 23, 2018 at 12:13 PM Joan Goyeau wrote: > > > Hi, > > > > As pointed out in this comment

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hello Boyang, Thanks for the updated proposal, a few questions: 1. Where will "change-group-timeout" be communicated to the broker? Will that be a new field in the JoinGroupRequest, or are we going to piggy-back on the existing session-timeout field (assuming that the original value will not be u

Build failed in Jenkins: kafka-trunk-jdk10 #431

See Changes: [jason] KAFKA-7225; Corrected system tests by generating external properties [jason] MINOR: Make JAAS configurable via template variables in system tests --

hi

Tony --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

Request for contributor permissions

JIRA ID: tony80720 Cwiki ID: 王又田 Thanks in advance! Tony --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

Request for contributor permissions

JIRA ID: tony80720 Cwiki ID: Yu Tien Wang Thanks in advance! Tony --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

Jenkins build is back to normal : kafka-trunk-jdk10 #432

See

Build failed in Jenkins: kafka-trunk-jdk8 #2922

See Changes: [jason] MINOR: Make JAAS configurable via template variables in system tests [wangguoz] MINOR: Fix streams Scala foreach recursive call (#5539) [mjsax] MINOR: restructure Windows to favor immutable

Re: [DISCUSS] KIP-357: Add support to list ACLs per principal

Hi Mani, Just a minor comment on the output of the command as given in KIP-357, you may want to remove "User:User1 has " as it is redundant for each ACL. It may be good to accept multiple principals option to avoid running this script multiple times with each principal to achieve the same. >> sh

Re: [DISCUSS] KIP-363: Allow performance tools to print final results to output file

Hi Attila, Thanks for the KIP. I think this would be a useful feature. Every time I have to benchmark using these performance tools, I end up redirecting the output to a file anyways. Just a couple minor questions... 1. If the configured file already exists, what would be the outcome? My intuit