Re: best practice of storing JSON that is searchable

Ok. thanks. > On Nov 27, 2018, at 11:31 PM, Jörn Franke wrote: > > Why not put it in Solr or similar ? > >> Am 28.11.2018 um 08:11 schrieb Jiayi Hoffman : >> >> Hi, >> >> We have an application that uses MySQL to store devices and its metrics. We >> are thinking of migrating all data to Kafk

Re: best practice of storing JSON that is searchable

Why not put it in Solr or similar ? > Am 28.11.2018 um 08:11 schrieb Jiayi Hoffman : > > Hi, > > We have an application that uses MySQL to store devices and its metrics. We > are thinking of migrating all data to Kafka, and using Kafka as the storage > for the benefit of scaling, event souring

[jira] [Created] (KAFKA-7684) kafka consumer SchemaException occurred: Error reading field 'brokers':

leibo created KAFKA-7684: Summary: kafka consumer SchemaException occurred: Error reading field 'brokers': Key: KAFKA-7684 URL: https://issues.apache.org/jira/browse/KAFKA-7684 Project: Kafka Issue

best practice of storing JSON that is searchable

Hi, We have an application that uses MySQL to store devices and its metrics. We are thinking of migrating all data to Kafka, and using Kafka as the storage for the benefit of scaling, event souring and searchability in JSON. I am looking for some advice on the topic design for my following us

[jira] [Resolved] (KAFKA-7620) ConfigProvider is broken for KafkaConnect when TTL is not null

[ https://issues.apache.org/jira/browse/KAFKA-7620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-7620. -- Resolution: Fixed Fix Version/s: 2.0.2 2.1.1

Re: problems in Kafka unit testing trunk

The unit test itself does not seem to use too many files. What is the output for `ulimit -n` on your system? Running `lsof` might also be helpful to determine how many open files you have while Kafka is not running. - Dhruvil On Tue, Nov 27, 2018 at 9:20 AM lk gen wrote: > When running ./gradle

Build failed in Jenkins: kafka-trunk-jdk8 #3221

See Changes: [github] KAFKA-7389: Enable spotBugs with Java 11 and disable false positive -- [...truncated 2.63 MB...] org.apache.kafka.streams.test.OutputVerifierTest >

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

Thanks for the KIP Boyang. I guess I am missing something, but I am still learning more details about the rebalance protocol, so maybe you can help me out? Assume a client sends UNKNOWN_MEMBER_ID in its first joinGroup request. The broker generates a `member.id` and sends it back via `MEMBER_ID_R

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

0. My original question is about the implementation details primarily, since current the handling logic of the APIVersionResponse is simply "use the highest supported version of the corresponding request", but if the returned response from APIVersionRequest says "I don't even know about the Describ

Jenkins build is back to normal : kafka-trunk-jdk11 #118

See

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

Thanks Stanislav and Jason for the suggestions! > Thanks for the KIP. Looks good overall. I think we will need to bump the > version of the JoinGroup protocol in order to indicate compatibility with > the new behavior. The coordinator needs to know when it is safe to assume > the client will hand

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Randall, have you considered something like: - introduce TopicCreationPolicy interface, with methods like partitionsForTopic(topic). - provide a DefaultTopicCreationPolicy implementation that implements the current behavior. - provide a SimpleTopicCreationPolicy that honors topic.creation.partitio

Jenkins build is back to normal : kafka-trunk-jdk8 #3220

See

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

Hi Florian, I've made a pass over the PR. There are some comments that are related to the function names which may be affecting the KIP wiki page, but overall I think it looks good already. Guozhang On Fri, Nov 16, 2018 at 4:21 PM Guozhang Wang wrote: > Thanks Florian! I will take a look at

[jira] [Created] (KAFKA-7683) Support ConfigDef.Type.MAP

Paul Czajka created KAFKA-7683: -- Summary: Support ConfigDef.Type.MAP Key: KAFKA-7683 URL: https://issues.apache.org/jira/browse/KAFKA-7683 Project: Kafka Issue Type: Bug Components: cl

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Thanks for the feedback. Some thoughts inline. On Tue, Nov 27, 2018 at 5:47 PM Ewen Cheslack-Postava wrote: > re: AdminClient vs this proposal, one consideration is that AdminClient > exposes a lot more surface area and probably a bunch of stuff we actually > don't want Connectors to be able to

Build failed in Jenkins: kafka-trunk-jdk11 #117

See Changes: [colin] KAFKA-7597: Add transaction support to ProduceBenchWorker (#5885) [wangguoz] KAFKA-7223: Suppression Buffer Metrics (#5795) [wangguoz] MINOR: Add system test for optimization upgrades (#591

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

re: AdminClient vs this proposal, one consideration is that AdminClient exposes a lot more surface area and probably a bunch of stuff we actually don't want Connectors to be able to do, such as deleting topics. You can always lock down by ACLs, but what the framework enables directly vs requiring t

[jira] [Created] (KAFKA-7682) turning on request logging for a subset of request types

Jun Rao created KAFKA-7682: -- Summary: turning on request logging for a subset of request types Key: KAFKA-7682 URL: https://issues.apache.org/jira/browse/KAFKA-7682 Project: Kafka Issue Type: Improv

[jira] [Created] (KAFKA-7681) new metric for request thread utilization by request type

Jun Rao created KAFKA-7681: -- Summary: new metric for request thread utilization by request type Key: KAFKA-7681 URL: https://issues.apache.org/jira/browse/KAFKA-7681 Project: Kafka Issue Type: Impro

[jira] [Created] (KAFKA-7680) fetching a refilled chunk of log can cause log divergence

Jun Rao created KAFKA-7680: -- Summary: fetching a refilled chunk of log can cause log divergence Key: KAFKA-7680 URL: https://issues.apache.org/jira/browse/KAFKA-7680 Project: Kafka Issue Type: Bug

[jira] [Resolved] (KAFKA-7367) Streams should not create state store directories unless they are needed

[ https://issues.apache.org/jira/browse/KAFKA-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-7367. Resolution: Fixed Fix Version/s: 2.2.0 > Streams should not create state store direc

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Hey y'all, I'd like you draw your attention to a new section in KIP-382 re MirrorMaker Clusters: https://cwiki.apache.org/confluence/display/KAFKA/KIP-382:+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-MirrorMakerClusters A common concern I hear about using Connect for replication is that all SourceConn

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Regarding Jason's question and Boyang's responses: 2) I once have a discussion about the LeaveGroupRequest for static members, and the reason for not having it for static members is that we'd need to make it a configurable behavior as well (i.e. the likelihood that a static member may shutdown but

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

Hi Boyang, The KIP looks very good. One small question I have is now that we have one and a half round-trips needed to join in a rebalance (1 full RT addition), is it worth it to consider increasing the default value of `group.initial.rebalance.delay.ms`? Best, Stanislav On Tue, Nov 27, 2018 at

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Thanks for the replies. Please find the follow up queries below. 5. Regarding "So in summary, *the member will only be removed due to session timeout*. We shall remove it from both in-memory static member name mapping and member list." If the rebalance is invoked manually using the

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

Hi Boyang, Thanks for the KIP. Looks good overall. I think we will need to bump the version of the JoinGroup protocol in order to indicate compatibility with the new behavior. The coordinator needs to know when it is safe to assume the client will handle the error code. Also, I was wondering if w

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

Hey Stanislav, Clients will then find that coordinator > and send `joinGroup` on it, effectively rebuilding the group, since the > cache of active consumers is not stored outside the Coordinator's memory. > (please do say if that is incorrect) Groups do not typically rebalance after a coordinato

problems in Kafka unit testing trunk

When running ./gradlew test on a centos machine with gradle and java set In the trunk version from today There are errors about too many files open of the form " kafka.admin.DeleteTopicTest > testDeletingPartiallyDeletedTopic FAILED org.apache.kafka.common.KafkaException: java.io.IOException:

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

Hi Mayuresh 1. we were envisioning the 1:1 case, however as long as topic names do not clash, you could replicate multiple cluster into a single replica, or use topic prefixes on the destination. 2. using an idempotent producer in the replicator would be recommended in the replicator. 3. Wh

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

Hi Jason we envisioned the replicator to replicate the __consumer_offsets topic too (although without producing-with-offsets to it!). As there is no client-side implementation yet using the leader epoch, we could not yet see the impact of writing to the destination cluster __consumer_offsets r

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

Thanks for the KIP, this is helpful. A few questions: 1. Have we considered whether we want to allow a similar topic config? 2. Can we rely on a method from the library to pick the default compression level if compression.level is not set? We do it for gzip and it would seem reasonable to do some

[jira] [Resolved] (KAFKA-7508) Kafka broker anonymous disconnected from Zookeeper

[ https://issues.apache.org/jira/browse/KAFKA-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Santilli resolved KAFKA-7508. -- Resolution: Not A Problem Closed since was requested by the reported. Sathish agree to

[jira] [Created] (KAFKA-7679) With acks=all a single "stuck" non-leader replica can cause a timeout

Corentin Chary created KAFKA-7679: - Summary: With acks=all a single "stuck" non-leader replica can cause a timeout Key: KAFKA-7679 URL: https://issues.apache.org/jira/browse/KAFKA-7679 Project: Kafka

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

Hi Radai > 1. how do you handle possible duplications caused by the "special" > producer timing-out/retrying? are you explicitely relying on the > "exactly once" sequencing? A duplicate ProduceRequest would be rejected with an INVALID_PRODUCE_OFFSET error. We envision using an idempotent produc

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

Is there anyone who is interested in this KIP? Thanks, Dongjin On Mon, Nov 19, 2018 at 7:06 AM Dongjin Lee wrote: > Hello dev, > > I hope to initiate the discussion of KIP-390: Add producer option to > adjust compression level >

[jira] [Created] (KAFKA-7678) Failed to close producer due to java.lang.NullPointerException

Jonathan Santilli created KAFKA-7678: Summary: Failed to close producer due to java.lang.NullPointerException Key: KAFKA-7678 URL: https://issues.apache.org/jira/browse/KAFKA-7678 Project: Kafka

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

Hey Jason and Boyang, those were important comments > One suggestion I have is that it would be helpful to put your reasoning on deciding the current default value. For example, in certain use cases at Pinterest we are very likely to have more consumers than 250 when we configure 8 stream instance

[jira] [Created] (KAFKA-7677) Client login with already existing JVM subject

Gabor Somogyi created KAFKA-7677: Summary: Client login with already existing JVM subject Key: KAFKA-7677 URL: https://issues.apache.org/jira/browse/KAFKA-7677 Project: Kafka Issue Type: New