Use Android App as a “Producing client” for Kafka?

Hi folks, is it possible / does it make sense to use an Android app as a "Producing client" for Apache Kafka? Let's say my Android App need to capture and analyse reaction time data. Goal is to collect all data and show the average reaction time in real-time in the App. The alternative is hav

Re: [DISCUSS] KAFKA-4345: Run decktape test for each pull request

Yeah, I'm all for getting these to run more frequently and on lighter weight infrastructure. (By the way, I also saw the use of docker; I'd really like to get a "native" docker cluster type into ducktape at some point so all you have to do is bake the image and then spawn containers on demand.) A

Re: Kafka Connect key.converter and value.converter properties for Avro encoding

You won't be accepting/returning SpecificRecords directly when working with Connect's API. Connect intentionally uses an interface that is different from Kafka serializers because we deal with structured data that the connectors need to be able to understand. We define a generic runtime representat

[jira] [Commented] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

[ https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642883#comment-15642883 ] Jun Yao commented on KAFKA-4385: yeah, the case is 1. broker has disabled topic auto crea

[jira] [Commented] (KAFKA-4381) Add per partition lag metric to KafkaConsumer.

[ https://issues.apache.org/jira/browse/KAFKA-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642796#comment-15642796 ] Jiangjie Qin commented on KAFKA-4381: - Yeah, it should be a simple one. I'll put toget

[jira] [Commented] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

[ https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642789#comment-15642789 ] Jiangjie Qin commented on KAFKA-4385: - It seems you are more worrying about the blocki

[jira] [Commented] (KAFKA-4381) Add per partition lag metric to KafkaConsumer.

[ https://issues.apache.org/jira/browse/KAFKA-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642780#comment-15642780 ] Ewen Cheslack-Postava commented on KAFKA-4381: -- I suppose since metrics can b

[GitHub] kafka pull request #2108: MINOR: Fix regex on connector path param in Connec...

GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/2108 MINOR: Fix regex on connector path param in ConnectorsResource You can merge this pull request into a Git repository by running: $ git pull https://github.com/ewencp/kafka fix-rest-api-regex A

[jira] [Commented] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

[ https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642343#comment-15642343 ] Jun Yao commented on KAFKA-4385: Thanks, I tried changing 'retry.backoff.ms', it does hel

Re: [DISCUSS] KIP-82 - Add Record Headers

making header _key_ serialization configurable potentially undermines the board usefulness of the feature (any point along the path must be able to read the header keys. the values may be whatever and require more intimate knowledge of the code that produced specific headers, but keys should be uni

[jira] [Commented] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

[ https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642072#comment-15642072 ] Jiangjie Qin commented on KAFKA-4385: - Thanks for the explanation. Sorry I misundersto

[jira] [Comment Edited] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

[ https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642021#comment-15642021 ] Jun Yao edited comment on KAFKA-4385 at 11/6/16 4:19 PM: - [~becket

[jira] [Commented] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

[ https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642021#comment-15642021 ] Jun Yao commented on KAFKA-4385: [~becket_qin] As said in this ticket title, the unneces

[jira] [Commented] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

[ https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15641989#comment-15641989 ] Jiangjie Qin commented on KAFKA-4385: - [~Jun Yao] Is this really a problem? The loop i

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

Hi David, I am thinking that depending on the use case, we may not need a separate tool to have the committed message based retention using the trim() method. One way to do this is to have a configuration like trim.on.offset.commit in the consumer so after committing the offset, the consumer will

?????? [DISCUSS] KIP-68 Add a consumed log retention before log retention

Hi Becket, The most important benefit of method (2) is we can safely delete the log segments, becasue all the deleted log segments are consumed. If the messages are very important, in this case we need to safely delete the log segments instead of forcing delete it after the retention time. K

[jira] [Commented] (KAFKA-4384) ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

[ https://issues.apache.org/jira/browse/KAFKA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15641511#comment-15641511 ] Jiangjie Qin commented on KAFKA-4384: - [~junhe] Thanks for reporting the issue. Yes, t