Kafka Streaming - Window expiration

Hello Experts, As per documentation in kafka docs - *Windowing* is a common prerequisite for stateful transformations which group records in a stream, for example, by their timestamps. A local state store is usually needed for a windowing operation to store recently received records based on the w

[jira] [Commented] (KAFKA-3775) Throttle maximum number of tasks assigned to a single KafkaStreams

[ https://issues.apache.org/jira/browse/KAFKA-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326937#comment-15326937 ] James Cheng commented on KAFKA-3775: [~kawamuray], for the network traffic, could you

Re: Kafka Streaming - Window expiration

Hi Pari, Try the .until method like this: > (TimeWindows) TimeWindows.of("tumbling-window-example", > windowSizeMs).until(60 * 1000L) Thanks Eno > On 13 Jun 2016, at 08:31, Pariksheet Barapatre wrote: > > Hello Experts, > > As per documentation in kafka docs - > *Windowing* is a common pre

[jira] [Created] (KAFKA-3829) Warn that kafka-connect group.id must not conflice with connector names

Barry Kaplan created KAFKA-3829: --- Summary: Warn that kafka-connect group.id must not conflice with connector names Key: KAFKA-3829 URL: https://issues.apache.org/jira/browse/KAFKA-3829 Project: Kafka

[jira] [Commented] (KAFKA-3828) Consumer thread stalls after consumer re balance for some partition

[ https://issues.apache.org/jira/browse/KAFKA-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327066#comment-15327066 ] Ismael Juma commented on KAFKA-3828: Can you please include the consumer and broker ve

Re: Kafka Streaming - Window expiration

Many Thanks Eno. I will try .until method. Cheers Pari On 13 June 2016 at 14:10, Eno Thereska wrote: > Hi Pari, > > Try the .until method like this: > > > (TimeWindows) TimeWindows.of("tumbling-window-example", > windowSizeMs).until(60 * 1000L) > > Thanks > Eno > > > > On 13 Jun 2016, at 08:31

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Thank you all for the feedback. Closing this voting thread to continue discussions on the updated KIP in the discuss thread. On Sat, Jun 11, 2016 at 7:45 AM, Gwen Shapira wrote: > I'd also like to see clarification regarding the ZK structures. > Currently they appear as if user-quotas and client

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[ https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327225#comment-15327225 ] Pablo Casares commented on KAFKA-3209: -- Hey, I'm interested in produce keys based on

Re: [VOTE] KIP-55: Secure quotas for authenticated users

I have updated KIP-55 to reflect the changes from the discussions in the voting thread ( https://www.mail-archive.com/dev@kafka.apache.org/msg51610.html). Jun/Gwen, Existing client-id quotas will be used as default client-id quotas for users when no user quotas are configured - i.e., default user

[jira] [Created] (KAFKA-3830) getTGT() debug logging exposes confidential information

Ismael Juma created KAFKA-3830: -- Summary: getTGT() debug logging exposes confidential information Key: KAFKA-3830 URL: https://issues.apache.org/jira/browse/KAFKA-3830 Project: Kafka Issue Type:

[jira] [Commented] (KAFKA-3775) Throttle maximum number of tasks assigned to a single KafkaStreams

[ https://issues.apache.org/jira/browse/KAFKA-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327326#comment-15327326 ] Matthias J. Sax commented on KAFKA-3775: [~tenggyut] [~wushujames] Kafka Streams a

[jira] [Commented] (KAFKA-3576) Unify KStream and KTable API

[ https://issues.apache.org/jira/browse/KAFKA-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327335#comment-15327335 ] Eno Thereska commented on KAFKA-3576: - I understand the goal, but why is the first opt

[GitHub] kafka pull request #1498: KAFKA-3830; getTGT() debug logging exposes confide...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/1498 KAFKA-3830; getTGT() debug logging exposes confidential information Only log the client and server principals, which is what ZooKeeper does after ZOOKEEPER-2405. You can merge this pull request into

[jira] [Commented] (KAFKA-3830) getTGT() debug logging exposes confidential information

[ https://issues.apache.org/jira/browse/KAFKA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327353#comment-15327353 ] ASF GitHub Bot commented on KAFKA-3830: --- GitHub user ijuma opened a pull request:

[jira] [Updated] (KAFKA-3830) getTGT() debug logging exposes confidential information

[ https://issues.apache.org/jira/browse/KAFKA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3830: --- Status: Patch Available (was: Open) > getTGT() debug logging exposes confidential information > -

[jira] [Commented] (KAFKA-2394) Use RollingFileAppender by default in log4j.properties

[ https://issues.apache.org/jira/browse/KAFKA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327390#comment-15327390 ] Dustin Cote commented on KAFKA-2394: [~ewencp] yeah I think that's for the best. I wo

[jira] [Commented] (KAFKA-3828) Consumer thread stalls after consumer re balance for some partition

[ https://issues.apache.org/jira/browse/KAFKA-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327462#comment-15327462 ] Joseph Aliase commented on KAFKA-3828: -- Broker version: 0.10.0.0 and 0.9.0.1 Consume

[jira] [Commented] (KAFKA-3828) Consumer thread stalls after consumer re balance for some partition

[ https://issues.apache.org/jira/browse/KAFKA-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327472#comment-15327472 ] Ismael Juma commented on KAFKA-3828: Thanks. There are known consumer bugs in 0.9.0.0

[jira] [Commented] (KAFKA-3828) Consumer thread stalls after consumer re balance for some partition

[ https://issues.apache.org/jira/browse/KAFKA-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327488#comment-15327488 ] Joseph Aliase commented on KAFKA-3828: -- I assume you are asking to upgrade to Consume

[jira] [Commented] (KAFKA-3828) Consumer thread stalls after consumer re balance for some partition

[ https://issues.apache.org/jira/browse/KAFKA-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327509#comment-15327509 ] Ismael Juma commented on KAFKA-3828: Yes, upgrading the consumer to 0.9.0.1 or 0.10.0.

[jira] [Commented] (KAFKA-3828) Consumer thread stalls after consumer re balance for some partition

[ https://issues.apache.org/jira/browse/KAFKA-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327513#comment-15327513 ] Joseph Aliase commented on KAFKA-3828: -- Sure will test and let you know > Consumer t

[jira] [Updated] (KAFKA-3818) Change Mirror Maker default assignment strategy to round robin

[ https://issues.apache.org/jira/browse/KAFKA-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-3818: --- Status: Open (was: Patch Available) > Change Mirror Maker default assignment strategy to roun

[jira] [Commented] (KAFKA-1464) Add a throttling option to the Kafka replication tool

[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327561#comment-15327561 ] Ralph Weires commented on KAFKA-1464: - We have similar problems as described by Jason

[jira] [Commented] (KAFKA-3818) Change Mirror Maker default assignment strategy to round robin

[ https://issues.apache.org/jira/browse/KAFKA-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327629#comment-15327629 ] Vahid Hashemian commented on KAFKA-3818: [~ewencp] [~hachikuji] I suggest we keep

[jira] [Created] (KAFKA-3831) Preparation for updating the default partition assignment strategy of Mirror Maker to round robin

Vahid Hashemian created KAFKA-3831: -- Summary: Preparation for updating the default partition assignment strategy of Mirror Maker to round robin Key: KAFKA-3831 URL: https://issues.apache.org/jira/browse/KAFKA-383

[jira] [Created] (KAFKA-3832) Kafka Connect's JSON Converter never outputs a null value

Randall Hauch created KAFKA-3832: Summary: Kafka Connect's JSON Converter never outputs a null value Key: KAFKA-3832 URL: https://issues.apache.org/jira/browse/KAFKA-3832 Project: Kafka Issue

[jira] [Created] (KAFKA-3833) Feature Request: allow Kafka clients to retrieve Kafka broker versions

Hari Sekhon created KAFKA-3833: -- Summary: Feature Request: allow Kafka clients to retrieve Kafka broker versions Key: KAFKA-3833 URL: https://issues.apache.org/jira/browse/KAFKA-3833 Project: Kafka

[jira] [Updated] (KAFKA-3833) Feature Request: allow Kafka clients to retrieve Kafka broker versions

[ https://issues.apache.org/jira/browse/KAFKA-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated KAFKA-3833: --- Description: Feature request for the ability of Kafka clients to query the Kafka brokers for their ve

[jira] [Created] (KAFKA-3834) Consumer should not block in poll on coordinator discovery

Jason Gustafson created KAFKA-3834: -- Summary: Consumer should not block in poll on coordinator discovery Key: KAFKA-3834 URL: https://issues.apache.org/jira/browse/KAFKA-3834 Project: Kafka

[GitHub] kafka pull request #1499: KAFKA-3831: Prepare for updating Mirror Maker's de...

GitHub user vahidhashemian opened a pull request: https://github.com/apache/kafka/pull/1499 KAFKA-3831: Prepare for updating Mirror Maker's default partition assignment strategy to round robin This patch adds proper warning message and necessary doc updates for updating the default

[jira] [Commented] (KAFKA-3831) Preparation for updating the default partition assignment strategy of Mirror Maker to round robin

[ https://issues.apache.org/jira/browse/KAFKA-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327770#comment-15327770 ] ASF GitHub Bot commented on KAFKA-3831: --- GitHub user vahidhashemian opened a pull re

[jira] [Updated] (KAFKA-3831) Preparation for updating the default partition assignment strategy of Mirror Maker to round robin

[ https://issues.apache.org/jira/browse/KAFKA-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-3831: --- Status: Patch Available (was: Open) > Preparation for updating the default partition assignme

RE: Migrating from 07.1 .100

Hi, I currently embed kafka 0.7.1 in my java process. To support SSL, we have decided to upgrade Kafka to 0.10.0. After upgrade, I am seeing following error during kafka startup. java.lang.NullPointerException at kafka.utils.Throttler.(Throttler.scala:45) at kafka.

[GitHub] kafka pull request #1500: MINOR: Fix javadoc typos in ConsumerRebalanceListe...

GitHub user vahidhashemian opened a pull request: https://github.com/apache/kafka/pull/1500 MINOR: Fix javadoc typos in ConsumerRebalanceListener You can merge this pull request into a Git repository by running: $ git pull https://github.com/vahidhashemian/kafka typo07/fix_ja

[jira] [Commented] (KAFKA-3576) Unify KStream and KTable API

[ https://issues.apache.org/jira/browse/KAFKA-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328242#comment-15328242 ] Matthias J. Sax commented on KAFKA-3576: I see two points: (1) this is similar to

[GitHub] kafka pull request #1465: KAFKA-3786: Let ConfigDef filter property key valu...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1465 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Resolved] (KAFKA-3786) Avoid unused property from parent configs causing WARN entries

[ https://issues.apache.org/jira/browse/KAFKA-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-3786. -- Resolution: Fixed Fix Version/s: 0.10.1.0 Issue resolved by pull request 1465 [https://gi

[jira] [Commented] (KAFKA-3786) Avoid unused property from parent configs causing WARN entries

[ https://issues.apache.org/jira/browse/KAFKA-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328287#comment-15328287 ] ASF GitHub Bot commented on KAFKA-3786: --- Github user asfgit closed the pull request

Re: [VOTE] KIP-33 - Add a time based log index

Thanks Jiangjie, I see the need for sensitive data purging, the above proposed change LGTM. One minor concern is that a wrongly marked timestamp on the first record could cause the segment to roll much later / earlier, though it may be rare. Guozhang On Fri, Jun 10, 2016 at 10:07 AM, Becket Qin

Re: [DISCUSS] KIP-63: Unify store and downstream caching in streams

I am just catching up on this thread. From my point of view, easy tuning for the user is the most important thing, because Kafka Streams is a library. Thus, a global cache size parameter should be the best. About dividing the memory vs a single global cache. I would argue that in the first place

[GitHub] kafka pull request #1501: MINOR: Expose window store sequence number

GitHub user theduderog opened a pull request: https://github.com/apache/kafka/pull/1501 MINOR: Expose window store sequence number @guozhangwang @mjsax @enothereska Currently, Kafka Streams does not have a util to get access to the sequence number added to the key of windo

Jenkins build is back to normal : kafka-trunk-jdk8 #693

See

[GitHub] kafka pull request #1502: MINOR: Clarify the at least once delivery with aut...

GitHub user Ishiihara opened a pull request: https://github.com/apache/kafka/pull/1502 MINOR: Clarify the at least once delivery with auto commit enabled. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Ishiihara/kafka fetcher-co

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[ https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328647#comment-15328647 ] Ewen Cheslack-Postava commented on KAFKA-3209: -- [~pablocasares] I don't think

[jira] [Commented] (KAFKA-2857) ConsumerGroupCommand throws GroupCoordinatorNotAvailableException when describing a non-existent group before the offset topic is created

[ https://issues.apache.org/jira/browse/KAFKA-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328708#comment-15328708 ] Vahid Hashemian commented on KAFKA-2857: [~hachikuji] Could you also clarify what

Re: [DISCUSS] KIP-62: Allow consumer to send heartbeats from a background thread

Hey Becket, Sorry for the late response. I agree there's a little more to think through on the implementation. The offset commit is the tricky one since we could execute a user-provided callback. I'm thinking if there is an inflight request to the coordinator, we may simply skip the heartbeat and

Re: [DISCUSS] KIP-4 Create Topic Schema

Hi Jay, Good point one of the main benefits of the create topic api is removing the server side auto create. The work is noted in the Follow Up Changes

Re: [DISCUSS] KIP-63: Unify store and downstream caching in streams

Although this KIP is not mainly for memory management of Kafka Streams, since it touches on quite some part of it I think it is good to first think of what we would REALLY want as an end goal for memory usage in order to make sure that whatever we proposed in this KIP aligns with that long-term pla

Re: [DISCUSS] KIP-4 Create Topic Schema

Thanks for the review Gwen. 1. The replica assignment protocol takes [replicas], there is the > implicit assumption that the first replica is the leader. This matches > current behavior elsewhere, but lets document it explicitly. I added this to the wiki and will update the protocol doc string i

Re: [DISCUSS] KIP-4 Create Topic Schema

Thanks for the review Jun. You probably want to make it clearer if timeout > 0, what waiting for topic > metadata is "complete" means. In the first implementation, it really means > that the topic metadata is propagated to the controller's metadata cache. I updated the wiki to be more descriptiv

[jira] [Commented] (KAFKA-3686) Kafka producer is not fault tolerant

[ https://issues.apache.org/jira/browse/KAFKA-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328781#comment-15328781 ] radha commented on KAFKA-3686: -- We have a similar issue and even with a very very large retry

[jira] [Commented] (KAFKA-2857) ConsumerGroupCommand throws GroupCoordinatorNotAvailableException when describing a non-existent group before the offset topic is created

[ https://issues.apache.org/jira/browse/KAFKA-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328866#comment-15328866 ] Jason Gustafson commented on KAFKA-2857: [~vahid] That's a good question. Maybe we

Re: Migrating from 07.1 .100

There has been a lot of changes, and it can also be quit a challenge to get the SSL working. And with only a null-pointer, there is little to go on. I would first focus on getting it to work with 0.10 (if possible also on a clustered test setup), and if it all works, try to configure the ssl. On M