[GitHub] kafka pull request #1508: MINOR: Fix quota violation exception message

2016-06-16 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1508


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] kafka pull request #1512: KAFKA-3838 zkClient and Zookeeper version bump

2016-06-16 Thread mangas
GitHub user mangas opened a pull request:

https://github.com/apache/kafka/pull/1512

KAFKA-3838 zkClient and Zookeeper version bump



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mangas/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1512


commit 968e2548c561392cc748e94749b46a2d2cdfe8fd
Author: Filipe Azevedo 
Date:   2016-06-16T07:57:02Z

KAFKA-3838 zkClient and Zookeeper version bump




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-3838) Bump zkclient and Zookeeper versions

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1511#comment-1511
 ] 

ASF GitHub Bot commented on KAFKA-3838:
---

GitHub user mangas opened a pull request:

https://github.com/apache/kafka/pull/1512

KAFKA-3838 zkClient and Zookeeper version bump



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mangas/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1512


commit 968e2548c561392cc748e94749b46a2d2cdfe8fd
Author: Filipe Azevedo 
Date:   2016-06-16T07:57:02Z

KAFKA-3838 zkClient and Zookeeper version bump




> Bump zkclient and Zookeeper versions
> 
>
> Key: KAFKA-3838
> URL: https://issues.apache.org/jira/browse/KAFKA-3838
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: FIlipe Azevedo
>
> Zookeeper 3.4.8 has some improvements, specifically it handles DNS 
> Re-resolution when a connection to zookeeper fails. This potentially allows 
> Round Robin DNS without the need to hardcode the IP Addresses in the config. 
> http://zookeeper.apache.org/doc/r3.4.8/releasenotes.html
> ZkClient has a new 0.9 release which uses zookeeper 3.4.8 which is already 
> marked as stable.
> Tests are passing.
> Here is the PR: https://github.com/apache/kafka/pull/1504



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #1504: KAFKA-3838 zkClient and Zookeeper version bump

2016-06-16 Thread mangas
Github user mangas closed the pull request at:

https://github.com/apache/kafka/pull/1504


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-3838) Bump zkclient and Zookeeper versions

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1513#comment-1513
 ] 

ASF GitHub Bot commented on KAFKA-3838:
---

Github user mangas closed the pull request at:

https://github.com/apache/kafka/pull/1504


> Bump zkclient and Zookeeper versions
> 
>
> Key: KAFKA-3838
> URL: https://issues.apache.org/jira/browse/KAFKA-3838
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: FIlipe Azevedo
>
> Zookeeper 3.4.8 has some improvements, specifically it handles DNS 
> Re-resolution when a connection to zookeeper fails. This potentially allows 
> Round Robin DNS without the need to hardcode the IP Addresses in the config. 
> http://zookeeper.apache.org/doc/r3.4.8/releasenotes.html
> ZkClient has a new 0.9 release which uses zookeeper 3.4.8 which is already 
> marked as stable.
> Tests are passing.
> Here is the PR: https://github.com/apache/kafka/pull/1504



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: FYI - Apache ZooKeeper Backup, a Treatise

2016-06-16 Thread Flavio Junqueira
Great write-up, Jordan, thanks!

Whether to backup zk data or not is possibly an open topic for this community, 
even though we have discussed it at times. My sense has been that precisely 
because of the issues you mention in your post, it is typically best to have a 
way to recreate its data upon a disaster rather than backup the data. I think 
there could be three general scenarios in which folks would prefer to backup 
data, but you correct me if these aren't accurate:

- The data in zk isn't elsewhere, so it can't be recreated: zk isn't a regular 
database, so I'd think it is best not to store data and focus on cluster data 
or metadata.
- There is a just a lot of data and I'd rather have a shorter time to recover: 
zk in general shouldn't have that much data in db, but let's go with the 
assumption that for the requirements of the application it is a lot. For such a 
case, it probably depends on whether your application can efficiently and 
effectively recover from a backup. Basically, as pointed out in the post, the 
data could be inconsistent and cause trouble if you don't think about the 
corner cases. 
- The code to recreate the zk metadata for my application is super complex: if 
you decide to code against zk, it is good to think whether reconstructing in 
the case of a disaster is doable and if it is design and implement to 
reconstruct the state upon a disaster.

Also, we typically provision enough replicas, often replicating across data 
centers, to make sure that the data isn't all gone. Having more replicas does 
not rule out completely the possibility of a disaster, but in such rare cases 
we resort to the expensive path.

I personally have never worked with an application that was taking backups of 
zk data in prod, so I'm really interested in what others think. 

-Flavio


> On 16 Jun 2016, at 00:43, Jordan Zimmerman  wrote:
> 
> FYI - I wrote a blog about backing up ZooKeeper:
> 
> https://www.elastic.co/blog/zookeeper-backup-a-treatise
> 
> -Jordan



Build failed in Jenkins: kafka-trunk-jdk8 #700

2016-06-16 Thread Apache Jenkins Server
See 

Changes:

[ismael] MINOR: Fix quota violation exception message

--
[...truncated 1883 lines...]
kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
STARTED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogApendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogApendTime PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate PASSED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions STARTED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testAsyncCommit STARTED

kafka.api.PlaintextConsumerTest > testAsyncCommit PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnStopPolling 
STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnStopPolling 
PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic PASSED

kafka.api.PlaintextConsumerTest > testSeek STARTED

kafka.api.PlaintextConsumerTest > testSeek PASSED

kafka.api.PlaintextConsumerTest > testPositionAndCommit STARTED

kafka.api.PlaintextConsumerTest > testPositionAndCommit PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnClose STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnClose PASSED

kafka.api.PlaintextConsumerTest > testFetchRecordTooLarge STARTED

kafka.api.PlaintextConsumerTest > testFetchRecordTooLarge PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerDefaultAssignment STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerDefaultAssignment PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnClose STARTED

kafka.api.PlaintextConsumerTest > testAutoCommitOnClose PASSED

kafka.api.PlaintextConsumerTest > testExpandingTopicSubscriptions STARTED

kafka.api.PlaintextConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testInterceptors STARTED

kafka.api.PlaintextConsumerTest > testInterceptors PASSED

kafka.api.PlaintextConsumerTest > testPatternUnsubscription STARTED

kafka.api.PlaintextConsumerTest > testPatternUnsubscription PASSED

kafka.api.PlaintextConsumerTest > testGroupConsumption STARTED

kafka.api.PlaintextConsumerTest > testGroupConsumption PASSED

kafka.api.PlaintextConsumerTest > testPartitionsFor STARTED

kafka.api.PlaintextConsumerTest > testPartitionsFor PASSED

kafka.api.PlaintextConsumerTest > testInterceptorsWithWrongKeyValue STARTED

kafka.api.PlaintextConsumerTest > testInterceptorsWithWrongKeyValue PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerRoundRobinAssignment STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerRoundRobinAssignment PASSED

kafka.api.PlaintextConsumerTest > testPartitionPauseAndResume STARTED

kafka.api.PlaintextConsumerTest > testPartitionPauseAndResume PASSED

kafka.api.PlaintextConsumerTest > testConsumeMessagesWithLogAppendTime STARTED

kafka.api.PlaintextConsumerTest > testConsumeMessagesWithLogAppendTime PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnCloseAfterWakeup STARTED

kafka.api.PlaintextConsumerTest > testAutoCommitOnCloseAfterWakeup PASSED

kafka.api.PlaintextConsumerTest > testMaxPollRecords STARTED

kafka.api.PlaintextConsumerTest > testMaxPollRecords PASSED

kafka.api.PlaintextConsumerTest > testAutoOffsetReset STARTED

kafka.api.PlaintextConsumerTest > testAutoOffsetReset PASSED

kafka.api.PlaintextConsumerTest > testFetchInvalidOffset STARTED

kafka.api.PlaintextConsumerTest > testFetchInvalidOffset PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitIntercept STARTED

kafka.api.PlaintextConsumerTest > testAutoCommitIntercept PASSED

kafka.api.PlaintextConsumerTest > testCommitMetadata STARTED

kafka.api.PlaintextConsumerTest > testCommitMetadata PASSED

kafka.api.PlaintextConsumerTest > testRoundRobinAssignment STARTED

kafka.api.PlaintextConsumerTest > testRoundRobinAssignment PASSED

kafka.api.PlaintextConsumerTest > testPatternSubscription STARTED

kafka.api.PlaintextConsumerTest > testPatternSubscription PASSED

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance STARTED

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance PASSED

kafka.api.PlaintextConsumerTest > testUnsubscribeTopic STAR

Re: Embedding zookeeper and kafka in java process.

2016-06-16 Thread Ismael Juma
Hi Subhash,

This would happen if `null` is passed as the `threadNamePrefix` argument
when instantiating `KafkaServer`:

class KafkaServer(val config: KafkaConfig, time: Time = SystemTime,
threadNamePrefix: Option[String] = None) extends Logging with
KafkaMetricsGroup

How are you starting Kafka in your Java process?

Ismael

On Thu, Jun 16, 2016 at 1:26 AM, Subhash Agrawal 
wrote:

> Thanks for quick response.
> I started zookeeper via zookeeper-server-start.bat and started kafka via
> my java process and I saw same error.
> But if I start zookeeper via java process and start kafka via
> kafka-server-start.bat, t works fine.
> It means it is not caused due to both getting started in same process. It
> must be some kafka specific issue.
>
> Subhash Agrawal
>
> -Original Message-
> From: Guozhang Wang [mailto:wangg...@gmail.com]
> Sent: Wednesday, June 15, 2016 3:42 PM
> To: dev@kafka.apache.org
> Subject: Re: Embedding zookeeper and kafka in java process.
>
> It seems "scala.MatchError: null" are not related to the settings that ZK
> and Kafka is embedded in the same process, and the only case that I can
> think of related is this: https://issues.apache.org/jira/browse/KAFKA-940.
>
> Could you clarify if you start these two services on two processes, the
> issue goes away?
>
> Guozhang
>
> On Wed, Jun 15, 2016 at 1:56 PM, Subhash Agrawal 
> wrote:
>
> > Hi All,
> > I am embedding Kafka 0.10.0 and corresponding zookeeper in java process.
> > In this process, I start zookeeper first and then wait for 10 seconds and
> > then start kafka. These are all running in the same process. Toward the
> > end of kafka startup, I see following exception. It seems zookeeper is
> not
> > able
> > to add the newly created kafka instance. Have you seen this error
> > earlier?  I have only single node kafka.
> >
> > Let me know if you have any suggestions. I will really appreciate any
> help
> > on this.
> >
> > Thanks
> > Subhash Agrawal.
> >
> > [2016-06-15 13:39:39,616] INFO [Logserver_Starter] Registered broker 0 at
> > path /brokers/ids/0 with addresses: PLAINTEXT ->
> > EndPoint(localhost,8392,PLAINTEXT) (kafka.utils.ZkUtils)
> > [2016-06-15 13:39:39,617] WARN [Logserver_Starter] No meta.properties
> file
> > under dir C:\development \newkafka-logs\meta.properties
> > (kafka.server.BrokerMetadataCheckpoint)
> > [2016-06-15 13:39:39,627] INFO [ZkClient-EventThread-24-localhost:2181]
> > New leader is 0
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > [2016-06-15 13:39:39,629] INFO [ZkClient-EventThread-24-localhost:2181]
> > [BrokerChangeListener on Controller 0]: Broker change listener fired for
> > path /brokers/ids with children 0
> > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > [2016-06-15 13:39:39,638] INFO [Logserver_Starter] Kafka version :
> > 0.10.0.0 (org.apache.kafka.common.utils.AppInfoParser)
> > [2016-06-15 13:39:39,638] INFO [Logserver_Starter] Kafka commitId :
> > b8642491e78c5a13 (org.apache.kafka.common.utils.AppInfoParser)
> > [2016-06-15 13:39:39,639] INFO [Logserver_Starter] [Kafka Server 0],
> > started (kafka.server.KafkaServer)
> > [2016-06-15 13:39:39,806] INFO [ZkClient-EventThread-24-localhost:2181]
> > [BrokerChangeListener on Controller 0]: Newly added brokers: 0, deleted
> > brokers: , all live brokers: 0
> > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > [2016-06-15 13:39:39,808] DEBUG [ZkClient-EventThread-24-localhost:2181]
> > [Channel manager on controller 0]: Controller 0 trying to connect to
> broker
> > 0 (kafka.controller.ControllerChannelManager)
> > [2016-06-15 13:39:39,818] ERROR [ZkClient-EventThread-24-localhost:2181]
> > [BrokerChangeListener on Controller 0]: Error while handling broker
> changes
> > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > scala.MatchError: null
> > at
> >
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:122)
> > at
> >
> kafka.controller.ControllerChannelManager.addBroker(ControllerChannelManager.scala:74)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(ReplicaStateMachine.scala:372)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(ReplicaStateMachine.scala:372)
> > at
> > scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:372)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
> > at
> >

Re: [VOTE] KIP-55: Secure quotas for authenticated users

2016-06-16 Thread Rajini Sivaram
Jun,

Thank you for the review. I have updated the KIP:


   1. Added an overview section. Slightly reworded since it is better to
   treat user and client-id as different levels rather than the same level.
   2. Yes, it is neater to store quota for each entity in a different path
   in Zookeeper. I put clients under users rather than the other way round
   since that reflects the hierarchy and also keeps a user's quotas together
   under a single sub-tree. I had initially used a single node to keep quotas
   and sub-quotas of a user together so that updates are atomic since changes
   to sub-quotas also affect remainder quotas for other clients. But I
   imagine, updates to configs are rare and it is not a big issue.
   3. I haven't modified the JSON for configuration change notifications.
   The entity_name can now be a subpath that has both user and client. Have
   added an example to the KIP. The downside of keeping clients under users in
   ZK in 2) is that the change notification for sub-quota has entity_type
   "users". I could extend the JSON to include client separately, but since
   changes to a client sub-quota does impact other clients of the user as well
   due to change in remainder quota, it may be ok as it is. Do let me know if
   it looks confusing in the example.
   4. Agree, updated.


On Wed, Jun 15, 2016 at 10:27 PM, Jun Rao  wrote:

> Hi, Rajini,
>
> Thanks for the updated wiki. Overall, I like the new approach. It covers
> the common use cases now, is extensible, and is backward compatible. A few
> comments below.
>
> 1. It would be useful to describe a bit at the high level, how the new
> approach works. I think this can be summarized as follows. Quotas can be
> set at user, client-id or  levels. For a given client
> connection, the most specific quota matching the connection will be
> applied. For example, if both a  and a  quota match
> a connection, the  quota will be used. If more than 1
> quota at the same level (e.g., a quota on a user and another quota on a
> client-id) match the connection, the user level quota will be used, i.e.,
> user quota takes precedence over client-id quota.
>
> 2. For the ZK data structure, would it be better to store 
> quota as the following. Then the format of the value in each path is the
> same. The wiki also mentions that we want to include the original user name
> in the ZK value. Could you describe that in an example?
> // Zookeeper persistence path /clients/clientA/users/
> {
> "version":1,
> "config": {
> "producer_byte_rate":"10",
> "consumer_byte_rate":"20"
> }
> }
>
> 3. Could you document the format change of the ZK value in
> /config/changes/config_change_xxx, if any?
>
> 4. For the config command, could we specify the sub-quota like the
> following, instead of in the config value? This seems more intuitive.
>
> bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config
> 'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-name
> clientA --entity-type clients --entity-name user2 --entity-type users
>
> Thanks,
>
> Jun
>
> On Wed, Jun 15, 2016 at 10:35 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Harsha,
> >
> > The sample configuration under
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users#KIP-55:SecureQuotasforAuthenticatedUsers-QuotaConfiguration
> > shows
> > the Zookeeper data for different scenarios.
> >
> >- *user1* (/users/user1 in ZK) has only user-level quotas
> >- *user2* (/users/user2 in ZK) defines user-level quotas and
> sub-quotas
> >for clients *clientA* and *clientB*. Other client-ids of *user2* share
> >the remaining quota of *user2*.
> >
> >
> > On Wed, Jun 15, 2016 at 5:30 PM, Harsha  wrote:
> >
> > > Rajini,
> > >   How does sub-quotas works in case of authenticated users.
> > >   Where are we maintaining the relation between users and their
> > >   client Ids. Can you add an example of zk data under /users.
> > > Thanks,
> > > Harsha
> > >
> > > On Mon, Jun 13, 2016, at 05:01 AM, Rajini Sivaram wrote:
> > > > I have updated KIP-55 to reflect the changes from the discussions in
> > the
> > > > voting thread (
> > > > https://www.mail-archive.com/dev@kafka.apache.org/msg51610.html).
> > > >
> > > > Jun/Gwen,
> > > >
> > > > Existing client-id quotas will be used as default client-id quotas
> for
> > > > users when no user quotas are configured - i.e., default user quota
> is
> > > > unlimited and no user-specific quota override is specified. This
> > enables
> > > > user rate limits to be configured for ANONYMOUS if required in a
> > cluster
> > > > that has both PLAINTEXT and SSL/SASL. By default, without any user
> rate
> > > > limits set, rate limits for client-ids will apply, retaining the
> > current
> > > > client-id quota configuration for single-user clusters.
> > > >
> > > > Zookeeper will have two paths /clients with client-id quotas that
> apply

Re: [VOTE] KIP-55: Secure quotas for authenticated users

2016-06-16 Thread Rajini Sivaram
Jun,

Actually, with quotas stored in different nodes in ZK, it is better to
store remainder quota rather than total quota under /users/ so that
quota calculations are not dependent on the order of notifications. I have
updated the KIP to reflect that. So the quotas in ZK now always reflect the
quota applied to a group of client connections and use the same format as
client-id quotas. But it is not hierarchical, making the configuration
simpler.

On Thu, Jun 16, 2016 at 11:49 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Jun,
>
> Thank you for the review. I have updated the KIP:
>
>
>1. Added an overview section. Slightly reworded since it is better to
>treat user and client-id as different levels rather than the same level.
>2. Yes, it is neater to store quota for each entity in a different
>path in Zookeeper. I put clients under users rather than the other way
>round since that reflects the hierarchy and also keeps a user's quotas
>together under a single sub-tree. I had initially used a single node to
>keep quotas and sub-quotas of a user together so that updates are atomic
>since changes to sub-quotas also affect remainder quotas for other clients.
>But I imagine, updates to configs are rare and it is not a big issue.
>3. I haven't modified the JSON for configuration change notifications.
>The entity_name can now be a subpath that has both user and client. Have
>added an example to the KIP. The downside of keeping clients under users in
>ZK in 2) is that the change notification for sub-quota has entity_type
>"users". I could extend the JSON to include client separately, but since
>changes to a client sub-quota does impact other clients of the user as well
>due to change in remainder quota, it may be ok as it is. Do let me know if
>it looks confusing in the example.
>4. Agree, updated.
>
>
> On Wed, Jun 15, 2016 at 10:27 PM, Jun Rao  wrote:
>
>> Hi, Rajini,
>>
>> Thanks for the updated wiki. Overall, I like the new approach. It covers
>> the common use cases now, is extensible, and is backward compatible. A few
>> comments below.
>>
>> 1. It would be useful to describe a bit at the high level, how the new
>> approach works. I think this can be summarized as follows. Quotas can be
>> set at user, client-id or  levels. For a given client
>> connection, the most specific quota matching the connection will be
>> applied. For example, if both a  and a  quota match
>> a connection, the  quota will be used. If more than 1
>> quota at the same level (e.g., a quota on a user and another quota on a
>> client-id) match the connection, the user level quota will be used, i.e.,
>> user quota takes precedence over client-id quota.
>>
>> 2. For the ZK data structure, would it be better to store > client-id>
>> quota as the following. Then the format of the value in each path is the
>> same. The wiki also mentions that we want to include the original user
>> name
>> in the ZK value. Could you describe that in an example?
>> // Zookeeper persistence path /clients/clientA/users/
>> {
>> "version":1,
>> "config": {
>> "producer_byte_rate":"10",
>> "consumer_byte_rate":"20"
>> }
>> }
>>
>> 3. Could you document the format change of the ZK value in
>> /config/changes/config_change_xxx, if any?
>>
>> 4. For the config command, could we specify the sub-quota like the
>> following, instead of in the config value? This seems more intuitive.
>>
>> bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config
>> 'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-name
>> clientA --entity-type clients --entity-name user2 --entity-type users
>>
>> Thanks,
>>
>> Jun
>>
>> On Wed, Jun 15, 2016 at 10:35 AM, Rajini Sivaram <
>> rajinisiva...@googlemail.com> wrote:
>>
>> > Harsha,
>> >
>> > The sample configuration under
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users#KIP-55:SecureQuotasforAuthenticatedUsers-QuotaConfiguration
>> > shows
>> > the Zookeeper data for different scenarios.
>> >
>> >- *user1* (/users/user1 in ZK) has only user-level quotas
>> >- *user2* (/users/user2 in ZK) defines user-level quotas and
>> sub-quotas
>> >for clients *clientA* and *clientB*. Other client-ids of *user2*
>> share
>> >the remaining quota of *user2*.
>> >
>> >
>> > On Wed, Jun 15, 2016 at 5:30 PM, Harsha  wrote:
>> >
>> > > Rajini,
>> > >   How does sub-quotas works in case of authenticated users.
>> > >   Where are we maintaining the relation between users and
>> their
>> > >   client Ids. Can you add an example of zk data under /users.
>> > > Thanks,
>> > > Harsha
>> > >
>> > > On Mon, Jun 13, 2016, at 05:01 AM, Rajini Sivaram wrote:
>> > > > I have updated KIP-55 to reflect the changes from the discussions in
>> > the
>> > > > voting thread (
>> > > > https://www.mail-archive.com/dev@kafka.apache.org/msg51610.html).
>>

[GitHub] kafka pull request #1507: KAFKA-3840; Allow clients default OS buffer sizes

2016-06-16 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1507


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-3840) OS auto tuning for socket buffer size in clients not allowed through configuration

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333684#comment-15333684
 ] 

ASF GitHub Bot commented on KAFKA-3840:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1507


> OS auto tuning for socket buffer size in clients not allowed through 
> configuration
> --
>
> Key: KAFKA-3840
> URL: https://issues.apache.org/jira/browse/KAFKA-3840
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sébastien Launay
>Priority: Minor
>
> KAFKA-724 added the following documentation for clients in order to provide 
> OS auto tuning of socket buffer sizes:
> {noformat}
> diff --git 
> a/clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> b/clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java
> index 750b8a1..3327815 100644
> --- a/clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java
> +++ b/clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java
> @@ -43,10 +43,10 @@
>  public static final String METADATA_MAX_AGE_DOC = "The period of time in 
> milliseconds after which we force a refresh of metadata even if we haven't 
> seen any partition leadership changes to proactively discover any new brokers 
> or partitions.";
>  
>  public static final String SEND_BUFFER_CONFIG = "send.buffer.bytes";
> -public static final String SEND_BUFFER_DOC = "The size of the TCP send 
> buffer (SO_SNDBUF) to use when sending data.";
> +public static final String SEND_BUFFER_DOC = "The size of the TCP send 
> buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS 
> default will be used.";
>  
>  public static final String RECEIVE_BUFFER_CONFIG = 
> "receive.buffer.bytes";
> -public static final String RECEIVE_BUFFER_DOC = "The size of the TCP 
> receive buffer (SO_RCVBUF) to use when reading data.";
> +public static final String RECEIVE_BUFFER_DOC = "The size of the TCP 
> receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the 
> OS default will be used.";
>  
>  public static final String CLIENT_ID_CONFIG = "client.id";
>  public static final String CLIENT_ID_DOC = "An id string to pass to the 
> server when making requests. The purpose of this is to be able to track the 
> source of requests beyond just ip/port by allowing a logical application name 
> to be included in server-side request logging.";
> {noformat}
> Unfortunately, the value {{-1}} is not allowed for clients configuration and 
> the following exception is thrown:
> {noformat}
> Exception in thread "main" org.apache.kafka.common.config.ConfigException: 
> Invalid value -1 for configuration receive.buffer.bytes: Value must be at 
> least 0
> at 
> org.apache.kafka.common.config.ConfigDef$Range.ensureValid(ConfigDef.java:308)
> at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:153)
> at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:49)
> at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:56)
> at 
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:315)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:156)
> ...
> {noformat}
> This is caused by an {{atLeast(0)}} validation in {{ProducerConfig.java}} and 
> {{ConsumerConfig.java}}::
> {noformat}
> ...
>   .define(SEND_BUFFER_CONFIG, Type.INT, 128 * 1024, atLeast(0), 
> Importance.MEDIUM, CommonClientConfigs.SEND_BUFFER_DOC)
>   .define(RECEIVE_BUFFER_CONFIG, Type.INT, 32 * 1024, atLeast(0), 
> Importance.MEDIUM, CommonClientConfigs.RECEIVE_BUFFER_DOC)
> ...
> {noformat}
> Note that the broker configuration for OS auto tuning is fine as there is no 
> such checks in {{KafkaConfig.scala}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (KAFKA-3840) OS auto tuning for socket buffer size in clients not allowed through configuration

2016-06-16 Thread Ismael Juma (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3840.

   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1507
[https://github.com/apache/kafka/pull/1507]

> OS auto tuning for socket buffer size in clients not allowed through 
> configuration
> --
>
> Key: KAFKA-3840
> URL: https://issues.apache.org/jira/browse/KAFKA-3840
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sébastien Launay
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> KAFKA-724 added the following documentation for clients in order to provide 
> OS auto tuning of socket buffer sizes:
> {noformat}
> diff --git 
> a/clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> b/clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java
> index 750b8a1..3327815 100644
> --- a/clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java
> +++ b/clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java
> @@ -43,10 +43,10 @@
>  public static final String METADATA_MAX_AGE_DOC = "The period of time in 
> milliseconds after which we force a refresh of metadata even if we haven't 
> seen any partition leadership changes to proactively discover any new brokers 
> or partitions.";
>  
>  public static final String SEND_BUFFER_CONFIG = "send.buffer.bytes";
> -public static final String SEND_BUFFER_DOC = "The size of the TCP send 
> buffer (SO_SNDBUF) to use when sending data.";
> +public static final String SEND_BUFFER_DOC = "The size of the TCP send 
> buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS 
> default will be used.";
>  
>  public static final String RECEIVE_BUFFER_CONFIG = 
> "receive.buffer.bytes";
> -public static final String RECEIVE_BUFFER_DOC = "The size of the TCP 
> receive buffer (SO_RCVBUF) to use when reading data.";
> +public static final String RECEIVE_BUFFER_DOC = "The size of the TCP 
> receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the 
> OS default will be used.";
>  
>  public static final String CLIENT_ID_CONFIG = "client.id";
>  public static final String CLIENT_ID_DOC = "An id string to pass to the 
> server when making requests. The purpose of this is to be able to track the 
> source of requests beyond just ip/port by allowing a logical application name 
> to be included in server-side request logging.";
> {noformat}
> Unfortunately, the value {{-1}} is not allowed for clients configuration and 
> the following exception is thrown:
> {noformat}
> Exception in thread "main" org.apache.kafka.common.config.ConfigException: 
> Invalid value -1 for configuration receive.buffer.bytes: Value must be at 
> least 0
> at 
> org.apache.kafka.common.config.ConfigDef$Range.ensureValid(ConfigDef.java:308)
> at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:153)
> at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:49)
> at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:56)
> at 
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:315)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:156)
> ...
> {noformat}
> This is caused by an {{atLeast(0)}} validation in {{ProducerConfig.java}} and 
> {{ConsumerConfig.java}}::
> {noformat}
> ...
>   .define(SEND_BUFFER_CONFIG, Type.INT, 128 * 1024, atLeast(0), 
> Importance.MEDIUM, CommonClientConfigs.SEND_BUFFER_DOC)
>   .define(RECEIVE_BUFFER_CONFIG, Type.INT, 32 * 1024, atLeast(0), 
> Importance.MEDIUM, CommonClientConfigs.RECEIVE_BUFFER_DOC)
> ...
> {noformat}
> Note that the broker configuration for OS auto tuning is fine as there is no 
> such checks in {{KafkaConfig.scala}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-3370) Add options to auto.offset.reset to reset offsets upon initialization only

2016-06-16 Thread James Clarke (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333736#comment-15333736
 ] 

James Clarke commented on KAFKA-3370:
-

I think those options make sense.  Without these options it is hard to get 
mirror maker to work correctly (without risk of duplicate data) for regex 
whitelist and newly created topics (KAFKA-3848).

> Add options to auto.offset.reset to reset offsets upon initialization only
> --
>
> Key: KAFKA-3370
> URL: https://issues.apache.org/jira/browse/KAFKA-3370
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Vahid Hashemian
> Fix For: 0.10.1.0
>
>
> Currently "auto.offset.reset" is applied in the following two cases:
> 1) upon starting the consumer for the first time (hence no committed offsets 
> before);
> 2) upon fetching offsets out-of-range.
> For scenarios where case 2) needs to be avoid (i.e. people need to be 
> notified upon offsets out-of-range rather than silently offset reset), 
> "auto.offset.reset" need to be set to "none". However for case 1) setting 
> "auto.offset.reset" to "none" will cause NoOffsetForPartitionException upon 
> polling. And in this case, seekToBeginning/seekToEnd is mistakenly applied 
> trying to set the offset at initialization, which are actually designed for 
> during the life time of the consumer (in rebalance callback, for example).
> The fix proposal is to add two more options to "auto.offset.reset", 
> "earliest-on-start", and "latest-on-start", whose semantics are "earliest" 
> and "latest" for case 1) only, and "none" for case 2).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Memory consumption of Kafka-examples Kafka-streams around 1.5 GB

2016-06-16 Thread Michael Noll
Phil,

I think what you're seeing w.r.t. memory consumption is that the packaging
step (`mvn package`) for the Kafka Streams examples under
https://github.com/confluentinc/examples also runs the full test suite, and
this test suite launches embedded Kafka brokers, ZooKeeper, and Confluent
schema registry.

As I mentioned to you via our offline email conversation, the easiest
workaround might be to disable the tests when packaging:

$ mvn -DskipTests=true clean package

I also updated the kafka-streams/README.md to include this tip.

Hope this helps,
Michael





On Thu, Jun 16, 2016 at 12:37 AM, Philippe Derome 
wrote:

> Guozhang,
>
> No two in particular, at first it was simply the last two that the target
> would choose: SumLambdaIntegrationTest and WordCountLambdaIntegrationTest .
> I tried another couple to exclude and it was fine as well. There's one
> Scala test that is included in the run and I run it as well as 7/9 of the
> Java ones.
>
> I am not convinced by what you say because after 2-3 tests, I'd think ZK
> and Kafka should be loaded in and I think anecdotally there's about 700MB
> loaded, so there's quite a bit loaded when running the next few tests, at
> least as far as I can see.
>
> The main thing for me is to be reassured that the order of magnitude of
> memory consumption looks right to you. I find it a bit on the high side but
> I won't argue that. So, all in all, I am satisfied with your answer.
>
> On Wed, Jun 15, 2016 at 6:25 PM, Guozhang Wang  wrote:
>
> > Hello Phillippe,
> >
> > I used to run the "SimpleBenchmark" on my laptop with 4GB also, and it
> > usually used close to, but less than 1GB.
> >
> >
> >
> https://www.codatlas.com/github.com/apache/kafka/HEAD/streams/src/test/java/org/apache/kafka/streams/perf/SimpleBenchmark.java
> >
> > Note that I need to bootstrap a real ZK instance and a Kafka instance in
> > order to run that benchmark, and I think those two instances are actually
> > taking the major memory usage than Kafka Streams instance itself.
> >
> > There may be some extra memory overhead from maven framework but I would
> be
> > surprised if that is taking large amount.
> >
> > Which two test cases specifically are causing OOMs on your laptop?
> >
> > Guozhang
> >
> >
> > On Tue, Jun 14, 2016 at 4:50 PM, Philippe Derome 
> > wrote:
> >
> > > I am running "mvn test" as per tip from
> > > https://github.com/confluentinc/examples/tree/master/kafka-streams
> > > README.MD.
> > > This uses embedded Kafka components from test alone (no ZK, Kafka,
> > > schema-registry running).
> > >
> > > I monitor on OSX El Capitan (10.11.5) memory usage and it grows on Java
> > > processes from nothing to about 1.3GB when it fails to execute last 2
> > tests
> > > with Java out of memory exceptions. Selecting 2 tests to avoid makes
> the
> > > test pass but with my 4GB system, I cannot pass them all.
> > >
> > > Is that relatively large memory consumption to be expected on these
> test
> > > cases?
> > >
> > > I'd like to run stand-alone from jar and I'll be able to do so by
> > excluding
> > > test cases.
> > >
> > > Fyi, on mailing list I see only 1 issue related to Streams and memory
> if
> > > that's any relevant (KAFKA-3738).
> > >
> > > Phil
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
Best regards,
Michael Noll



*Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
Apache Kafka and Confluent Platform: www.confluent.io/download
*


Re: Complexe Event Processing on top of KafkaStreams

2016-06-16 Thread Michael Noll
Thanks for sharing, Florian!

-Michael

On Thu, Jun 16, 2016 at 12:32 AM, Guozhang Wang  wrote:

> Hello Florian,
>
> Thanks for your interests! As mentioned in our release notes we are
> considering to add the SQL support (e.g. using Calcite) on top of Kafka
> Streams as a near-term future work.
>
> My own experience with CEP originates completely from a research project
> called "Cayuga" from my grad school:
> http://www.cs.cornell.edu/bigreddata/cayuga/, but from that limited
> experience I personally think that may be naturally fit for CEP use cases.
>
> Let me know what do you think.
>
>
> Guozhang
>
>
> On Wed, Jun 15, 2016 at 3:59 AM, Florian Hussonnois  >
> wrote:
>
> > Hi Team Kafka,
> >
> > Currently, I'm working on an small library to implement "complex event
> > processing" on top of Kafka Streams :
> > https://github.com/fhussonnois/kafkastreams-cep
> >
> > The idea came from the flink-cep library and the project is based on the
> > same research paper.
> >
> > I'm developping this project for fun. But I'm not expert about CEP so
> maybe
> > I'm doing things wrong ^^
> >
> > I would like to share with you my work because I think this could
> interest
> > the kafka community.
> > The project is still in progress but things seem to be on a right way.
> >
> > I've already planned to add a support for KStream DSL.
> >
> > Also, I would like to know if you will plan to add an "external" module
> in
> > order to add contributions without impacting the kafkastreams APIs?
> >
> > Please feel free to give me your feedback about my API.
> >
> > Thanks for you time and the amazing work you are doing on Kafka.
> >
> > Florian.
> >
> > --
> > Florian HUSSONNOIS
> > @fhussonnois
> >
>
>
>
> --
> -- Guozhang
>


Build failed in Jenkins: kafka-trunk-jdk8 #701

2016-06-16 Thread Apache Jenkins Server
See 

Changes:

[ismael] KAFKA-3840; Allow clients default OS buffer sizes

--
[...truncated 8122 lines...]

org.apache.kafka.common.metrics.JmxReporterTest > testJmxRegistration PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testHistogram STARTED

org.apache.kafka.common.metrics.stats.HistogramTest > testHistogram PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testConstantBinScheme 
STARTED

org.apache.kafka.common.metrics.stats.HistogramTest > testConstantBinScheme 
PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testLinearBinScheme 
STARTED

org.apache.kafka.common.metrics.stats.HistogramTest > testLinearBinScheme PASSED

org.apache.kafka.common.utils.CrcTest > testUpdateInt STARTED

org.apache.kafka.common.utils.CrcTest > testUpdateInt PASSED

org.apache.kafka.common.utils.CrcTest > testUpdate STARTED

org.apache.kafka.common.utils.CrcTest > testUpdate PASSED

org.apache.kafka.common.utils.UtilsTest > testAbs STARTED

org.apache.kafka.common.utils.UtilsTest > testAbs PASSED

org.apache.kafka.common.utils.UtilsTest > testMin STARTED

org.apache.kafka.common.utils.UtilsTest > testMin PASSED

org.apache.kafka.common.utils.UtilsTest > testJoin STARTED

org.apache.kafka.common.utils.UtilsTest > testJoin PASSED

org.apache.kafka.common.utils.UtilsTest > testReadBytes STARTED

org.apache.kafka.common.utils.UtilsTest > testReadBytes PASSED

org.apache.kafka.common.utils.UtilsTest > testGetHost STARTED

org.apache.kafka.common.utils.UtilsTest > testGetHost PASSED

org.apache.kafka.common.utils.UtilsTest > testGetPort STARTED

org.apache.kafka.common.utils.UtilsTest > testGetPort PASSED

org.apache.kafka.common.utils.UtilsTest > testCloseAll STARTED

org.apache.kafka.common.utils.UtilsTest > testCloseAll PASSED

org.apache.kafka.common.utils.UtilsTest > testFormatAddress STARTED

org.apache.kafka.common.utils.UtilsTest > testFormatAddress PASSED

org.apache.kafka.common.utils.AbstractIteratorTest > testEmptyIterator STARTED

org.apache.kafka.common.utils.AbstractIteratorTest > testEmptyIterator PASSED

org.apache.kafka.common.utils.AbstractIteratorTest > testIterator STARTED

org.apache.kafka.common.utils.AbstractIteratorTest > testIterator PASSED

org.apache.kafka.common.ClusterTest > testBootstrap STARTED

org.apache.kafka.common.ClusterTest > testBootstrap PASSED

org.apache.kafka.common.cache.LRUCacheTest > testEviction STARTED

org.apache.kafka.common.cache.LRUCacheTest > testEviction PASSED

org.apache.kafka.common.cache.LRUCacheTest > testPutGet STARTED

org.apache.kafka.common.cache.LRUCacheTest > testPutGet PASSED

org.apache.kafka.common.cache.LRUCacheTest > testRemove STARTED

org.apache.kafka.common.cache.LRUCacheTest > testRemove PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequestedValidProvided STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequestedValidProvided PASSED

org.apache.kafka.common.network.SslTransportLayerTest > testUnsupportedCiphers 
STARTED

org.apache.kafka.common.network.SslTransportLayerTest > testUnsupportedCiphers 
PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testUnsupportedTLSVersion STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testUnsupportedTLSVersion PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequiredNotProvided STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequiredNotProvided PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequestedNotProvided STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequestedNotProvided PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testInvalidKeystorePassword STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testInvalidKeystorePassword PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationDisabledNotProvided STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationDisabledNotProvided PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testInvalidEndpointIdentification STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testInvalidEndpointIdentification PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testEndpointIdentificationDisabled STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testEndpointIdentificationDisabled PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequiredUntrustedProvided STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequiredUntrustedProvided PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationDisabledUntrustedPr

Re: [DISCUSS] KIP-4 Create Topic Schema

2016-06-16 Thread Grant Henke
I have updated the wiki and pull request based on the feedback. If there
are no objections I will start a vote at the end of the day.

Details for this implementation can be read here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicRequest

The updated pull request can be found here (feel free to review):
https://github.com/apache/kafka/pull/1489

Below is the exact content for clarity:

> Create Topics Request (KAFKA-2945
> )
>
>
>
> CreateTopics Request (Version: 0) => [create_topic_requests] timeout
>   create_topic_requests => topic num_partitions replication_factor 
> [replica_assignment] [configs]
> topic => STRING
> num_partitions => INT32
> replication_factor => INT16
> replica_assignment => partition_id [replicas]
>   partition_id => INT32
>   replicas => INT32
> configs => config_key config_value
>   config_key => STRING
>   config_value => STRING
>   timeout => INT32
>
> CreateTopicsRequest is a batch request to initiate topic creation with
> either predefined or automatic replica assignment and optionally topic
> configuration.
>
> Request semantics:
>
>1. Must be sent to the controller broker
>2. If there are multiple instructions for the same topic in one
>request an InvalidRequestException will be logged on the broker and the
>client will be disconnected.
>   - This is because the list of topics is modeled server side as a
>   map with TopicName as the key
>3. The principal must be authorized to the "Create" Operation on the
>"Cluster" resource to create topics.
>   - Unauthorized requests will receive a ClusterAuthorizationException
>4.
>
>Only one from ReplicaAssignment or (num_partitions + replication_factor
>), can be defined in one instruction.
>- If both parameters are specified an InvalidRequestException will be
>   logged on the broker and the client will be disconnected.
>   - In the case ReplicaAssignment is defined number of partitions and
>   replicas will be calculated from the supplied replica_assignment.
>   - In the case of defined (num_partitions + replication_factor)
>   replica assignment will be automatically generated by the server.
>   - One or the other must be defined. The existing broker side auto
>   create defaults will not be used
>   (default.replication.factor, num.partitions). The client implementation 
> can
>   have defaults for these options when generating the messages.
>   - The first replica in [replicas] is assumed to be the preferred
>   leader. This matches current behavior elsewhere.
>5. Setting a timeout > 0 will allow the request to block until the
>topic metadata is "complete" on the controller node.
>   - Complete means the local topic metadata cache been completely
>   populated and all partitions have leaders
>  - The topic metadata is updated when the controller sends out
>  update metadata requests to the brokers
>   - If a timeout error occurs, the topic could still be created
>   successfully at a later time. Its up to the client to query for the 
> state
>   at that point.
>6. Setting a timeout <= 0 will validate arguments and trigger the
>create topics and return immediately.
>   - This is essentially the fully asynchronous mode we have in the
>   Zookeeper tools today.
>   - The error code in the response will either contain an argument
>   validation exception or a timeout exception. If you receive a timeout
>   exception, because you asked for 0 timeout, you can assume the message 
> was
>   valid and the topic creation was triggered.
>7. The request is not transactional.
>   1. If an error occurs on one topic, the others could still be
>   created.
>   2. Errors are reported independently.
>
> QA:
>
>- Why is CreateTopicsRequest a batch request?
>   - Scenarios where tools or admins want to create many topics should
>   be able to with fewer requests
>   - Example: MirrorMaker may want to create the topics downstream
>- What happens if some topics error immediately? Will it
>return immediately?
>   - The request will block until all topics have either been created,
>   errors, or the timeout has been hit
>   - There is no "short circuiting" where 1 error stops the other
>   topics from being created
>- Why implement "partial blocking" instead of fully async or fully
>consistent?
>   - See Cluster Consistent Blocking
>   
> 
>below
>- Why require the request to go to the contr

[jira] [Created] (KAFKA-3851) Add references to important installation/upgrade notes to release notes

2016-06-16 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-3851:


 Summary: Add references to important installation/upgrade notes to 
release notes 
 Key: KAFKA-3851
 URL: https://issues.apache.org/jira/browse/KAFKA-3851
 Project: Kafka
  Issue Type: Bug
Reporter: Ewen Cheslack-Postava
Priority: Blocker
 Fix For: 0.10.0.1


Today we use the release notes exactly as exported from JIRA (see "Prepare 
release notes" on 
https://cwiki.apache.org/confluence/display/KAFKA/Release+Process) and then 
rely on users to dig through our documentation (none of which starts with 
release-specific notes) to upgrade and installation notes.

Especially for folks who aren't intimately familiar with our docs, the 
information is *very* easy to miss. Ideally we could automate the release notes 
process a bit and then have them automatically modified to *at least* include 
links at the very top (it'd be nice if we had some other header material added 
as well since the automatically generated release notes are very barebones...). 
Even better would be if we could pull in the version-specific 
installation/upgrade notes directly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (KAFKA-3852) Clarify how to handle message format upgrade without killing performance

2016-06-16 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-3852:


 Summary: Clarify how to handle message format upgrade without 
killing performance
 Key: KAFKA-3852
 URL: https://issues.apache.org/jira/browse/KAFKA-3852
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.0.0
Reporter: Ewen Cheslack-Postava
Priority: Blocker
 Fix For: 0.10.0.1


The upgrade notes re: performance impact of the message format change in 
0.10.0.0 are a bit wishy washy in terms of what you ​*should*​ do. They 
describe the potential impact and then say (at the end of a paragraph): "To 
avoid such message conversion before consumers are upgraded to 0.10.0.0, one 
can set". This should probably be written as a playbook for doing the upgrade 
without affecting performance and should be a recommendation, not presented as 
just an option. Nobody with an existing cluster wants the perf impact of 
switching their brokers to the new format while consumers are still using the 
old format unless they have a cluster with extremely light load.

Additionally, if we have some simple benchmark numbers on the performance 
impact on brokers, we should include that information. Today it is very unclear 
how bad that change will be -- 5%, 20%, 90% worse?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-3851) Add references to important installation/upgrade notes to release notes

2016-06-16 Thread Ewen Cheslack-Postava (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3851:
-
Affects Version/s: 0.10.0.0

> Add references to important installation/upgrade notes to release notes 
> 
>
> Key: KAFKA-3851
> URL: https://issues.apache.org/jira/browse/KAFKA-3851
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0
>Reporter: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.0.1
>
>
> Today we use the release notes exactly as exported from JIRA (see "Prepare 
> release notes" on 
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process) and then 
> rely on users to dig through our documentation (none of which starts with 
> release-specific notes) to upgrade and installation notes.
> Especially for folks who aren't intimately familiar with our docs, the 
> information is *very* easy to miss. Ideally we could automate the release 
> notes process a bit and then have them automatically modified to *at least* 
> include links at the very top (it'd be nice if we had some other header 
> material added as well since the automatically generated release notes are 
> very barebones...). Even better would be if we could pull in the 
> version-specific installation/upgrade notes directly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-3144) report members with no assigned partitions in ConsumerGroupCommand

2016-06-16 Thread Vahid Hashemian (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334161#comment-15334161
 ] 

Vahid Hashemian commented on KAFKA-3144:


[~hachikuji] Isn't what you are suggesting similar to what [~junrao] suggested 
at the very first comment 
[here|https://issues.apache.org/jira/browse/KAFKA-3144?focusedCommentId=15115594&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15115594]?

> report members with no assigned partitions in ConsumerGroupCommand
> --
>
> Key: KAFKA-3144
> URL: https://issues.apache.org/jira/browse/KAFKA-3144
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
>  Labels: newbie
>
> A couple of suggestions on improving ConsumerGroupCommand. 
> 1. It would be useful to list members with no assigned partitions when doing 
> describe in ConsumerGroupCommand.
> 2. Currently, we show the client.id of each member when doing describe in 
> ConsumerGroupCommand. Since client.id is supposed to be the logical 
> application id, all members in the same group are supposed to set the same 
> client.id. So, it would be clearer if we show the client id as well as the 
> member id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #1513: MINOR: Follow-up from KAFKA-2720 with comment/styl...

2016-06-16 Thread hachikuji
GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1513

MINOR: Follow-up from KAFKA-2720 with comment/style fixes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka followup-for-kafka-2720

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1513.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1513


commit d013a1c8c1a77d3f6c2d8eace99419bec961ab1d
Author: Jason Gustafson 
Date:   2016-06-16T17:02:55Z

MINOR: Follow-up from KAFKA-2720 with comment/style fixes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-2720) Periodic purging groups in the coordinator

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334164#comment-15334164
 ] 

ASF GitHub Bot commented on KAFKA-2720:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1513

MINOR: Follow-up from KAFKA-2720 with comment/style fixes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka followup-for-kafka-2720

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1513.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1513


commit d013a1c8c1a77d3f6c2d8eace99419bec961ab1d
Author: Jason Gustafson 
Date:   2016-06-16T17:02:55Z

MINOR: Follow-up from KAFKA-2720 with comment/style fixes




> Periodic purging groups in the coordinator
> --
>
> Key: KAFKA-2720
> URL: https://issues.apache.org/jira/browse/KAFKA-2720
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Jason Gustafson
> Fix For: 0.10.1.0
>
>
> Currently the coordinator removes the group (i.e. both removing it from the 
> cache and writing the tombstone message on its local replica without waiting 
> for ack) once it becomes an empty group.
> This can lead to a few issues such as 1) group removal and creation churns 
> when a group with very few members are being rebalanced, 2) if the local 
> write is failed / not propagated to other followers, they can only be removed 
> again when a new coordinator is migrated and detects the group has no members 
> already.
> We could instead piggy-back the periodic offset expiration along with the 
> group purging as well which removes any groups that had no corresponding 
> offsets in the cache any more.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (KAFKA-3144) report members with no assigned partitions in ConsumerGroupCommand

2016-06-16 Thread Vahid Hashemian (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334161#comment-15334161
 ] 

Vahid Hashemian edited comment on KAFKA-3144 at 6/16/16 5:05 PM:
-

[~hachikuji] Isn't what you are suggesting similar to what [~junrao] suggested 
at the very first comment 
[here|https://issues.apache.org/jira/browse/KAFKA-3144?focusedCommentId=15115594&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15115594]?
 If so, I suggested it to go under a different JIRA since scope-wise it is a 
bit different from the rest of this JIRA. Having said that I'm okay both ways.


was (Author: vahid):
[~hachikuji] Isn't what you are suggesting similar to what [~junrao] suggested 
at the very first comment 
[here|https://issues.apache.org/jira/browse/KAFKA-3144?focusedCommentId=15115594&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15115594]?

> report members with no assigned partitions in ConsumerGroupCommand
> --
>
> Key: KAFKA-3144
> URL: https://issues.apache.org/jira/browse/KAFKA-3144
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
>  Labels: newbie
>
> A couple of suggestions on improving ConsumerGroupCommand. 
> 1. It would be useful to list members with no assigned partitions when doing 
> describe in ConsumerGroupCommand.
> 2. Currently, we show the client.id of each member when doing describe in 
> ConsumerGroupCommand. Since client.id is supposed to be the logical 
> application id, all members in the same group are supposed to set the same 
> client.id. So, it would be clearer if we show the client id as well as the 
> member id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-3144) report members with no assigned partitions in ConsumerGroupCommand

2016-06-16 Thread Jason Gustafson (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334175#comment-15334175
 ] 

Jason Gustafson commented on KAFKA-3144:


Haha, yeah. Should have known that Jun would already have thought of it. 

> report members with no assigned partitions in ConsumerGroupCommand
> --
>
> Key: KAFKA-3144
> URL: https://issues.apache.org/jira/browse/KAFKA-3144
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
>  Labels: newbie
>
> A couple of suggestions on improving ConsumerGroupCommand. 
> 1. It would be useful to list members with no assigned partitions when doing 
> describe in ConsumerGroupCommand.
> 2. Currently, we show the client.id of each member when doing describe in 
> ConsumerGroupCommand. Since client.id is supposed to be the logical 
> application id, all members in the same group are supposed to set the same 
> client.id. So, it would be clearer if we show the client id as well as the 
> member id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-3144) report members with no assigned partitions in ConsumerGroupCommand

2016-06-16 Thread Jason Gustafson (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334201#comment-15334201
 ] 

Jason Gustafson commented on KAFKA-3144:


[~vahid] I created KAFKA-3853. Feel free to pick it up if you want. I'll try to 
help out review on this issue.

> report members with no assigned partitions in ConsumerGroupCommand
> --
>
> Key: KAFKA-3144
> URL: https://issues.apache.org/jira/browse/KAFKA-3144
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
>  Labels: newbie
>
> A couple of suggestions on improving ConsumerGroupCommand. 
> 1. It would be useful to list members with no assigned partitions when doing 
> describe in ConsumerGroupCommand.
> 2. Currently, we show the client.id of each member when doing describe in 
> ConsumerGroupCommand. Since client.id is supposed to be the logical 
> application id, all members in the same group are supposed to set the same 
> client.id. So, it would be clearer if we show the client id as well as the 
> member id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (KAFKA-3853) Report offsets for empty groups in ConsumerGroupCommand

2016-06-16 Thread Vahid Hashemian (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian reassigned KAFKA-3853:
--

Assignee: Vahid Hashemian

> Report offsets for empty groups in ConsumerGroupCommand
> ---
>
> Key: KAFKA-3853
> URL: https://issues.apache.org/jira/browse/KAFKA-3853
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
>
> We ought to be able to display offsets for groups which either have no active 
> members or which are not using group management. The owner column can be left 
> empty or set to "N/A". If a group is active, I'm not sure it would make sense 
> to report all offsets, in particular when partitions are unassigned, but if 
> it seems problematic to do so, we could enable the behavior with a flag (e.g. 
> --include-unassigned).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-3144) report members with no assigned partitions in ConsumerGroupCommand

2016-06-16 Thread Vahid Hashemian (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334216#comment-15334216
 ] 

Vahid Hashemian commented on KAFKA-3144:


Thanks [~hachikuji].

> report members with no assigned partitions in ConsumerGroupCommand
> --
>
> Key: KAFKA-3144
> URL: https://issues.apache.org/jira/browse/KAFKA-3144
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
>  Labels: newbie
>
> A couple of suggestions on improving ConsumerGroupCommand. 
> 1. It would be useful to list members with no assigned partitions when doing 
> describe in ConsumerGroupCommand.
> 2. Currently, we show the client.id of each member when doing describe in 
> ConsumerGroupCommand. Since client.id is supposed to be the logical 
> application id, all members in the same group are supposed to set the same 
> client.id. So, it would be clearer if we show the client id as well as the 
> member id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (KAFKA-3853) Report offsets for empty groups in ConsumerGroupCommand

2016-06-16 Thread Jason Gustafson (JIRA)
Jason Gustafson created KAFKA-3853:
--

 Summary: Report offsets for empty groups in ConsumerGroupCommand
 Key: KAFKA-3853
 URL: https://issues.apache.org/jira/browse/KAFKA-3853
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


We ought to be able to display offsets for groups which either have no active 
members or which are not using group management. The owner column can be left 
empty or set to "N/A". If a group is active, I'm not sure it would make sense 
to report all offsets, in particular when partitions are unassigned, but if it 
seems problematic to do so, we could enable the behavior with a flag (e.g. 
--include-unassigned).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


RE: Embedding zookeeper and kafka in java process.

2016-06-16 Thread Subhash Agrawal
Thanks Ismael.
I am instantiating kafkaserver instance like this.
new KafkaServer(kafkaConfig,null,null);

I tried to use 
new KafkaServer(kafkaConfig); but it does not compile with kafka 0.10.0.

All the example I see uses 
new KafkaServer(kafkaConfig);  

Do we support new KafkaServer(kafkaConfig);   with kafka 0.10.0? if not, how 
can I pass
these parameters? It used to work with kafka 0.7.1.

Thanks
Subhash Agrawal


-Original Message-
From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of Ismael Juma
Sent: Thursday, June 16, 2016 1:38 AM
To: dev@kafka.apache.org
Subject: Re: Embedding zookeeper and kafka in java process.

Hi Subhash,

This would happen if `null` is passed as the `threadNamePrefix` argument
when instantiating `KafkaServer`:

class KafkaServer(val config: KafkaConfig, time: Time = SystemTime,
threadNamePrefix: Option[String] = None) extends Logging with
KafkaMetricsGroup

How are you starting Kafka in your Java process?

Ismael

On Thu, Jun 16, 2016 at 1:26 AM, Subhash Agrawal 
wrote:

> Thanks for quick response.
> I started zookeeper via zookeeper-server-start.bat and started kafka via
> my java process and I saw same error.
> But if I start zookeeper via java process and start kafka via
> kafka-server-start.bat, t works fine.
> It means it is not caused due to both getting started in same process. It
> must be some kafka specific issue.
>
> Subhash Agrawal
>
> -Original Message-
> From: Guozhang Wang [mailto:wangg...@gmail.com]
> Sent: Wednesday, June 15, 2016 3:42 PM
> To: dev@kafka.apache.org
> Subject: Re: Embedding zookeeper and kafka in java process.
>
> It seems "scala.MatchError: null" are not related to the settings that ZK
> and Kafka is embedded in the same process, and the only case that I can
> think of related is this: https://issues.apache.org/jira/browse/KAFKA-940.
>
> Could you clarify if you start these two services on two processes, the
> issue goes away?
>
> Guozhang
>
> On Wed, Jun 15, 2016 at 1:56 PM, Subhash Agrawal 
> wrote:
>
> > Hi All,
> > I am embedding Kafka 0.10.0 and corresponding zookeeper in java process.
> > In this process, I start zookeeper first and then wait for 10 seconds and
> > then start kafka. These are all running in the same process. Toward the
> > end of kafka startup, I see following exception. It seems zookeeper is
> not
> > able
> > to add the newly created kafka instance. Have you seen this error
> > earlier?  I have only single node kafka.
> >
> > Let me know if you have any suggestions. I will really appreciate any
> help
> > on this.
> >
> > Thanks
> > Subhash Agrawal.
> >
> > [2016-06-15 13:39:39,616] INFO [Logserver_Starter] Registered broker 0 at
> > path /brokers/ids/0 with addresses: PLAINTEXT ->
> > EndPoint(localhost,8392,PLAINTEXT) (kafka.utils.ZkUtils)
> > [2016-06-15 13:39:39,617] WARN [Logserver_Starter] No meta.properties
> file
> > under dir C:\development \newkafka-logs\meta.properties
> > (kafka.server.BrokerMetadataCheckpoint)
> > [2016-06-15 13:39:39,627] INFO [ZkClient-EventThread-24-localhost:2181]
> > New leader is 0
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > [2016-06-15 13:39:39,629] INFO [ZkClient-EventThread-24-localhost:2181]
> > [BrokerChangeListener on Controller 0]: Broker change listener fired for
> > path /brokers/ids with children 0
> > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > [2016-06-15 13:39:39,638] INFO [Logserver_Starter] Kafka version :
> > 0.10.0.0 (org.apache.kafka.common.utils.AppInfoParser)
> > [2016-06-15 13:39:39,638] INFO [Logserver_Starter] Kafka commitId :
> > b8642491e78c5a13 (org.apache.kafka.common.utils.AppInfoParser)
> > [2016-06-15 13:39:39,639] INFO [Logserver_Starter] [Kafka Server 0],
> > started (kafka.server.KafkaServer)
> > [2016-06-15 13:39:39,806] INFO [ZkClient-EventThread-24-localhost:2181]
> > [BrokerChangeListener on Controller 0]: Newly added brokers: 0, deleted
> > brokers: , all live brokers: 0
> > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > [2016-06-15 13:39:39,808] DEBUG [ZkClient-EventThread-24-localhost:2181]
> > [Channel manager on controller 0]: Controller 0 trying to connect to
> broker
> > 0 (kafka.controller.ControllerChannelManager)
> > [2016-06-15 13:39:39,818] ERROR [ZkClient-EventThread-24-localhost:2181]
> > [BrokerChangeListener on Controller 0]: Error while handling broker
> changes
> > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > scala.MatchError: null
> > at
> >
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:122)
> > at
> >
> kafka.controller.ControllerChannelManager.addBroker(ControllerChannelManager.scala:74)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(ReplicaStateMachine.scala:372)
> > a

[jira] [Commented] (KAFKA-3144) report members with no assigned partitions in ConsumerGroupCommand

2016-06-16 Thread Vahid Hashemian (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334223#comment-15334223
 ] 

Vahid Hashemian commented on KAFKA-3144:


This improvement request was moved to its own JIRA 
([KAFKA-3853|https://issues.apache.org/jira/browse/KAFKA-3853]).

> report members with no assigned partitions in ConsumerGroupCommand
> --
>
> Key: KAFKA-3144
> URL: https://issues.apache.org/jira/browse/KAFKA-3144
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
>  Labels: newbie
>
> A couple of suggestions on improving ConsumerGroupCommand. 
> 1. It would be useful to list members with no assigned partitions when doing 
> describe in ConsumerGroupCommand.
> 2. Currently, we show the client.id of each member when doing describe in 
> ConsumerGroupCommand. Since client.id is supposed to be the logical 
> application id, all members in the same group are supposed to set the same 
> client.id. So, it would be clearer if we show the client id as well as the 
> member id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (KAFKA-3854) Consecutive regex subscription calls fail

2016-06-16 Thread Vahid Hashemian (JIRA)
Vahid Hashemian created KAFKA-3854:
--

 Summary: Consecutive regex subscription calls fail
 Key: KAFKA-3854
 URL: https://issues.apache.org/jira/browse/KAFKA-3854
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Reporter: Vahid Hashemian
Assignee: Vahid Hashemian


When consecutive calls are made to new consumer's [regex 
subscription|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L850],
 like below: 

{code}
consumer.subscribe(Pattern.compile("..."), listener);
consumer.poll(0);

consumer.subscribe(Pattern.compile("f.."), listener);
consumer.poll(0);
{code}

the second call fails with the following error:

{code}
Exception in thread "main" java.lang.IllegalStateException: Subscription to 
topics, partitions and pattern are mutually exclusive
at 
org.apache.kafka.clients.consumer.internals.SubscriptionState.subscribe(SubscriptionState.java:175)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(KafkaConsumer.java:854)
at 
ConsumerSubscriptionSemantics.tryRegexSubscriptionSemantics(ConsumerSubscriptionSemantics.java:76)
at 
ConsumerSubscriptionSemantics.main(ConsumerSubscriptionSemantics.java:88)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


avro-console-producer issue for Kafka Stream example PageViewRegionLambdaExample

2016-06-16 Thread Philippe Derome
I am able to produce Avro from producer for pageview.avsc but not for
userprofile.avsc. Here are two samples below, first one leaves out Avro
optional experience, second one specifies it.

Note: I do not register using REST curl calls schemas to schema-registry as
I assume that kafka-avro-console-producer takes care of that.

First case has Avro disagreeing that experience can be optional with an
exception:

AvroTypeException: Expected field name not found: experience

Second case generates another AvroTypeException complaining about the union
type:

Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
VALUE_STRING

./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
UserProfile --property value.schema='{

 "type": "record",

 "name": "UserProfile",

 "fields": [

 {"name": "experience", "type": ["string", "null"]},

 {"name": "region", "type": "string"}

 ]

}'

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

{"region":"US"}

org.apache.kafka.common.errors.SerializationException: Error deserializing
json {"region":"US"} to Avro of schema
{"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}

Caused by: org.apache.avro.AvroTypeException: Expected field name not
found: experience

at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)

at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)

at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)

at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:429)

at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)

at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)

at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)

at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)

at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)

at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)

at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)

at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)

at
io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:189)

at
io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:157)

at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)

at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)


./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
UserProfile --property value.schema='{

 "type": "record",

 "name": "UserProfile",

 "fields": [

 {"name": "experience", "type": ["string", "null"]},

 {"name": "region", "type": "string"}

 ]

}'

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in
[jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

{"region":"US","experience":"scala"}

org.apache.kafka.common.errors.SerializationException: Error deserializing
json {"region":"US","experience":"scala"} to Avro of schema
{"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}

Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
VALUE_STRING

at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:697)

at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:441)

at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)

at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)

at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)

at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)

at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatum

[VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

2016-06-16 Thread Jason Gustafson
Hi All,

I'd like to open the vote for KIP-62. This proposal attempts to address one
of the recurring usability problems that users of the new consumer have
faced with as little impact as possible. You can read the full details
here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread
.

After some discussion on this list, I think we were in agreement that this
change addresses a major part of the problem and we've left the door open
for further improvements, such as adding a heartbeat() API or a separately
configured rebalance timeout. Thanks in advance to everyone who helped
review the proposal.

-Jason


[jira] [Created] (KAFKA-3855) Guard race conditions in TopologyBuilder

2016-06-16 Thread Guozhang Wang (JIRA)
Guozhang Wang created KAFKA-3855:


 Summary: Guard race conditions in TopologyBuilder
 Key: KAFKA-3855
 URL: https://issues.apache.org/jira/browse/KAFKA-3855
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Guozhang Wang


The user specified {{TopologyBuilder}} is shared among all stream threads to 
build the processor topology instance, one for each thread. It public functions 
that can be accessed by the threads is not synchronized, and we need to double 
check if it could cause race conditions, and if yes guard against these 
concurrent access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (KAFKA-3856) Move inner classes accessible only functions in TopologyBuilder out of public APIs

2016-06-16 Thread Guozhang Wang (JIRA)
Guozhang Wang created KAFKA-3856:


 Summary: Move inner classes accessible only functions in 
TopologyBuilder out of public APIs
 Key: KAFKA-3856
 URL: https://issues.apache.org/jira/browse/KAFKA-3856
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Guozhang Wang


In {{TopologyBuilder}} there are a couple of public functions that are actually 
only used in the internal classes such as StreamThread and 
StreamPartitionAssignor, and some accessible only in high-level DSL inner 
classes, examples include {{addInternalTopic}}, {{sourceGroups}} and 
{{copartitionGroups}}, etc. But they are still listed in Javadocs since this 
class is part of public APIs.

We should think about moving them out of the public functions. Unfortunately 
there is no "friend" access mode as in C++, so we need to think of another way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (KAFKA-3561) Auto create through topic for KStream aggregation and join

2016-06-16 Thread Guozhang Wang (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-3561.
--
Resolution: Fixed

Issue resolved by pull request 1472
[https://github.com/apache/kafka/pull/1472]

> Auto create through topic for KStream aggregation and join
> --
>
> Key: KAFKA-3561
> URL: https://issues.apache.org/jira/browse/KAFKA-3561
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Damian Guy
>  Labels: api
> Fix For: 0.10.1.0
>
>
> For KStream.join / aggregateByKey operations that requires the streams to be 
> partitioned on the record key, today users should repartition themselves 
> through the "through" call:
> {code}
> stream1 = builder.stream("topic1");
> stream2 = builder.stream("topic2");
> stream3 = stream1.map(/* set the right key for join*/).through("topic3");
> stream4 = stream2.map(/* set the right key for join*/).through("topic4");
> stream3.join(stream4, ..)
> {code}
> This pattern can actually be done by the Streams DSL itself instead of 
> requiring users to specify themselves, i.e. users can just set the right key 
> like (see KAFKA-3430) and then call join, which will be translated by adding 
> the "internal topic for repartition".
> Another thing is that today if user do not call "through" after setting a new 
> key, the aggregation result would not be correct as the aggregation is based 
> on key B while the source partitions is partitioned by key A and hence each 
> task will only get a partial aggregation for all keys. But this is not 
> validated in the DSL today. We should do both the auto-translation and 
> validation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #1472: KAFKA-3561: Auto create through topic for KStream ...

2016-06-16 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1472


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-3561) Auto create through topic for KStream aggregation and join

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334446#comment-15334446
 ] 

ASF GitHub Bot commented on KAFKA-3561:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1472


> Auto create through topic for KStream aggregation and join
> --
>
> Key: KAFKA-3561
> URL: https://issues.apache.org/jira/browse/KAFKA-3561
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Damian Guy
>  Labels: api
> Fix For: 0.10.1.0
>
>
> For KStream.join / aggregateByKey operations that requires the streams to be 
> partitioned on the record key, today users should repartition themselves 
> through the "through" call:
> {code}
> stream1 = builder.stream("topic1");
> stream2 = builder.stream("topic2");
> stream3 = stream1.map(/* set the right key for join*/).through("topic3");
> stream4 = stream2.map(/* set the right key for join*/).through("topic4");
> stream3.join(stream4, ..)
> {code}
> This pattern can actually be done by the Streams DSL itself instead of 
> requiring users to specify themselves, i.e. users can just set the right key 
> like (see KAFKA-3430) and then call join, which will be translated by adding 
> the "internal topic for repartition".
> Another thing is that today if user do not call "through" after setting a new 
> key, the aggregation result would not be correct as the aggregation is based 
> on key B while the source partitions is partitioned by key A and hence each 
> task will only get a partial aggregation for all keys. But this is not 
> validated in the DSL today. We should do both the auto-translation and 
> validation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #1514: MINOR: Mention `log.message.format.version=0.10.0`...

2016-06-16 Thread ijuma
GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1514

MINOR: Mention `log.message.format.version=0.10.0` in rolling upgrade 
section

We had mentioned this step in the performance impact section in the middle 
of a long paragraph, which made it easy to miss.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka tweak-upgrade-notes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1514.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1514


commit 478fc3104e97bcdee9883f9d92f9db47f4a2eb66
Author: Ismael Juma 
Date:   2016-06-16T18:58:41Z

MINOR: Mention `log.message.format.version=0.10.0` in rolling upgrade 
section




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Created] (KAFKA-3857) Additional log cleaner metrics

2016-06-16 Thread Kiran Pillarisetty (JIRA)
Kiran Pillarisetty created KAFKA-3857:
-

 Summary: Additional log cleaner metrics
 Key: KAFKA-3857
 URL: https://issues.apache.org/jira/browse/KAFKA-3857
 Project: Kafka
  Issue Type: Improvement
Reporter: Kiran Pillarisetty


The proposal would be to add a couple of additional log cleaner metrics: 
1. Time of last log cleaner run 
2. Cumulative number of successful log cleaner runs since last broker restart.

Existing log cleaner metrics (max-buffer-utilization-percent, 
cleaner-recopy-percent, max-clean-time-secs, max-dirty-percent) do not 
differentiate an idle log cleaner from a dead log cleaner. It would be useful 
to have the above two metrics added, to indicate whether log cleaner is alive 
(and successfully cleaning) or not.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: FYI - Apache ZooKeeper Backup, a Treatise

2016-06-16 Thread Chris Nauroth
Yes, thank you to Jordan for the article!

Like Flavio, I personally have never come across the requirement for
ZooKeeper backups.  I've generally followed the pattern that data stored
in ZooKeeper is truly transient, and applications are built either to
tolerate loss of that data or reconstruct it from first principles if it
goes missing.  Adding observers in a second data center would give a
rudimentary approximation of off-site backup in the case of a data center
disaster, with the usual caveats around propagation delays.

Jordan, I'd be curious if you can share more specific details about the
kind of data that you have that necessitates a backup/restore.  (If you're
not at liberty to share this, then I can understand that.)  It might
inform if we have a motivating use case for backup/restore features within
ZooKeeper, such as some of the transaction log filtering that the article
mentions.

--Chris Nauroth




On 6/16/16, 1:03 AM, "Flavio Junqueira"  wrote:

>Great write-up, Jordan, thanks!
>
>Whether to backup zk data or not is possibly an open topic for this
>community, even though we have discussed it at times. My sense has been
>that precisely because of the issues you mention in your post, it is
>typically best to have a way to recreate its data upon a disaster rather
>than backup the data. I think there could be three general scenarios in
>which folks would prefer to backup data, but you correct me if these
>aren't accurate:
>
>- The data in zk isn't elsewhere, so it can't be recreated: zk isn't a
>regular database, so I'd think it is best not to store data and focus on
>cluster data or metadata.
>- There is a just a lot of data and I'd rather have a shorter time to
>recover: zk in general shouldn't have that much data in db, but let's go
>with the assumption that for the requirements of the application it is a
>lot. For such a case, it probably depends on whether your application can
>efficiently and effectively recover from a backup. Basically, as pointed
>out in the post, the data could be inconsistent and cause trouble if you
>don't think about the corner cases.
>- The code to recreate the zk metadata for my application is super
>complex: if you decide to code against zk, it is good to think whether
>reconstructing in the case of a disaster is doable and if it is design
>and implement to reconstruct the state upon a disaster.
>
>Also, we typically provision enough replicas, often replicating across
>data centers, to make sure that the data isn't all gone. Having more
>replicas does not rule out completely the possibility of a disaster, but
>in such rare cases we resort to the expensive path.
>
>I personally have never worked with an application that was taking
>backups of zk data in prod, so I'm really interested in what others
>think. 
>
>-Flavio
>
>
>> On 16 Jun 2016, at 00:43, Jordan Zimmerman 
>>wrote:
>> 
>> FYI - I wrote a blog about backing up ZooKeeper:
>> 
>> https://www.elastic.co/blog/zookeeper-backup-a-treatise
>> 
>> -Jordan
>



[jira] [Commented] (KAFKA-789) Producer-side persistence for delivery guarantee

2016-06-16 Thread radha (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334523#comment-15334523
 ] 

radha commented on KAFKA-789:
-

This says resolved but does not mention anything about how this works?

> Producer-side persistence for delivery guarantee
> 
>
> Key: KAFKA-789
> URL: https://issues.apache.org/jira/browse/KAFKA-789
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Reporter: Matan Safriel
>Assignee: Jun Rao
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> A suggestion for higher guarantee for the part of entering messages into 
> Kafka through it's producer. It aims to address the case that the entire set 
> of broker replicas for a topic and partition is not available. Currently, in 
> that case, data is lost. When a message set exhausts the send retry counter, 
> the message set will be simply dropped. It would be nice being able to 
> provide higher guarantee that a message passed to the producer would 
> eventually be received by the broker. 
> In an environment with some disk space to spare for this on the producer 
> side, persisting to disk would seem to enable keeping messages for later 
> retry (until defined space limits are exhausted). Thus somewhat elevating the 
> level of guarantee. 
> One way to facilitate this would be capitalizing on 
> https://issues.apache.org/jira/browse/KAFKA-496, as the feedback it will add 
> will enable knowing what needs to be retried again later. Changes to the 
> producer or a wrapper around it (that may require access to the partitioning 
> functions) would be able to persist failed message sets and manage delivery 
> with a nice level of guarantee. As it would affect performance and use disks, 
> should probably be a non-default option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #1515: KAFKA-3849: Make consumer poll time in MirrorMaker...

2016-06-16 Thread SinghAsDev
GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/1515

KAFKA-3849: Make consumer poll time in MirrorMaker configurable



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-3849

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1515.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1515


commit ff45b7085d54ba9055b97b2139576f4b6213cf28
Author: Ashish Singh 
Date:   2016-06-16T19:41:39Z

KAFKA-3849: Make consumer poll time in MirrorMaker configurable




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-3849) Make consumer poll time in MirrorMaker configurable.

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334531#comment-15334531
 ] 

ASF GitHub Bot commented on KAFKA-3849:
---

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/1515

KAFKA-3849: Make consumer poll time in MirrorMaker configurable



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-3849

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1515.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1515


commit ff45b7085d54ba9055b97b2139576f4b6213cf28
Author: Ashish Singh 
Date:   2016-06-16T19:41:39Z

KAFKA-3849: Make consumer poll time in MirrorMaker configurable




> Make consumer poll time in MirrorMaker configurable.
> 
>
> Key: KAFKA-3849
> URL: https://issues.apache.org/jira/browse/KAFKA-3849
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.0.1
>
>
> MirrorMaker has consumer poll time, consumer timeout time for new consumer, 
> hard coded at 1000 ms. This should be configurable as it is in case of old 
> consumer. Default can stay as 1000 ms though.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-3849) Make consumer poll time in MirrorMaker configurable.

2016-06-16 Thread Ashish K Singh (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-3849:
--
Status: Patch Available  (was: Open)

> Make consumer poll time in MirrorMaker configurable.
> 
>
> Key: KAFKA-3849
> URL: https://issues.apache.org/jira/browse/KAFKA-3849
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.0.1
>
>
> MirrorMaker has consumer poll time, consumer timeout time for new consumer, 
> hard coded at 1000 ms. This should be configurable as it is in case of old 
> consumer. Default can stay as 1000 ms though.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Build failed in Jenkins: kafka-trunk-jdk8 #702

2016-06-16 Thread Apache Jenkins Server
See 

Changes:

[wangguoz] KAFKA-3561: Auto create through topic for KStream aggregation and 
join

--
[...truncated 6592 lines...]

kafka.utils.ByteBoundedBlockingQueueTest > testByteBoundedBlockingQueue PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr STARTED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg STARTED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.UtilsTest > testAbs STARTED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix STARTED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testCircularIterator STARTED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes STARTED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList STARTED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt STARTED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.JsonTest > testJsonEncoding STARTED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.consumer.TopicFilterTest > testWhitelists STARTED

kafka.consumer.TopicFilterTest > testWhitelists PASSED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson STARTED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson PASSED

kafka.consumer.TopicFilterTest > testBlacklists STARTED

kafka.consumer.TopicFilterTest > testBlacklists PASSED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor PASSED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 
PASSED

kafka.consumer.ZookeeperConsumerConn

[DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Ismael Juma
Hi all,

I would like to start a discussion on making Java 8 a minimum requirement
for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
is the first discussion on the topic so the idea is to understand how
people feel about it. If people feel it's too soon, then we can pick up the
conversation again after Kafka 0.10.1.0. If the feedback is mostly
positive, I will start a vote thread.

Let's start with some dates. Java 7 hasn't received public updates since
April 2015[1], Java 8 was released in March 2014[2] and Java 9 is scheduled
to be released in March 2017[3].

The first argument for dropping support for Java 7 is that the last public
release by Oracle contains a large number of known security
vulnerabilities. The effectiveness of Kafka's security features is reduced
if the underlying runtime is not itself secure.

The second argument for moving to Java 8 is that it adds a number of
compelling features:

* Lambda expressions and method references (particularly useful for the
Kafka Streams DSL)
* Default methods (very useful for maintaining compatibility when adding
methods to interfaces)
* java.util.stream (helpful for making collection transformations more
concise)
* Lots of improvements to java.util.concurrent (CompletableFuture,
DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
* Other nice things: SplittableRandom, Optional (and many others I have not
mentioned)

The third argument is that it will simplify our testing matrix, we won't
have to test with Java 7 any longer (this is particularly useful for system
tests that take hours to run). It will also make it easier to support Scala
2.12, which requires Java 8.

The fourth argument is that many other open-source projects have taken the
leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
support Java 8 in the next version (although it will take a while before
most phones will use that version sadly). This reduces (but does not
eliminate) the chance that we would be the first project that would cause a
user to consider a Java upgrade.

The main argument for not making the change is that a reasonable number of
users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
More specifically, we care about the subset who would be able to upgrade to
Kafka 0.10.1.0, but would not be able to upgrade the Java version. It would
be great if we could quantify this in some way.

What do you think?

Ismael

[1] https://java.com/en/download/faq/java_7.xml
[2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
[3] http://openjdk.java.net/projects/jdk9/
[4] https://github.com/apache/cassandra/blob/trunk/README.asc
[5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
[6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
[7] https://issues.apache.org/jira/browse/HADOOP-11858
[8] https://webtide.com/jetty-9-3-features/
[9] http://markmail.org/message/l7s276y3xkga2eqf
[10]
https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
[11] http://markmail.org/message/l7s276y3xkga2eqf


Re: Embedding zookeeper and kafka in java process.

2016-06-16 Thread Ismael Juma
Try using kafka.server.KafkaServerStartable instead. It should do the right
thing.

Ismael

On Thu, Jun 16, 2016 at 7:18 PM, Subhash Agrawal 
wrote:

> Thanks Ismael.
> I am instantiating kafkaserver instance like this.
> new KafkaServer(kafkaConfig,null,null);
>
> I tried to use
> new KafkaServer(kafkaConfig); but it does not compile with kafka 0.10.0.
>
> All the example I see uses
> new KafkaServer(kafkaConfig);
>
> Do we support new KafkaServer(kafkaConfig);   with kafka 0.10.0? if not,
> how can I pass
> these parameters? It used to work with kafka 0.7.1.
>
> Thanks
> Subhash Agrawal
>
>
> -Original Message-
> From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of Ismael
> Juma
> Sent: Thursday, June 16, 2016 1:38 AM
> To: dev@kafka.apache.org
> Subject: Re: Embedding zookeeper and kafka in java process.
>
> Hi Subhash,
>
> This would happen if `null` is passed as the `threadNamePrefix` argument
> when instantiating `KafkaServer`:
>
> class KafkaServer(val config: KafkaConfig, time: Time = SystemTime,
> threadNamePrefix: Option[String] = None) extends Logging with
> KafkaMetricsGroup
>
> How are you starting Kafka in your Java process?
>
> Ismael
>
> On Thu, Jun 16, 2016 at 1:26 AM, Subhash Agrawal 
> wrote:
>
> > Thanks for quick response.
> > I started zookeeper via zookeeper-server-start.bat and started kafka via
> > my java process and I saw same error.
> > But if I start zookeeper via java process and start kafka via
> > kafka-server-start.bat, t works fine.
> > It means it is not caused due to both getting started in same process. It
> > must be some kafka specific issue.
> >
> > Subhash Agrawal
> >
> > -Original Message-
> > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > Sent: Wednesday, June 15, 2016 3:42 PM
> > To: dev@kafka.apache.org
> > Subject: Re: Embedding zookeeper and kafka in java process.
> >
> > It seems "scala.MatchError: null" are not related to the settings that ZK
> > and Kafka is embedded in the same process, and the only case that I can
> > think of related is this:
> https://issues.apache.org/jira/browse/KAFKA-940.
> >
> > Could you clarify if you start these two services on two processes, the
> > issue goes away?
> >
> > Guozhang
> >
> > On Wed, Jun 15, 2016 at 1:56 PM, Subhash Agrawal 
> > wrote:
> >
> > > Hi All,
> > > I am embedding Kafka 0.10.0 and corresponding zookeeper in java
> process.
> > > In this process, I start zookeeper first and then wait for 10 seconds
> and
> > > then start kafka. These are all running in the same process. Toward the
> > > end of kafka startup, I see following exception. It seems zookeeper is
> > not
> > > able
> > > to add the newly created kafka instance. Have you seen this error
> > > earlier?  I have only single node kafka.
> > >
> > > Let me know if you have any suggestions. I will really appreciate any
> > help
> > > on this.
> > >
> > > Thanks
> > > Subhash Agrawal.
> > >
> > > [2016-06-15 13:39:39,616] INFO [Logserver_Starter] Registered broker 0
> at
> > > path /brokers/ids/0 with addresses: PLAINTEXT ->
> > > EndPoint(localhost,8392,PLAINTEXT) (kafka.utils.ZkUtils)
> > > [2016-06-15 13:39:39,617] WARN [Logserver_Starter] No meta.properties
> > file
> > > under dir C:\development \newkafka-logs\meta.properties
> > > (kafka.server.BrokerMetadataCheckpoint)
> > > [2016-06-15 13:39:39,627] INFO [ZkClient-EventThread-24-localhost:2181]
> > > New leader is 0
> > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > > [2016-06-15 13:39:39,629] INFO [ZkClient-EventThread-24-localhost:2181]
> > > [BrokerChangeListener on Controller 0]: Broker change listener fired
> for
> > > path /brokers/ids with children 0
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > [2016-06-15 13:39:39,638] INFO [Logserver_Starter] Kafka version :
> > > 0.10.0.0 (org.apache.kafka.common.utils.AppInfoParser)
> > > [2016-06-15 13:39:39,638] INFO [Logserver_Starter] Kafka commitId :
> > > b8642491e78c5a13 (org.apache.kafka.common.utils.AppInfoParser)
> > > [2016-06-15 13:39:39,639] INFO [Logserver_Starter] [Kafka Server 0],
> > > started (kafka.server.KafkaServer)
> > > [2016-06-15 13:39:39,806] INFO [ZkClient-EventThread-24-localhost:2181]
> > > [BrokerChangeListener on Controller 0]: Newly added brokers: 0, deleted
> > > brokers: , all live brokers: 0
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > [2016-06-15 13:39:39,808] DEBUG
> [ZkClient-EventThread-24-localhost:2181]
> > > [Channel manager on controller 0]: Controller 0 trying to connect to
> > broker
> > > 0 (kafka.controller.ControllerChannelManager)
> > > [2016-06-15 13:39:39,818] ERROR
> [ZkClient-EventThread-24-localhost:2181]
> > > [BrokerChangeListener on Controller 0]: Error while handling broker
> > changes
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > scala.MatchError: null
> > > at
> > >
> >
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelMana

Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Henry Cai
+1 for Lambda expression.

On Thu, Jun 16, 2016 at 1:48 PM, Rajiv Kurian  wrote:

> +1
>
> On Thu, Jun 16, 2016 at 1:45 PM, Ismael Juma  wrote:
>
> > Hi all,
> >
> > I would like to start a discussion on making Java 8 a minimum requirement
> > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> > is the first discussion on the topic so the idea is to understand how
> > people feel about it. If people feel it's too soon, then we can pick up
> the
> > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > positive, I will start a vote thread.
> >
> > Let's start with some dates. Java 7 hasn't received public updates since
> > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> scheduled
> > to be released in March 2017[3].
> >
> > The first argument for dropping support for Java 7 is that the last
> public
> > release by Oracle contains a large number of known security
> > vulnerabilities. The effectiveness of Kafka's security features is
> reduced
> > if the underlying runtime is not itself secure.
> >
> > The second argument for moving to Java 8 is that it adds a number of
> > compelling features:
> >
> > * Lambda expressions and method references (particularly useful for the
> > Kafka Streams DSL)
> > * Default methods (very useful for maintaining compatibility when adding
> > methods to interfaces)
> > * java.util.stream (helpful for making collection transformations more
> > concise)
> > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> > * Other nice things: SplittableRandom, Optional (and many others I have
> not
> > mentioned)
> >
> > The third argument is that it will simplify our testing matrix, we won't
> > have to test with Java 7 any longer (this is particularly useful for
> system
> > tests that take hours to run). It will also make it easier to support
> Scala
> > 2.12, which requires Java 8.
> >
> > The fourth argument is that many other open-source projects have taken
> the
> > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> > support Java 8 in the next version (although it will take a while before
> > most phones will use that version sadly). This reduces (but does not
> > eliminate) the chance that we would be the first project that would
> cause a
> > user to consider a Java upgrade.
> >
> > The main argument for not making the change is that a reasonable number
> of
> > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > More specifically, we care about the subset who would be able to upgrade
> to
> > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> would
> > be great if we could quantify this in some way.
> >
> > What do you think?
> >
> > Ismael
> >
> > [1] https://java.com/en/download/faq/java_7.xml
> > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > [3] http://openjdk.java.net/projects/jdk9/
> > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > [8] https://webtide.com/jetty-9-3-features/
> > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > [10]
> >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > [11] http://markmail.org/message/l7s276y3xkga2eqf
> >
>


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Ben Davison
+ 1

On Thursday, 16 June 2016, Craig Swift 
wrote:

> +1
>
> Craig J. Swift
> Principal Software Engineer - Data Pipeline
> ReturnPath Inc.
> Work: 303-999-3220 Cell: 720-560-7038
>
> On Thu, Jun 16, 2016 at 2:50 PM, Henry Cai 
> wrote:
>
> > +1 for Lambda expression.
> >
> > On Thu, Jun 16, 2016 at 1:48 PM, Rajiv Kurian  > wrote:
> >
> > > +1
> > >
> > > On Thu, Jun 16, 2016 at 1:45 PM, Ismael Juma  > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I would like to start a discussion on making Java 8 a minimum
> > requirement
> > > > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now).
> > This
> > > > is the first discussion on the topic so the idea is to understand how
> > > > people feel about it. If people feel it's too soon, then we can pick
> up
> > > the
> > > > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > > > positive, I will start a vote thread.
> > > >
> > > > Let's start with some dates. Java 7 hasn't received public updates
> > since
> > > > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> > > scheduled
> > > > to be released in March 2017[3].
> > > >
> > > > The first argument for dropping support for Java 7 is that the last
> > > public
> > > > release by Oracle contains a large number of known security
> > > > vulnerabilities. The effectiveness of Kafka's security features is
> > > reduced
> > > > if the underlying runtime is not itself secure.
> > > >
> > > > The second argument for moving to Java 8 is that it adds a number of
> > > > compelling features:
> > > >
> > > > * Lambda expressions and method references (particularly useful for
> the
> > > > Kafka Streams DSL)
> > > > * Default methods (very useful for maintaining compatibility when
> > adding
> > > > methods to interfaces)
> > > > * java.util.stream (helpful for making collection transformations
> more
> > > > concise)
> > > > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > > > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder,
> > LongAccumulator)
> > > > * Other nice things: SplittableRandom, Optional (and many others I
> have
> > > not
> > > > mentioned)
> > > >
> > > > The third argument is that it will simplify our testing matrix, we
> > won't
> > > > have to test with Java 7 any longer (this is particularly useful for
> > > system
> > > > tests that take hours to run). It will also make it easier to support
> > > Scala
> > > > 2.12, which requires Java 8.
> > > >
> > > > The fourth argument is that many other open-source projects have
> taken
> > > the
> > > > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop
> > 3[7],
> > > > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android
> > will
> > > > support Java 8 in the next version (although it will take a while
> > before
> > > > most phones will use that version sadly). This reduces (but does not
> > > > eliminate) the chance that we would be the first project that would
> > > cause a
> > > > user to consider a Java upgrade.
> > > >
> > > > The main argument for not making the change is that a reasonable
> number
> > > of
> > > > users may still be using Java 7 by the time Kafka 0.10.1.0 is
> released.
> > > > More specifically, we care about the subset who would be able to
> > upgrade
> > > to
> > > > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> > > would
> > > > be great if we could quantify this in some way.
> > > >
> > > > What do you think?
> > > >
> > > > Ismael
> > > >
> > > > [1] https://java.com/en/download/faq/java_7.xml
> > > > [2]
> https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > > > [3] http://openjdk.java.net/projects/jdk9/
> > > > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > > > [5]
> > https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > > > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > > > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > > > [8] https://webtide.com/jetty-9-3-features/
> > > > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > > > [10]
> > > >
> > > >
> > >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > > > [11] http://markmail.org/message/l7s276y3xkga2eqf
> > > >
> > >
> >
>

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Adam Kunicki
+1

Adam Kunicki
StreamSets | Field Engineer
mobile: 415.890.DATA (3282) | linkedin


On Thu, Jun 16, 2016 at 1:56 PM, Craig Swift <
craig.sw...@returnpath.com.invalid> wrote:

> +1
>
> Craig J. Swift
> Principal Software Engineer - Data Pipeline
> ReturnPath Inc.
> Work: 303-999-3220 Cell: 720-560-7038
>
> On Thu, Jun 16, 2016 at 2:50 PM, Henry Cai 
> wrote:
>
> > +1 for Lambda expression.
> >
> > On Thu, Jun 16, 2016 at 1:48 PM, Rajiv Kurian 
> wrote:
> >
> > > +1
> > >
> > > On Thu, Jun 16, 2016 at 1:45 PM, Ismael Juma 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I would like to start a discussion on making Java 8 a minimum
> > requirement
> > > > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now).
> > This
> > > > is the first discussion on the topic so the idea is to understand how
> > > > people feel about it. If people feel it's too soon, then we can pick
> up
> > > the
> > > > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > > > positive, I will start a vote thread.
> > > >
> > > > Let's start with some dates. Java 7 hasn't received public updates
> > since
> > > > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> > > scheduled
> > > > to be released in March 2017[3].
> > > >
> > > > The first argument for dropping support for Java 7 is that the last
> > > public
> > > > release by Oracle contains a large number of known security
> > > > vulnerabilities. The effectiveness of Kafka's security features is
> > > reduced
> > > > if the underlying runtime is not itself secure.
> > > >
> > > > The second argument for moving to Java 8 is that it adds a number of
> > > > compelling features:
> > > >
> > > > * Lambda expressions and method references (particularly useful for
> the
> > > > Kafka Streams DSL)
> > > > * Default methods (very useful for maintaining compatibility when
> > adding
> > > > methods to interfaces)
> > > > * java.util.stream (helpful for making collection transformations
> more
> > > > concise)
> > > > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > > > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder,
> > LongAccumulator)
> > > > * Other nice things: SplittableRandom, Optional (and many others I
> have
> > > not
> > > > mentioned)
> > > >
> > > > The third argument is that it will simplify our testing matrix, we
> > won't
> > > > have to test with Java 7 any longer (this is particularly useful for
> > > system
> > > > tests that take hours to run). It will also make it easier to support
> > > Scala
> > > > 2.12, which requires Java 8.
> > > >
> > > > The fourth argument is that many other open-source projects have
> taken
> > > the
> > > > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop
> > 3[7],
> > > > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android
> > will
> > > > support Java 8 in the next version (although it will take a while
> > before
> > > > most phones will use that version sadly). This reduces (but does not
> > > > eliminate) the chance that we would be the first project that would
> > > cause a
> > > > user to consider a Java upgrade.
> > > >
> > > > The main argument for not making the change is that a reasonable
> number
> > > of
> > > > users may still be using Java 7 by the time Kafka 0.10.1.0 is
> released.
> > > > More specifically, we care about the subset who would be able to
> > upgrade
> > > to
> > > > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> > > would
> > > > be great if we could quantify this in some way.
> > > >
> > > > What do you think?
> > > >
> > > > Ismael
> > > >
> > > > [1] https://java.com/en/download/faq/java_7.xml
> > > > [2]
> https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > > > [3] http://openjdk.java.net/projects/jdk9/
> > > > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > > > [5]
> > https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > > > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > > > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > > > [8] https://webtide.com/jetty-9-3-features/
> > > > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > > > [10]
> > > >
> > > >
> > >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > > > [11] http://markmail.org/message/l7s276y3xkga2eqf
> > > >
> > >
> >
>


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Craig Swift
+1

Craig J. Swift
Principal Software Engineer - Data Pipeline
ReturnPath Inc.
Work: 303-999-3220 Cell: 720-560-7038

On Thu, Jun 16, 2016 at 2:50 PM, Henry Cai 
wrote:

> +1 for Lambda expression.
>
> On Thu, Jun 16, 2016 at 1:48 PM, Rajiv Kurian  wrote:
>
> > +1
> >
> > On Thu, Jun 16, 2016 at 1:45 PM, Ismael Juma  wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a discussion on making Java 8 a minimum
> requirement
> > > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now).
> This
> > > is the first discussion on the topic so the idea is to understand how
> > > people feel about it. If people feel it's too soon, then we can pick up
> > the
> > > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > > positive, I will start a vote thread.
> > >
> > > Let's start with some dates. Java 7 hasn't received public updates
> since
> > > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> > scheduled
> > > to be released in March 2017[3].
> > >
> > > The first argument for dropping support for Java 7 is that the last
> > public
> > > release by Oracle contains a large number of known security
> > > vulnerabilities. The effectiveness of Kafka's security features is
> > reduced
> > > if the underlying runtime is not itself secure.
> > >
> > > The second argument for moving to Java 8 is that it adds a number of
> > > compelling features:
> > >
> > > * Lambda expressions and method references (particularly useful for the
> > > Kafka Streams DSL)
> > > * Default methods (very useful for maintaining compatibility when
> adding
> > > methods to interfaces)
> > > * java.util.stream (helpful for making collection transformations more
> > > concise)
> > > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder,
> LongAccumulator)
> > > * Other nice things: SplittableRandom, Optional (and many others I have
> > not
> > > mentioned)
> > >
> > > The third argument is that it will simplify our testing matrix, we
> won't
> > > have to test with Java 7 any longer (this is particularly useful for
> > system
> > > tests that take hours to run). It will also make it easier to support
> > Scala
> > > 2.12, which requires Java 8.
> > >
> > > The fourth argument is that many other open-source projects have taken
> > the
> > > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop
> 3[7],
> > > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android
> will
> > > support Java 8 in the next version (although it will take a while
> before
> > > most phones will use that version sadly). This reduces (but does not
> > > eliminate) the chance that we would be the first project that would
> > cause a
> > > user to consider a Java upgrade.
> > >
> > > The main argument for not making the change is that a reasonable number
> > of
> > > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > > More specifically, we care about the subset who would be able to
> upgrade
> > to
> > > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> > would
> > > be great if we could quantify this in some way.
> > >
> > > What do you think?
> > >
> > > Ismael
> > >
> > > [1] https://java.com/en/download/faq/java_7.xml
> > > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > > [3] http://openjdk.java.net/projects/jdk9/
> > > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > > [5]
> https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > > [8] https://webtide.com/jetty-9-3-features/
> > > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > > [10]
> > >
> > >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > > [11] http://markmail.org/message/l7s276y3xkga2eqf
> > >
> >
>


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Rajiv Kurian
+1

On Thu, Jun 16, 2016 at 1:45 PM, Ismael Juma  wrote:

> Hi all,
>
> I would like to start a discussion on making Java 8 a minimum requirement
> for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> is the first discussion on the topic so the idea is to understand how
> people feel about it. If people feel it's too soon, then we can pick up the
> conversation again after Kafka 0.10.1.0. If the feedback is mostly
> positive, I will start a vote thread.
>
> Let's start with some dates. Java 7 hasn't received public updates since
> April 2015[1], Java 8 was released in March 2014[2] and Java 9 is scheduled
> to be released in March 2017[3].
>
> The first argument for dropping support for Java 7 is that the last public
> release by Oracle contains a large number of known security
> vulnerabilities. The effectiveness of Kafka's security features is reduced
> if the underlying runtime is not itself secure.
>
> The second argument for moving to Java 8 is that it adds a number of
> compelling features:
>
> * Lambda expressions and method references (particularly useful for the
> Kafka Streams DSL)
> * Default methods (very useful for maintaining compatibility when adding
> methods to interfaces)
> * java.util.stream (helpful for making collection transformations more
> concise)
> * Lots of improvements to java.util.concurrent (CompletableFuture,
> DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> * Other nice things: SplittableRandom, Optional (and many others I have not
> mentioned)
>
> The third argument is that it will simplify our testing matrix, we won't
> have to test with Java 7 any longer (this is particularly useful for system
> tests that take hours to run). It will also make it easier to support Scala
> 2.12, which requires Java 8.
>
> The fourth argument is that many other open-source projects have taken the
> leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> support Java 8 in the next version (although it will take a while before
> most phones will use that version sadly). This reduces (but does not
> eliminate) the chance that we would be the first project that would cause a
> user to consider a Java upgrade.
>
> The main argument for not making the change is that a reasonable number of
> users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> More specifically, we care about the subset who would be able to upgrade to
> Kafka 0.10.1.0, but would not be able to upgrade the Java version. It would
> be great if we could quantify this in some way.
>
> What do you think?
>
> Ismael
>
> [1] https://java.com/en/download/faq/java_7.xml
> [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> [3] http://openjdk.java.net/projects/jdk9/
> [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> [7] https://issues.apache.org/jira/browse/HADOOP-11858
> [8] https://webtide.com/jetty-9-3-features/
> [9] http://markmail.org/message/l7s276y3xkga2eqf
> [10]
>
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> [11] http://markmail.org/message/l7s276y3xkga2eqf
>


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Philippe Derome
I strongly support motion having difficulty running (Apache Kafka as
opposed to Confluent) Stream examples with JDK 8 today.
On 16 Jun 2016 4:46 p.m., "Ismael Juma"  wrote:

> Hi all,
>
> I would like to start a discussion on making Java 8 a minimum requirement
> for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> is the first discussion on the topic so the idea is to understand how
> people feel about it. If people feel it's too soon, then we can pick up the
> conversation again after Kafka 0.10.1.0. If the feedback is mostly
> positive, I will start a vote thread.
>
> Let's start with some dates. Java 7 hasn't received public updates since
> April 2015[1], Java 8 was released in March 2014[2] and Java 9 is scheduled
> to be released in March 2017[3].
>
> The first argument for dropping support for Java 7 is that the last public
> release by Oracle contains a large number of known security
> vulnerabilities. The effectiveness of Kafka's security features is reduced
> if the underlying runtime is not itself secure.
>
> The second argument for moving to Java 8 is that it adds a number of
> compelling features:
>
> * Lambda expressions and method references (particularly useful for the
> Kafka Streams DSL)
> * Default methods (very useful for maintaining compatibility when adding
> methods to interfaces)
> * java.util.stream (helpful for making collection transformations more
> concise)
> * Lots of improvements to java.util.concurrent (CompletableFuture,
> DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> * Other nice things: SplittableRandom, Optional (and many others I have not
> mentioned)
>
> The third argument is that it will simplify our testing matrix, we won't
> have to test with Java 7 any longer (this is particularly useful for system
> tests that take hours to run). It will also make it easier to support Scala
> 2.12, which requires Java 8.
>
> The fourth argument is that many other open-source projects have taken the
> leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> support Java 8 in the next version (although it will take a while before
> most phones will use that version sadly). This reduces (but does not
> eliminate) the chance that we would be the first project that would cause a
> user to consider a Java upgrade.
>
> The main argument for not making the change is that a reasonable number of
> users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> More specifically, we care about the subset who would be able to upgrade to
> Kafka 0.10.1.0, but would not be able to upgrade the Java version. It would
> be great if we could quantify this in some way.
>
> What do you think?
>
> Ismael
>
> [1] https://java.com/en/download/faq/java_7.xml
> [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> [3] http://openjdk.java.net/projects/jdk9/
> [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> [7] https://issues.apache.org/jira/browse/HADOOP-11858
> [8] https://webtide.com/jetty-9-3-features/
> [9] http://markmail.org/message/l7s276y3xkga2eqf
> [10]
>
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> [11] http://markmail.org/message/l7s276y3xkga2eqf
>


[jira] [Created] (KAFKA-3858) Add functions to print stream topologies

2016-06-16 Thread Roger Hoover (JIRA)
Roger Hoover created KAFKA-3858:
---

 Summary: Add functions to print stream topologies
 Key: KAFKA-3858
 URL: https://issues.apache.org/jira/browse/KAFKA-3858
 Project: Kafka
  Issue Type: New Feature
  Components: streams
Affects Versions: 0.10.0.0
Reporter: Roger Hoover
Assignee: Guozhang Wang


For debugging and development, it would be very useful to be able to print 
Kafka streams topologies.  At a minimum, it would be great to be able to see 
the logical topology including with Kafka topics linked by sub-topologies.  I 
think that this information does not depend on partitioning.  For more detail, 
it would be great to be able to print the same logical topology but also 
showing number of tasks (an perhaps task ids?).  Finally, it would be great to 
show the physical topology after the tasks have been mapped to JVMs + threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: avro-console-producer issue for Kafka Stream example PageViewRegionLambdaExample

2016-06-16 Thread Philippe Derome
Sorry, this thread belongs to confluent not Kafka.apache.org as it pertains
to Confluent's examples.

On Thu, Jun 16, 2016 at 2:33 PM, Philippe Derome  wrote:

> I am able to produce Avro from producer for pageview.avsc but not for
> userprofile.avsc. Here are two samples below, first one leaves out Avro
> optional experience, second one specifies it.
>
> Note: I do not register using REST curl calls schemas to schema-registry
> as I assume that kafka-avro-console-producer takes care of that.
>
> First case has Avro disagreeing that experience can be optional with an
> exception:
>
> AvroTypeException: Expected field name not found: experience
>
> Second case generates another AvroTypeException complaining about the
> union type:
>
> Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
> VALUE_STRING
>
> ./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
> UserProfile --property value.schema='{
>
>  "type": "record",
>
>  "name": "UserProfile",
>
>  "fields": [
>
>  {"name": "experience", "type": ["string", "null"]},
>
>  {"name": "region", "type": "string"}
>
>  ]
>
> }'
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> {"region":"US"}
>
> org.apache.kafka.common.errors.SerializationException: Error deserializing
> json {"region":"US"} to Avro of schema
> {"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}
>
> Caused by: org.apache.avro.AvroTypeException: Expected field name not
> found: experience
>
> at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
>
> at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:429)
>
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
>
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
>
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
>
> at
> io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:189)
>
> at
> io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:157)
>
> at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)
>
> at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
>
>
> ./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
> UserProfile --property value.schema='{
>
>  "type": "record",
>
>  "name": "UserProfile",
>
>  "fields": [
>
>  {"name": "experience", "type": ["string", "null"]},
>
>  {"name": "region", "type": "string"}
>
>  ]
>
> }'
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> {"region":"US","experience":"scala"}
>
> org.apache.kafka.common.errors.SerializationException: Error deserializing
> json {"region":"US","experience":"scala"} to Avro of schema
> {"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}
>
> Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
> VALUE_STRING
>
> at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:697)
>
> at org.apache.avro.io.JsonDecoder.readI

Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Jeff Klukas
Would the move to Java 8 be for all modules? I'd have some concern about
removing Java 7 compatibility for kafka-clients and for kafka streams
(though less so since it's still so new). I don't know how hard it will be
to transition a Scala 2.11 application to Scala 2.12. Are we comfortable
with the idea of applications stuck on Scala 2.11 or otherwise unable to
update to Java 8 not having access to new client releases?

On Thu, Jun 16, 2016 at 5:05 PM, Philippe Derome  wrote:

> I strongly support motion having difficulty running (Apache Kafka as
> opposed to Confluent) Stream examples with JDK 8 today.
> On 16 Jun 2016 4:46 p.m., "Ismael Juma"  wrote:
>
> > Hi all,
> >
> > I would like to start a discussion on making Java 8 a minimum requirement
> > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> > is the first discussion on the topic so the idea is to understand how
> > people feel about it. If people feel it's too soon, then we can pick up
> the
> > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > positive, I will start a vote thread.
> >
> > Let's start with some dates. Java 7 hasn't received public updates since
> > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> scheduled
> > to be released in March 2017[3].
> >
> > The first argument for dropping support for Java 7 is that the last
> public
> > release by Oracle contains a large number of known security
> > vulnerabilities. The effectiveness of Kafka's security features is
> reduced
> > if the underlying runtime is not itself secure.
> >
> > The second argument for moving to Java 8 is that it adds a number of
> > compelling features:
> >
> > * Lambda expressions and method references (particularly useful for the
> > Kafka Streams DSL)
> > * Default methods (very useful for maintaining compatibility when adding
> > methods to interfaces)
> > * java.util.stream (helpful for making collection transformations more
> > concise)
> > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> > * Other nice things: SplittableRandom, Optional (and many others I have
> not
> > mentioned)
> >
> > The third argument is that it will simplify our testing matrix, we won't
> > have to test with Java 7 any longer (this is particularly useful for
> system
> > tests that take hours to run). It will also make it easier to support
> Scala
> > 2.12, which requires Java 8.
> >
> > The fourth argument is that many other open-source projects have taken
> the
> > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> > support Java 8 in the next version (although it will take a while before
> > most phones will use that version sadly). This reduces (but does not
> > eliminate) the chance that we would be the first project that would
> cause a
> > user to consider a Java upgrade.
> >
> > The main argument for not making the change is that a reasonable number
> of
> > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > More specifically, we care about the subset who would be able to upgrade
> to
> > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> would
> > be great if we could quantify this in some way.
> >
> > What do you think?
> >
> > Ismael
> >
> > [1] https://java.com/en/download/faq/java_7.xml
> > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > [3] http://openjdk.java.net/projects/jdk9/
> > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > [8] https://webtide.com/jetty-9-3-features/
> > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > [10]
> >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > [11] http://markmail.org/message/l7s276y3xkga2eqf
> >
>


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Grant Henke
I definitely think its time, but I am not sure if 0.10.1.0 is the release
version to drop support in.

Would it make sense to drop Java 1.7 and Scala 2.10 support in Kafka 0.11
to maintain upgrade compatibility throughout the 0.10.x series?

As a side note, I personally like the pattern of always supporting 2 java
and 2 Scala versions for any given release because it give the user a way
to upgrade one or the other independently. This creates a relatively
"static" testing matrix as well.





On Thu, Jun 16, 2016 at 3:59 PM, Adam Kunicki  wrote:

> +1
>
> Adam Kunicki
> StreamSets | Field Engineer
> mobile: 415.890.DATA (3282) | linkedin
> <
> https://mailtrack.io/trace/link/3e560367e0508b2f285512f39bd070275e70f571?url=http%3A%2F%2Fwww.adamkunicki.com&signature=aabcc9d816de2753
> >
>
> On Thu, Jun 16, 2016 at 1:56 PM, Craig Swift <
> craig.sw...@returnpath.com.invalid> wrote:
>
> > +1
> >
> > Craig J. Swift
> > Principal Software Engineer - Data Pipeline
> > ReturnPath Inc.
> > Work: 303-999-3220 Cell: 720-560-7038
> >
> > On Thu, Jun 16, 2016 at 2:50 PM, Henry Cai 
> > wrote:
> >
> > > +1 for Lambda expression.
> > >
> > > On Thu, Jun 16, 2016 at 1:48 PM, Rajiv Kurian 
> > wrote:
> > >
> > > > +1
> > > >
> > > > On Thu, Jun 16, 2016 at 1:45 PM, Ismael Juma 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to start a discussion on making Java 8 a minimum
> > > requirement
> > > > > for Kafka's next feature release (let's say Kafka 0.10.1.0 for
> now).
> > > This
> > > > > is the first discussion on the topic so the idea is to understand
> how
> > > > > people feel about it. If people feel it's too soon, then we can
> pick
> > up
> > > > the
> > > > > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > > > > positive, I will start a vote thread.
> > > > >
> > > > > Let's start with some dates. Java 7 hasn't received public updates
> > > since
> > > > > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> > > > scheduled
> > > > > to be released in March 2017[3].
> > > > >
> > > > > The first argument for dropping support for Java 7 is that the last
> > > > public
> > > > > release by Oracle contains a large number of known security
> > > > > vulnerabilities. The effectiveness of Kafka's security features is
> > > > reduced
> > > > > if the underlying runtime is not itself secure.
> > > > >
> > > > > The second argument for moving to Java 8 is that it adds a number
> of
> > > > > compelling features:
> > > > >
> > > > > * Lambda expressions and method references (particularly useful for
> > the
> > > > > Kafka Streams DSL)
> > > > > * Default methods (very useful for maintaining compatibility when
> > > adding
> > > > > methods to interfaces)
> > > > > * java.util.stream (helpful for making collection transformations
> > more
> > > > > concise)
> > > > > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > > > > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder,
> > > LongAccumulator)
> > > > > * Other nice things: SplittableRandom, Optional (and many others I
> > have
> > > > not
> > > > > mentioned)
> > > > >
> > > > > The third argument is that it will simplify our testing matrix, we
> > > won't
> > > > > have to test with Java 7 any longer (this is particularly useful
> for
> > > > system
> > > > > tests that take hours to run). It will also make it easier to
> support
> > > > Scala
> > > > > 2.12, which requires Java 8.
> > > > >
> > > > > The fourth argument is that many other open-source projects have
> > taken
> > > > the
> > > > > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop
> > > 3[7],
> > > > > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even
> Android
> > > will
> > > > > support Java 8 in the next version (although it will take a while
> > > before
> > > > > most phones will use that version sadly). This reduces (but does
> not
> > > > > eliminate) the chance that we would be the first project that would
> > > > cause a
> > > > > user to consider a Java upgrade.
> > > > >
> > > > > The main argument for not making the change is that a reasonable
> > number
> > > > of
> > > > > users may still be using Java 7 by the time Kafka 0.10.1.0 is
> > released.
> > > > > More specifically, we care about the subset who would be able to
> > > upgrade
> > > > to
> > > > > Kafka 0.10.1.0, but would not be able to upgrade the Java version.
> It
> > > > would
> > > > > be great if we could quantify this in some way.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Ismael
> > > > >
> > > > > [1] https://java.com/en/download/faq/java_7.xml
> > > > > [2]
> > https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > > > > [3] http://openjdk.java.net/projects/jdk9/
> > > > > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > > > > [5]
> > > https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > > > > [6] http://akka.io/news/2015/09/30/akka-2.4.0-releas

Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Bill Bejeck
+1 for lambda expressions

On Thu, Jun 16, 2016 at 5:13 PM, Grant Henke  wrote:

> I definitely think its time, but I am not sure if 0.10.1.0 is the release
> version to drop support in.
>
> Would it make sense to drop Java 1.7 and Scala 2.10 support in Kafka 0.11
> to maintain upgrade compatibility throughout the 0.10.x series?
>
> As a side note, I personally like the pattern of always supporting 2 java
> and 2 Scala versions for any given release because it give the user a way
> to upgrade one or the other independently. This creates a relatively
> "static" testing matrix as well.
>
>
>
>
>
> On Thu, Jun 16, 2016 at 3:59 PM, Adam Kunicki  wrote:
>
> > +1
> >
> > Adam Kunicki
> > StreamSets | Field Engineer
> > mobile: 415.890.DATA (3282) | linkedin
> > <
> >
> https://mailtrack.io/trace/link/3e560367e0508b2f285512f39bd070275e70f571?url=http%3A%2F%2Fwww.adamkunicki.com&signature=aabcc9d816de2753
> > >
> >
> > On Thu, Jun 16, 2016 at 1:56 PM, Craig Swift <
> > craig.sw...@returnpath.com.invalid> wrote:
> >
> > > +1
> > >
> > > Craig J. Swift
> > > Principal Software Engineer - Data Pipeline
> > > ReturnPath Inc.
> > > Work: 303-999-3220 Cell: 720-560-7038
> > >
> > > On Thu, Jun 16, 2016 at 2:50 PM, Henry Cai  >
> > > wrote:
> > >
> > > > +1 for Lambda expression.
> > > >
> > > > On Thu, Jun 16, 2016 at 1:48 PM, Rajiv Kurian 
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Thu, Jun 16, 2016 at 1:45 PM, Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I would like to start a discussion on making Java 8 a minimum
> > > > requirement
> > > > > > for Kafka's next feature release (let's say Kafka 0.10.1.0 for
> > now).
> > > > This
> > > > > > is the first discussion on the topic so the idea is to understand
> > how
> > > > > > people feel about it. If people feel it's too soon, then we can
> > pick
> > > up
> > > > > the
> > > > > > conversation again after Kafka 0.10.1.0. If the feedback is
> mostly
> > > > > > positive, I will start a vote thread.
> > > > > >
> > > > > > Let's start with some dates. Java 7 hasn't received public
> updates
> > > > since
> > > > > > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> > > > > scheduled
> > > > > > to be released in March 2017[3].
> > > > > >
> > > > > > The first argument for dropping support for Java 7 is that the
> last
> > > > > public
> > > > > > release by Oracle contains a large number of known security
> > > > > > vulnerabilities. The effectiveness of Kafka's security features
> is
> > > > > reduced
> > > > > > if the underlying runtime is not itself secure.
> > > > > >
> > > > > > The second argument for moving to Java 8 is that it adds a number
> > of
> > > > > > compelling features:
> > > > > >
> > > > > > * Lambda expressions and method references (particularly useful
> for
> > > the
> > > > > > Kafka Streams DSL)
> > > > > > * Default methods (very useful for maintaining compatibility when
> > > > adding
> > > > > > methods to interfaces)
> > > > > > * java.util.stream (helpful for making collection transformations
> > > more
> > > > > > concise)
> > > > > > * Lots of improvements to java.util.concurrent
> (CompletableFuture,
> > > > > > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder,
> > > > LongAccumulator)
> > > > > > * Other nice things: SplittableRandom, Optional (and many others
> I
> > > have
> > > > > not
> > > > > > mentioned)
> > > > > >
> > > > > > The third argument is that it will simplify our testing matrix,
> we
> > > > won't
> > > > > > have to test with Java 7 any longer (this is particularly useful
> > for
> > > > > system
> > > > > > tests that take hours to run). It will also make it easier to
> > support
> > > > > Scala
> > > > > > 2.12, which requires Java 8.
> > > > > >
> > > > > > The fourth argument is that many other open-source projects have
> > > taken
> > > > > the
> > > > > > leap already. Examples are Cassandra[4], Lucene[5], Akka[6],
> Hadoop
> > > > 3[7],
> > > > > > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even
> > Android
> > > > will
> > > > > > support Java 8 in the next version (although it will take a while
> > > > before
> > > > > > most phones will use that version sadly). This reduces (but does
> > not
> > > > > > eliminate) the chance that we would be the first project that
> would
> > > > > cause a
> > > > > > user to consider a Java upgrade.
> > > > > >
> > > > > > The main argument for not making the change is that a reasonable
> > > number
> > > > > of
> > > > > > users may still be using Java 7 by the time Kafka 0.10.1.0 is
> > > released.
> > > > > > More specifically, we care about the subset who would be able to
> > > > upgrade
> > > > > to
> > > > > > Kafka 0.10.1.0, but would not be able to upgrade the Java
> version.
> > It
> > > > > would
> > > > > > be great if we could quantify this in some way.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > [1] https://java.com/en

[jira] [Updated] (KAFKA-3858) Add functions to print stream topologies

2016-06-16 Thread Eno Thereska (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska updated KAFKA-3858:

Affects Version/s: (was: 0.10.0.0)
   0.10.1.0

> Add functions to print stream topologies
> 
>
> Key: KAFKA-3858
> URL: https://issues.apache.org/jira/browse/KAFKA-3858
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Roger Hoover
>Assignee: Eno Thereska
>
> For debugging and development, it would be very useful to be able to print 
> Kafka streams topologies.  At a minimum, it would be great to be able to see 
> the logical topology including with Kafka topics linked by sub-topologies.  I 
> think that this information does not depend on partitioning.  For more 
> detail, it would be great to be able to print the same logical topology but 
> also showing number of tasks (an perhaps task ids?).  Finally, it would be 
> great to show the physical topology after the tasks have been mapped to JVMs 
> + threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (KAFKA-3858) Add functions to print stream topologies

2016-06-16 Thread Eno Thereska (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska reassigned KAFKA-3858:
---

Assignee: Eno Thereska  (was: Guozhang Wang)

> Add functions to print stream topologies
> 
>
> Key: KAFKA-3858
> URL: https://issues.apache.org/jira/browse/KAFKA-3858
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Roger Hoover
>Assignee: Eno Thereska
>
> For debugging and development, it would be very useful to be able to print 
> Kafka streams topologies.  At a minimum, it would be great to be able to see 
> the logical topology including with Kafka topics linked by sub-topologies.  I 
> think that this information does not depend on partitioning.  For more 
> detail, it would be great to be able to print the same logical topology but 
> also showing number of tasks (an perhaps task ids?).  Finally, it would be 
> great to show the physical topology after the tasks have been mapped to JVMs 
> + threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Ismael Juma
On Thu, Jun 16, 2016 at 11:13 PM, Stephen Boesch  wrote:

> @Jeff Klukas What is the concern about scala 2.11 vs 2.12?   2.11 runs on
> both java7 and java8
>

Scala 2.10.5 and 2.10.6 also support Java 8 for what it's worth.

Ismael


Build failed in Jenkins: kafka-trunk-jdk7 #1369

2016-06-16 Thread Apache Jenkins Server
See 

Changes:

[wangguoz] KAFKA-3561: Auto create through topic for KStream aggregation and 
join

--
[...truncated 10553 lines...]
org.apache.kafka.common.config.ConfigDefTest > testValidate PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultString STARTED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultString PASSED

org.apache.kafka.common.config.ConfigDefTest > testSslPasswords STARTED

org.apache.kafka.common.config.ConfigDefTest > testSslPasswords PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefault STARTED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > testMissingRequired STARTED

org.apache.kafka.common.config.ConfigDefTest > testMissingRequired PASSED

org.apache.kafka.common.config.ConfigDefTest > testNullDefaultWithValidator 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testNullDefaultWithValidator 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testDefinedTwice STARTED

org.apache.kafka.common.config.ConfigDefTest > testDefinedTwice PASSED

org.apache.kafka.common.config.ConfigDefTest > testBadInputs STARTED

org.apache.kafka.common.config.ConfigDefTest > testBadInputs PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidateMissingConfigKey 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testValidateMissingConfigKey 
PASSED

org.apache.kafka.common.config.AbstractConfigTest > testOriginalsWithPrefix 
STARTED

org.apache.kafka.common.config.AbstractConfigTest > testOriginalsWithPrefix 
PASSED

org.apache.kafka.common.config.AbstractConfigTest > testConfiguredInstances 
STARTED

org.apache.kafka.common.config.AbstractConfigTest > testConfiguredInstances 
PASSED

org.apache.kafka.common.config.AbstractConfigTest > testUnused STARTED

org.apache.kafka.common.config.AbstractConfigTest > testUnused PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testUnauthenticatedApiVersionsRequestOverPlaintext STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testUnauthenticatedApiVersionsRequestOverPlaintext PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMechanismPluggability STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMechanismPluggability PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingPasswordSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingPasswordSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidLoginModule STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidLoginModule PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidMechanism STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidMechanism PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testDisabledMechanism STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testDisabledMechanism PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testPacketSizeTooBig STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testPacketSizeTooBig PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidUsernameSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidUsernameSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMultipleServerMechanisms STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMultipleServerMechanisms PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testValidSaslPlainOverPlaintext STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testValidSaslPlainOverPlaintext PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testValidSaslPlainOverSsl STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testValidSaslPlainOverSsl PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidApiVersionsRequestSequence STARTED

or

[jira] [Updated] (KAFKA-3837) Report the name of the blocking thread when throwing ConcurrentModificationException

2016-06-16 Thread Bharat Viswanadham (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated KAFKA-3837:
--
Status: Patch Available  (was: In Progress)

> Report the name of the blocking thread when throwing 
> ConcurrentModificationException
> 
>
> Key: KAFKA-3837
> URL: https://issues.apache.org/jira/browse/KAFKA-3837
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Simon Cooper
>Assignee: Bharat Viswanadham
>Priority: Minor
>
> {{KafkaConsumer.acquire}} throws {{ConcurrentModificationException}} if the 
> current thread it does not match the {{currentThread}} field. It would be 
> useful if the name of the other thread was included in the exception message 
> to help debug the problem when this exception occurs.
> As it stands, it can be really difficult to work out what's going wrong when 
> there's several threads all accessing the consumer at the same time, and your 
> existing exclusive access logic doesn't seem to be working as it should.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Ismael Juma
Hi Grant,

Comments below.

On Thu, Jun 16, 2016 at 11:13 PM, Grant Henke  wrote:

> Would it make sense to drop Java 1.7 and Scala 2.10 support in Kafka 0.11
> to maintain upgrade compatibility throughout the 0.10.x series?
>

The issue is that we don't know when Kafka 0.11 will be released. Also, I
think it would be better not to bump the `major` version just for this
change (if we have to do it for some other reason, then that's even
better). The reason is that your code will work without change, you just
have to make sure you have the right environment. And many (most?) already
do. For what it's worth, this is the same approach Akka took with their
2.4.0 release (the first to require Java 8).

I would prefer to have the discussion about Scala 2.10 separately. Maybe
when the 2.12 release is nearer.

As a side note, I personally like the pattern of always supporting 2 java
> and 2 Scala versions for any given release because it give the user a way
> to upgrade one or the other independently. This creates a relatively
> "static" testing matrix as well.
>

That is my preference as well. I am raising this sooner this time because
of what I consider exceptional conditions. The long period between Java 7
and 8 combined with the delay of Java 9 means that Java 7 has been EOL'd
for 14 months and we are only starting the 0.10.1.0 cycle. And Java 8 has
some high-impact features for Kafka Streams (it is possible to support
lambdas by having our own SAMs, but this is painful for a number of
reasons).

Ismael


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Stephen Boesch
@Jeff Klukas What is the concern about scala 2.11 vs 2.12?   2.11 runs on
both java7 and java8

2016-06-16 14:12 GMT-07:00 Jeff Klukas :

> Would the move to Java 8 be for all modules? I'd have some concern about
> removing Java 7 compatibility for kafka-clients and for kafka streams
> (though less so since it's still so new). I don't know how hard it will be
> to transition a Scala 2.11 application to Scala 2.12. Are we comfortable
> with the idea of applications stuck on Scala 2.11 or otherwise unable to
> update to Java 8 not having access to new client releases?
>
> On Thu, Jun 16, 2016 at 5:05 PM, Philippe Derome 
> wrote:
>
> > I strongly support motion having difficulty running (Apache Kafka as
> > opposed to Confluent) Stream examples with JDK 8 today.
> > On 16 Jun 2016 4:46 p.m., "Ismael Juma"  wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a discussion on making Java 8 a minimum
> requirement
> > > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now).
> This
> > > is the first discussion on the topic so the idea is to understand how
> > > people feel about it. If people feel it's too soon, then we can pick up
> > the
> > > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > > positive, I will start a vote thread.
> > >
> > > Let's start with some dates. Java 7 hasn't received public updates
> since
> > > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> > scheduled
> > > to be released in March 2017[3].
> > >
> > > The first argument for dropping support for Java 7 is that the last
> > public
> > > release by Oracle contains a large number of known security
> > > vulnerabilities. The effectiveness of Kafka's security features is
> > reduced
> > > if the underlying runtime is not itself secure.
> > >
> > > The second argument for moving to Java 8 is that it adds a number of
> > > compelling features:
> > >
> > > * Lambda expressions and method references (particularly useful for the
> > > Kafka Streams DSL)
> > > * Default methods (very useful for maintaining compatibility when
> adding
> > > methods to interfaces)
> > > * java.util.stream (helpful for making collection transformations more
> > > concise)
> > > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder,
> LongAccumulator)
> > > * Other nice things: SplittableRandom, Optional (and many others I have
> > not
> > > mentioned)
> > >
> > > The third argument is that it will simplify our testing matrix, we
> won't
> > > have to test with Java 7 any longer (this is particularly useful for
> > system
> > > tests that take hours to run). It will also make it easier to support
> > Scala
> > > 2.12, which requires Java 8.
> > >
> > > The fourth argument is that many other open-source projects have taken
> > the
> > > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop
> 3[7],
> > > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android
> will
> > > support Java 8 in the next version (although it will take a while
> before
> > > most phones will use that version sadly). This reduces (but does not
> > > eliminate) the chance that we would be the first project that would
> > cause a
> > > user to consider a Java upgrade.
> > >
> > > The main argument for not making the change is that a reasonable number
> > of
> > > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > > More specifically, we care about the subset who would be able to
> upgrade
> > to
> > > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> > would
> > > be great if we could quantify this in some way.
> > >
> > > What do you think?
> > >
> > > Ismael
> > >
> > > [1] https://java.com/en/download/faq/java_7.xml
> > > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > > [3] http://openjdk.java.net/projects/jdk9/
> > > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > > [5]
> https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > > [8] https://webtide.com/jetty-9-3-features/
> > > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > > [10]
> > >
> > >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > > [11] http://markmail.org/message/l7s276y3xkga2eqf
> > >
> >
>


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Stephen Powis
+1

On Thu, Jun 16, 2016 at 5:05 PM, Philippe Derome  wrote:

> I strongly support motion having difficulty running (Apache Kafka as
> opposed to Confluent) Stream examples with JDK 8 today.
> On 16 Jun 2016 4:46 p.m., "Ismael Juma"  wrote:
>
> > Hi all,
> >
> > I would like to start a discussion on making Java 8 a minimum requirement
> > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> > is the first discussion on the topic so the idea is to understand how
> > people feel about it. If people feel it's too soon, then we can pick up
> the
> > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > positive, I will start a vote thread.
> >
> > Let's start with some dates. Java 7 hasn't received public updates since
> > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> scheduled
> > to be released in March 2017[3].
> >
> > The first argument for dropping support for Java 7 is that the last
> public
> > release by Oracle contains a large number of known security
> > vulnerabilities. The effectiveness of Kafka's security features is
> reduced
> > if the underlying runtime is not itself secure.
> >
> > The second argument for moving to Java 8 is that it adds a number of
> > compelling features:
> >
> > * Lambda expressions and method references (particularly useful for the
> > Kafka Streams DSL)
> > * Default methods (very useful for maintaining compatibility when adding
> > methods to interfaces)
> > * java.util.stream (helpful for making collection transformations more
> > concise)
> > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> > * Other nice things: SplittableRandom, Optional (and many others I have
> not
> > mentioned)
> >
> > The third argument is that it will simplify our testing matrix, we won't
> > have to test with Java 7 any longer (this is particularly useful for
> system
> > tests that take hours to run). It will also make it easier to support
> Scala
> > 2.12, which requires Java 8.
> >
> > The fourth argument is that many other open-source projects have taken
> the
> > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> > support Java 8 in the next version (although it will take a while before
> > most phones will use that version sadly). This reduces (but does not
> > eliminate) the chance that we would be the first project that would
> cause a
> > user to consider a Java upgrade.
> >
> > The main argument for not making the change is that a reasonable number
> of
> > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > More specifically, we care about the subset who would be able to upgrade
> to
> > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> would
> > be great if we could quantify this in some way.
> >
> > What do you think?
> >
> > Ismael
> >
> > [1] https://java.com/en/download/faq/java_7.xml
> > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > [3] http://openjdk.java.net/projects/jdk9/
> > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > [8] https://webtide.com/jetty-9-3-features/
> > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > [10]
> >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > [11] http://markmail.org/message/l7s276y3xkga2eqf
> >
>


Re: FYI - Apache ZooKeeper Backup, a Treatise

2016-06-16 Thread Jordan Zimmerman
Contrary to recommendations everywhere, my experience is that almost everyone 
is storing source of truth data in ZooKeeper. It’s just too tempting. You have 
a distributed file system just sitting there and it’s too easy to use. You get 
a lot of great features like watches, etc. People are using it to store 
configuration data, sequence numbers, etc. They are storing these things 
without a good means of reproducing them in case of a catastrophic outage. 
Further, I’ve heard of several orgs who just back up the transaction logs and 
think they can restore them for DR. Anyway, that’s the genesis of my blog post.

-Jordan

> On Jun 16, 2016, at 2:39 PM, Chris Nauroth  wrote:
> 
> Yes, thank you to Jordan for the article!
> 
> Like Flavio, I personally have never come across the requirement for
> ZooKeeper backups.  I've generally followed the pattern that data stored
> in ZooKeeper is truly transient, and applications are built either to
> tolerate loss of that data or reconstruct it from first principles if it
> goes missing.  Adding observers in a second data center would give a
> rudimentary approximation of off-site backup in the case of a data center
> disaster, with the usual caveats around propagation delays.
> 
> Jordan, I'd be curious if you can share more specific details about the
> kind of data that you have that necessitates a backup/restore.  (If you're
> not at liberty to share this, then I can understand that.)  It might
> inform if we have a motivating use case for backup/restore features within
> ZooKeeper, such as some of the transaction log filtering that the article
> mentions.
> 
> --Chris Nauroth
> 
> 
> 
> 
> On 6/16/16, 1:03 AM, "Flavio Junqueira"  wrote:
> 
>> Great write-up, Jordan, thanks!
>> 
>> Whether to backup zk data or not is possibly an open topic for this
>> community, even though we have discussed it at times. My sense has been
>> that precisely because of the issues you mention in your post, it is
>> typically best to have a way to recreate its data upon a disaster rather
>> than backup the data. I think there could be three general scenarios in
>> which folks would prefer to backup data, but you correct me if these
>> aren't accurate:
>> 
>> - The data in zk isn't elsewhere, so it can't be recreated: zk isn't a
>> regular database, so I'd think it is best not to store data and focus on
>> cluster data or metadata.
>> - There is a just a lot of data and I'd rather have a shorter time to
>> recover: zk in general shouldn't have that much data in db, but let's go
>> with the assumption that for the requirements of the application it is a
>> lot. For such a case, it probably depends on whether your application can
>> efficiently and effectively recover from a backup. Basically, as pointed
>> out in the post, the data could be inconsistent and cause trouble if you
>> don't think about the corner cases.
>> - The code to recreate the zk metadata for my application is super
>> complex: if you decide to code against zk, it is good to think whether
>> reconstructing in the case of a disaster is doable and if it is design
>> and implement to reconstruct the state upon a disaster.
>> 
>> Also, we typically provision enough replicas, often replicating across
>> data centers, to make sure that the data isn't all gone. Having more
>> replicas does not rule out completely the possibility of a disaster, but
>> in such rare cases we resort to the expensive path.
>> 
>> I personally have never worked with an application that was taking
>> backups of zk data in prod, so I'm really interested in what others
>> think. 
>> 
>> -Flavio
>> 
>> 
>>> On 16 Jun 2016, at 00:43, Jordan Zimmerman 
>>> wrote:
>>> 
>>> FYI - I wrote a blog about backing up ZooKeeper:
>>> 
>>> https://www.elastic.co/blog/zookeeper-backup-a-treatise
>>> 
>>> -Jordan
>> 
> 



Re: FYI - Apache ZooKeeper Backup, a Treatise

2016-06-16 Thread Ismael Juma
Hi Jordan,

Kafka stores ACLs as well as client and topic configs in ZooKeeper so that
lends credence to your argument, I think.

Ismael

On Thu, Jun 16, 2016 at 11:41 PM, Jordan Zimmerman <
jor...@jordanzimmerman.com> wrote:

> Contrary to recommendations everywhere, my experience is that almost
> everyone is storing source of truth data in ZooKeeper. It’s just too
> tempting. You have a distributed file system just sitting there and it’s
> too easy to use. You get a lot of great features like watches, etc. People
> are using it to store configuration data, sequence numbers, etc. They are
> storing these things without a good means of reproducing them in case of a
> catastrophic outage. Further, I’ve heard of several orgs who just back up
> the transaction logs and think they can restore them for DR. Anyway, that’s
> the genesis of my blog post.
>
> -Jordan
>
> > On Jun 16, 2016, at 2:39 PM, Chris Nauroth 
> wrote:
> >
> > Yes, thank you to Jordan for the article!
> >
> > Like Flavio, I personally have never come across the requirement for
> > ZooKeeper backups.  I've generally followed the pattern that data stored
> > in ZooKeeper is truly transient, and applications are built either to
> > tolerate loss of that data or reconstruct it from first principles if it
> > goes missing.  Adding observers in a second data center would give a
> > rudimentary approximation of off-site backup in the case of a data center
> > disaster, with the usual caveats around propagation delays.
> >
> > Jordan, I'd be curious if you can share more specific details about the
> > kind of data that you have that necessitates a backup/restore.  (If
> you're
> > not at liberty to share this, then I can understand that.)  It might
> > inform if we have a motivating use case for backup/restore features
> within
> > ZooKeeper, such as some of the transaction log filtering that the article
> > mentions.
> >
> > --Chris Nauroth
> >
> >
> >
> >
> > On 6/16/16, 1:03 AM, "Flavio Junqueira"  wrote:
> >
> >> Great write-up, Jordan, thanks!
> >>
> >> Whether to backup zk data or not is possibly an open topic for this
> >> community, even though we have discussed it at times. My sense has been
> >> that precisely because of the issues you mention in your post, it is
> >> typically best to have a way to recreate its data upon a disaster rather
> >> than backup the data. I think there could be three general scenarios in
> >> which folks would prefer to backup data, but you correct me if these
> >> aren't accurate:
> >>
> >> - The data in zk isn't elsewhere, so it can't be recreated: zk isn't a
> >> regular database, so I'd think it is best not to store data and focus on
> >> cluster data or metadata.
> >> - There is a just a lot of data and I'd rather have a shorter time to
> >> recover: zk in general shouldn't have that much data in db, but let's go
> >> with the assumption that for the requirements of the application it is a
> >> lot. For such a case, it probably depends on whether your application
> can
> >> efficiently and effectively recover from a backup. Basically, as pointed
> >> out in the post, the data could be inconsistent and cause trouble if you
> >> don't think about the corner cases.
> >> - The code to recreate the zk metadata for my application is super
> >> complex: if you decide to code against zk, it is good to think whether
> >> reconstructing in the case of a disaster is doable and if it is design
> >> and implement to reconstruct the state upon a disaster.
> >>
> >> Also, we typically provision enough replicas, often replicating across
> >> data centers, to make sure that the data isn't all gone. Having more
> >> replicas does not rule out completely the possibility of a disaster, but
> >> in such rare cases we resort to the expensive path.
> >>
> >> I personally have never worked with an application that was taking
> >> backups of zk data in prod, so I'm really interested in what others
> >> think.
> >>
> >> -Flavio
> >>
> >>
> >>> On 16 Jun 2016, at 00:43, Jordan Zimmerman  >
> >>> wrote:
> >>>
> >>> FYI - I wrote a blog about backing up ZooKeeper:
> >>>
> >>> https://www.elastic.co/blog/zookeeper-backup-a-treatise
> >>>
> >>> -Jordan
> >>
> >
>
>


[VOTE] KIP-4 Create Topics Schema

2016-06-16 Thread Grant Henke
I would like to initiate the voting process for the "KIP-4 Create Topics
Schema changes". This is not a vote for all of KIP-4, but specifically for
the create topics changes. I have included the exact changes below for
clarity:
>
> Create Topics Request (KAFKA-2945
> )
>
> CreateTopics Request (Version: 0) => [create_topic_requests] timeout
>   create_topic_requests => topic num_partitions replication_factor 
> [replica_assignment] [configs]
> topic => STRING
> num_partitions => INT32
> replication_factor => INT16
> replica_assignment => partition_id [replicas]
>   partition_id => INT32
>   replicas => INT32
> configs => config_key config_value
>   config_key => STRING
>   config_value => STRING
>   timeout => INT32
>
> CreateTopicsRequest is a batch request to initiate topic creation with
> either predefined or automatic replica assignment and optionally topic
> configuration.
>
> Request semantics:
>
>1. Must be sent to the controller broker
>2. If there are multiple instructions for the same topic in one
>request an InvalidRequestException will be logged on the broker and the
>client will be disconnected.
>   - This is because the list of topics is modeled server side as a
>   map with TopicName as the key
>3. The principal must be authorized to the "Create" Operation on the
>"Cluster" resource to create topics.
>   - Unauthorized requests will receive a ClusterAuthorizationException
>4.
>
>Only one from ReplicaAssignment or (num_partitions + replication_factor
>), can be defined in one instruction.
>- If both parameters are specified an InvalidRequestException will be
>   logged on the broker and the client will be disconnected.
>   - In the case ReplicaAssignment is defined number of partitions and
>   replicas will be calculated from the supplied replica_assignment.
>   - In the case of defined (num_partitions + replication_factor)
>   replica assignment will be automatically generated by the server.
>   - One or the other must be defined. The existing broker side auto
>   create defaults will not be used
>   (default.replication.factor, num.partitions). The client implementation 
> can
>   have defaults for these options when generating the messages.
>   - The first replica in [replicas] is assumed to be the preferred
>   leader. This matches current behavior elsewhere.
>5. Setting a timeout > 0 will allow the request to block until the
>topic metadata is "complete" on the controller node.
>   - Complete means the local topic metadata cache been completely
>   populated and all partitions have leaders
>  - The topic metadata is updated when the controller sends out
>  update metadata requests to the brokers
>   - If a timeout error occurs, the topic could still be created
>   successfully at a later time. Its up to the client to query for the 
> state
>   at that point.
>6. Setting a timeout <= 0 will validate arguments and trigger the
>create topics and return immediately.
>   - This is essentially the fully asynchronous mode we have in the
>   Zookeeper tools today.
>   - The error code in the response will either contain an argument
>   validation exception or a timeout exception. If you receive a timeout
>   exception, because you asked for 0 timeout, you can assume the message 
> was
>   valid and the topic creation was triggered.
>7. The request is not transactional.
>   1. If an error occurs on one topic, the others could still be
>   created.
>   2. Errors are reported independently.
>
> QA:
>
>- Why is CreateTopicsRequest a batch request?
>   - Scenarios where tools or admins want to create many topics should
>   be able to with fewer requests
>   - Example: MirrorMaker may want to create the topics downstream
>- What happens if some topics error immediately? Will it
>return immediately?
>   - The request will block until all topics have either been created,
>   errors, or the timeout has been hit
>   - There is no "short circuiting" where 1 error stops the other
>   topics from being created
>- Why implement "partial blocking" instead of fully async or fully
>consistent?
>   - See Cluster Consistent Blocking
>   
> 
>below
>- Why require the request to go to the controller?
>   - The controller is responsible for the cluster metadata and its
>   propagation
>   - See Request Forwarding
>   
> 

[GitHub] kafka pull request #1516: KAFKA-3837: Report the thread name of the blocking...

2016-06-16 Thread bharatviswa504
GitHub user bharatviswa504 opened a pull request:

https://github.com/apache/kafka/pull/1516

KAFKA-3837: Report the thread name of the blocking thread when throwing 
ConcurrentModificationException



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bharatviswa504/kafka KAFKA-3837-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1516.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1516


commit 026bc6b03a40b33b7e9c9f244a8370a395564c81
Author: Bharat Viswanadham 
Date:   2016-06-14T21:37:36Z

KAFKA-3837:  Report the thread name  of the blocking thread when throwing 
ConcurrentModificationException




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-3837) Report the name of the blocking thread when throwing ConcurrentModificationException

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334768#comment-15334768
 ] 

ASF GitHub Bot commented on KAFKA-3837:
---

GitHub user bharatviswa504 opened a pull request:

https://github.com/apache/kafka/pull/1516

KAFKA-3837: Report the thread name of the blocking thread when throwing 
ConcurrentModificationException



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bharatviswa504/kafka KAFKA-3837-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1516.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1516


commit 026bc6b03a40b33b7e9c9f244a8370a395564c81
Author: Bharat Viswanadham 
Date:   2016-06-14T21:37:36Z

KAFKA-3837:  Report the thread name  of the blocking thread when throwing 
ConcurrentModificationException




> Report the name of the blocking thread when throwing 
> ConcurrentModificationException
> 
>
> Key: KAFKA-3837
> URL: https://issues.apache.org/jira/browse/KAFKA-3837
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Simon Cooper
>Assignee: Bharat Viswanadham
>Priority: Minor
>
> {{KafkaConsumer.acquire}} throws {{ConcurrentModificationException}} if the 
> current thread it does not match the {{currentThread}} field. It would be 
> useful if the name of the other thread was included in the exception message 
> to help debug the problem when this exception occurs.
> As it stands, it can be really difficult to work out what's going wrong when 
> there's several threads all accessing the consumer at the same time, and your 
> existing exclusive access logic doesn't seem to be working as it should.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Reg: Build Validation Jenkins

2016-06-16 Thread BigData dev
Hi,
I have opened pull request for one of the JIRA.
My build has been marked has failed due to time out exception. I think
configuration setup is like that after 2 hours mark it as build fail.

I have seen other builds on jenkins they also has same issue.

Is this jenkins server executing machines have been slow or what could have
been the reason for this?

Can you please let me know, how can i resolve this issue, and validate my
build.



Thanks,
Bharat


[GitHub] kafka pull request #1513: MINOR: Follow-up from KAFKA-2720 with comment/styl...

2016-06-16 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1513


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-2720) Periodic purging groups in the coordinator

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334787#comment-15334787
 ] 

ASF GitHub Bot commented on KAFKA-2720:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1513


> Periodic purging groups in the coordinator
> --
>
> Key: KAFKA-2720
> URL: https://issues.apache.org/jira/browse/KAFKA-2720
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Jason Gustafson
> Fix For: 0.10.1.0
>
>
> Currently the coordinator removes the group (i.e. both removing it from the 
> cache and writing the tombstone message on its local replica without waiting 
> for ack) once it becomes an empty group.
> This can lead to a few issues such as 1) group removal and creation churns 
> when a group with very few members are being rebalanced, 2) if the local 
> write is failed / not propagated to other followers, they can only be removed 
> again when a new coordinator is migrated and detects the group has no members 
> already.
> We could instead piggy-back the periodic offset expiration along with the 
> group purging as well which removes any groups that had no corresponding 
> offsets in the cache any more.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Ismael Juma
Hi Jan,

That's interesting. Do you have some references you can share on this? It
would be good to know which Java 8 versions have been tested and whether it
is something that is being worked on.

Ismael

On Fri, Jun 17, 2016 at 12:02 AM,  wrote:

>
> Hi Ismael,
>
> Unfortunately Java 8 doesn't play nice with FreeBSD. We have seen a lot of
> JVM crashes running our 0.9 brokers on Java 8... Java 7 on the other hand
> is totally stable.
>
> Until these issues have been addressed, this would cause some serious
> issues for us.
>
> Regards
>
> Jan


Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread jan . omar

Hi Ismael,

Unfortunately Java 8 doesn't play nice with FreeBSD. We have seen a lot of JVM 
crashes running our 0.9 brokers on Java 8... Java 7 on the other hand is 
totally stable. 

Until these issues have been addressed, this would cause some serious issues 
for us.

Regards 

Jan

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

2016-06-16 Thread Roger Hoover (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334812#comment-15334812
 ] 

Roger Hoover commented on KAFKA-3740:
-

[~h...@pinterest.com], any update on this?  I'm wondering if you have any idea 
when it might be available.

> Add configs for RocksDBStores
> -
>
> Key: KAFKA-3740
> URL: https://issues.apache.org/jira/browse/KAFKA-3740
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Henry Cai
>  Labels: api, newbie
>
> Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, 
> or the default values are directly used. We need to make them configurable 
> for advanced users. For example, some default values may not work perfectly 
> for some scenarios: 
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
>  
> One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar 
> to "StreamsConfig", which defines all related rocksDB options configs, that 
> can be passed as key-value pairs to "StreamsConfig".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #1509: KAFKA-3691: Confusing logging during metadata upda...

2016-06-16 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1509


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-3691) Confusing logging during metadata update timeout

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334834#comment-15334834
 ] 

ASF GitHub Bot commented on KAFKA-3691:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1509


> Confusing logging during metadata update timeout
> 
>
> Key: KAFKA-3691
> URL: https://issues.apache.org/jira/browse/KAFKA-3691
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.10.1.0, 0.10.0.1
>
>
> When the KafkaProducer calls waitOnMetadata it will loop decrementing the 
> remainingWaitMs until it either receives the request metadata or runs out of 
> time. Inside the loop Metadata.awaitUpdate is called with the value in 
> remainingWaitMs. Inside Metadata.awaitUpdate a timeout execption could be 
> thrown using the remainingWaitMs which results in messages like:
> {noformat}
> org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
> after 3 ms.
> {noformat}
> Perhaps we should catch the exception and log the real maxWaitMs or change 
> the language to make the exception more clear. 
> Note: I still need to investigate further to be sure exactly when this 
> happens, but wanted to log the jira to make sure this is not forgotten. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-3691) Confusing logging during metadata update timeout

2016-06-16 Thread Ismael Juma (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3691:
---
   Resolution: Fixed
Fix Version/s: (was: 0.10.0.1)
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1509
[https://github.com/apache/kafka/pull/1509]

> Confusing logging during metadata update timeout
> 
>
> Key: KAFKA-3691
> URL: https://issues.apache.org/jira/browse/KAFKA-3691
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.10.1.0
>
>
> When the KafkaProducer calls waitOnMetadata it will loop decrementing the 
> remainingWaitMs until it either receives the request metadata or runs out of 
> time. Inside the loop Metadata.awaitUpdate is called with the value in 
> remainingWaitMs. Inside Metadata.awaitUpdate a timeout execption could be 
> thrown using the remainingWaitMs which results in messages like:
> {noformat}
> org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
> after 3 ms.
> {noformat}
> Perhaps we should catch the exception and log the real maxWaitMs or change 
> the language to make the exception more clear. 
> Note: I still need to investigate further to be sure exactly when this 
> happens, but wanted to log the jira to make sure this is not forgotten. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [VOTE] KIP-55: Secure quotas for authenticated users

2016-06-16 Thread Jun Rao
Rajini,

Thanks for the update. A few more questions/comments.

10. For the quota value stored in ZK, since we are adding an optional
user_principal field in the json, we should bump the version from 1 to 2.
Also, user_principal is not really part of the config values. So, perhaps
we should represent it as the following?
{
"version":2,
"config": {
"producer_byte_rate":"1024",
"consumer_byte_rate":"2048"
},
"user_principal" : "user1"
}

 Also, we should store user_principal in the following json too, right?
// Zookeeper persistence path /users//clients/clientA
{
"version":1,
"config": {
"producer_byte_rate":"10",
"consumer_byte_rate":"30"
}
}

11. For the change notification path, would it be better to change it to
something like the following and bump up version to 2?
// Change notification for quota of 
{
"version":2,
[
  { "entity_type": "users",
"entity_name": "user2"
  },
  { "entity_type": "clients",
"entity_name": "clientA"
  }
]
 }

Alternatively, we could change it to
// Change notification for quota of 
{
"version":2,
"entity_path": "users/user2"
}

{
"version":2,
"entity_path": "users/user2/clients/clientA"
}

12. Just to clarify on the meaning of remainder quota. If you have quotas
like the following,
  = 5
  = 10
  = 12
it means that all connections with user1 whose client-id is neither client1
nor client2 will be sharing a quota of 12, right? In other words, the quota
of  doesn't include the quota for  and .

Thanks,

Jun


On Thu, Jun 16, 2016 at 5:03 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Jun,
>
> Actually, with quotas stored in different nodes in ZK, it is better to
> store remainder quota rather than total quota under /users/ so that
> quota calculations are not dependent on the order of notifications. I have
> updated the KIP to reflect that. So the quotas in ZK now always reflect the
> quota applied to a group of client connections and use the same format as
> client-id quotas. But it is not hierarchical, making the configuration
> simpler.
>
> On Thu, Jun 16, 2016 at 11:49 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Jun,
> >
> > Thank you for the review. I have updated the KIP:
> >
> >
> >1. Added an overview section. Slightly reworded since it is better to
> >treat user and client-id as different levels rather than the same
> level.
> >2. Yes, it is neater to store quota for each entity in a different
> >path in Zookeeper. I put clients under users rather than the other way
> >round since that reflects the hierarchy and also keeps a user's quotas
> >together under a single sub-tree. I had initially used a single node
> to
> >keep quotas and sub-quotas of a user together so that updates are
> atomic
> >since changes to sub-quotas also affect remainder quotas for other
> clients.
> >But I imagine, updates to configs are rare and it is not a big issue.
> >3. I haven't modified the JSON for configuration change notifications.
> >The entity_name can now be a subpath that has both user and client.
> Have
> >added an example to the KIP. The downside of keeping clients under
> users in
> >ZK in 2) is that the change notification for sub-quota has entity_type
> >"users". I could extend the JSON to include client separately, but
> since
> >changes to a client sub-quota does impact other clients of the user
> as well
> >due to change in remainder quota, it may be ok as it is. Do let me
> know if
> >it looks confusing in the example.
> >4. Agree, updated.
> >
> >
> > On Wed, Jun 15, 2016 at 10:27 PM, Jun Rao  wrote:
> >
> >> Hi, Rajini,
> >>
> >> Thanks for the updated wiki. Overall, I like the new approach. It covers
> >> the common use cases now, is extensible, and is backward compatible. A
> few
> >> comments below.
> >>
> >> 1. It would be useful to describe a bit at the high level, how the new
> >> approach works. I think this can be summarized as follows. Quotas can be
> >> set at user, client-id or  levels. For a given client
> >> connection, the most specific quota matching the connection will be
> >> applied. For example, if both a  and a  quota
> match
> >> a connection, the  quota will be used. If more than 1
> >> quota at the same level (e.g., a quota on a user and another quota on a
> >> client-id) match the connection, the user level quota will be used,
> i.e.,
> >> user quota takes precedence over client-id quota.
> >>
> >> 2. For the ZK data structure, would it be better to store  >> client-id>
> >> quota as the following. Then the format of the value in each path is the
> >> same. The wiki also mentions that we want to include the original user
> >> name
> >> in the ZK value. Could you describe that in an example?
> >> // Zookeeper persistence path /clients/clientA/users/
> >> {
> >> "version":1,
> >> "config": {
> >> "producer_byte_

[jira] [Commented] (KAFKA-3185) Allow users to cleanup internal data

2016-06-16 Thread Ishita Mandhan (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334860#comment-15334860
 ] 

Ishita Mandhan commented on KAFKA-3185:
---

[~guozhang] Do you think this is a bug a newbie(aka me) could work on? If not, 
could you point me to something? Thanks

> Allow users to cleanup internal data
> 
>
> Key: KAFKA-3185
> URL: https://issues.apache.org/jira/browse/KAFKA-3185
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Guozhang Wang
>Priority: Blocker
>  Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Currently the internal data is managed completely by Kafka Streams framework 
> and users cannot clean them up actively. This results in a bad out-of-the-box 
> user experience especially for running demo programs since it results 
> internal data (changelog topics, RocksDB files, etc) that need to be cleaned 
> manually. It will be better to add a
> {code}
> KafkaStreams.cleanup()
> {code}
> function call to clean up these internal data programmatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

2016-06-16 Thread Guozhang Wang
+1.

On Thu, Jun 16, 2016 at 11:44 AM, Jason Gustafson 
wrote:

> Hi All,
>
> I'd like to open the vote for KIP-62. This proposal attempts to address one
> of the recurring usability problems that users of the new consumer have
> faced with as little impact as possible. You can read the full details
> here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread
> .
>
> After some discussion on this list, I think we were in agreement that this
> change addresses a major part of the problem and we've left the door open
> for further improvements, such as adding a heartbeat() API or a separately
> configured rebalance timeout. Thanks in advance to everyone who helped
> review the proposal.
>
> -Jason
>



-- 
-- Guozhang


Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

2016-06-16 Thread Ismael Juma
+1 (binding)

On Fri, Jun 17, 2016 at 12:44 AM, Guozhang Wang  wrote:

> +1.
>
> On Thu, Jun 16, 2016 at 11:44 AM, Jason Gustafson 
> wrote:
>
> > Hi All,
> >
> > I'd like to open the vote for KIP-62. This proposal attempts to address
> one
> > of the recurring usability problems that users of the new consumer have
> > faced with as little impact as possible. You can read the full details
> > here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread
> > .
> >
> > After some discussion on this list, I think we were in agreement that
> this
> > change addresses a major part of the problem and we've left the door open
> > for further improvements, such as adding a heartbeat() API or a
> separately
> > configured rebalance timeout. Thanks in advance to everyone who helped
> > review the proposal.
> >
> > -Jason
> >
>
>
>
> --
> -- Guozhang
>


Re: [VOTE] KIP-4 Create Topics Schema

2016-06-16 Thread Ismael Juma
+1 (binding)

On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke  wrote:

> I would like to initiate the voting process for the "KIP-4 Create Topics
> Schema changes". This is not a vote for all of KIP-4, but specifically for
> the create topics changes. I have included the exact changes below for
> clarity:
> >
> > Create Topics Request (KAFKA-2945
> > )
> >
> > CreateTopics Request (Version: 0) => [create_topic_requests] timeout
> >   create_topic_requests => topic num_partitions replication_factor
> [replica_assignment] [configs]
> > topic => STRING
> > num_partitions => INT32
> > replication_factor => INT16
> > replica_assignment => partition_id [replicas]
> >   partition_id => INT32
> >   replicas => INT32
> > configs => config_key config_value
> >   config_key => STRING
> >   config_value => STRING
> >   timeout => INT32
> >
> > CreateTopicsRequest is a batch request to initiate topic creation with
> > either predefined or automatic replica assignment and optionally topic
> > configuration.
> >
> > Request semantics:
> >
> >1. Must be sent to the controller broker
> >2. If there are multiple instructions for the same topic in one
> >request an InvalidRequestException will be logged on the broker and
> the
> >client will be disconnected.
> >   - This is because the list of topics is modeled server side as a
> >   map with TopicName as the key
> >3. The principal must be authorized to the "Create" Operation on the
> >"Cluster" resource to create topics.
> >   - Unauthorized requests will receive a
> ClusterAuthorizationException
> >4.
> >
> >Only one from ReplicaAssignment or (num_partitions +
> replication_factor
> >), can be defined in one instruction.
> >- If both parameters are specified an InvalidRequestException will be
> >   logged on the broker and the client will be disconnected.
> >   - In the case ReplicaAssignment is defined number of partitions and
> >   replicas will be calculated from the supplied replica_assignment.
> >   - In the case of defined (num_partitions + replication_factor)
> >   replica assignment will be automatically generated by the server.
> >   - One or the other must be defined. The existing broker side auto
> >   create defaults will not be used
> >   (default.replication.factor, num.partitions). The client
> implementation can
> >   have defaults for these options when generating the messages.
> >   - The first replica in [replicas] is assumed to be the preferred
> >   leader. This matches current behavior elsewhere.
> >5. Setting a timeout > 0 will allow the request to block until the
> >topic metadata is "complete" on the controller node.
> >   - Complete means the local topic metadata cache been completely
> >   populated and all partitions have leaders
> >  - The topic metadata is updated when the controller sends out
> >  update metadata requests to the brokers
> >   - If a timeout error occurs, the topic could still be created
> >   successfully at a later time. Its up to the client to query for
> the state
> >   at that point.
> >6. Setting a timeout <= 0 will validate arguments and trigger the
> >create topics and return immediately.
> >   - This is essentially the fully asynchronous mode we have in the
> >   Zookeeper tools today.
> >   - The error code in the response will either contain an argument
> >   validation exception or a timeout exception. If you receive a
> timeout
> >   exception, because you asked for 0 timeout, you can assume the
> message was
> >   valid and the topic creation was triggered.
> >7. The request is not transactional.
> >   1. If an error occurs on one topic, the others could still be
> >   created.
> >   2. Errors are reported independently.
> >
> > QA:
> >
> >- Why is CreateTopicsRequest a batch request?
> >   - Scenarios where tools or admins want to create many topics should
> >   be able to with fewer requests
> >   - Example: MirrorMaker may want to create the topics downstream
> >- What happens if some topics error immediately? Will it
> >return immediately?
> >   - The request will block until all topics have either been created,
> >   errors, or the timeout has been hit
> >   - There is no "short circuiting" where 1 error stops the other
> >   topics from being created
> >- Why implement "partial blocking" instead of fully async or fully
> >consistent?
> >   - See Cluster Consistent Blocking
> >   <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> >
> >below
> >- Why require the request to go to the controller?
> >   - The controller is r

[jira] [Resolved] (KAFKA-3755) tightening the offset check in ReplicaFetcherThread

2016-06-16 Thread Jun Rao (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-3755.

Resolution: Not A Problem

As notes in the PR, looks like Log.append() already has that check. See 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/Log.scala#L370
 .

Closing the jira.

> tightening the offset check in ReplicaFetcherThread
> ---
>
> Key: KAFKA-3755
> URL: https://issues.apache.org/jira/browse/KAFKA-3755
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>
> Currently, in ReplicaFetcherThread.processPartitionData(), we have the 
> following code to make sure that the fetchOffset matches the log end offset.
>   if (fetchOffset != replica.logEndOffset.messageOffset)
> throw new RuntimeException("Offset mismatch for partition %s: fetched 
> offset = %d, log end offset = %d.".format(topicAndPartition, fetchOffset, 
> replica.logEndOffset.messageOffset))
> It would be useful to further assert that the first offset in the messageSet 
> to be appended to the log is >= than the log end offset.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #1506: KAFKA-3755 Tightening the offset check in ReplicaF...

2016-06-16 Thread imandhan
Github user imandhan closed the pull request at:

https://github.com/apache/kafka/pull/1506


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-3755) tightening the offset check in ReplicaFetcherThread

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334892#comment-15334892
 ] 

ASF GitHub Bot commented on KAFKA-3755:
---

Github user imandhan closed the pull request at:

https://github.com/apache/kafka/pull/1506


> tightening the offset check in ReplicaFetcherThread
> ---
>
> Key: KAFKA-3755
> URL: https://issues.apache.org/jira/browse/KAFKA-3755
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>
> Currently, in ReplicaFetcherThread.processPartitionData(), we have the 
> following code to make sure that the fetchOffset matches the log end offset.
>   if (fetchOffset != replica.logEndOffset.messageOffset)
> throw new RuntimeException("Offset mismatch for partition %s: fetched 
> offset = %d, log end offset = %d.".format(topicAndPartition, fetchOffset, 
> replica.logEndOffset.messageOffset))
> It would be useful to further assert that the first offset in the messageSet 
> to be appended to the log is >= than the log end offset.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

2016-06-16 Thread Jun Rao
Harsha,

Another question.

9. How would the delegation token be configured in the client? The standard
way is to do this through JAAS. However, we will need to think through if
this is convenient in a shared environment. For example, when a new task is
added to a Storm worker node, do we need to dynamically add a new section
in the JAAS file? It may be more convenient if we can pass in the token
through the config directly w/o going through JAAS.

Are you or Parth still actively working on this KIP?

Thanks,

Jun



On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao  wrote:

> Just to add on that list.
>
> 2. It would be good to document the format of the data stored in ZK.
> 7. Earlier, there was a discussion on whether the tokens should be
> propagated through ZK like config/acl/quota, or through the controller.
> Currently, the controller is only designed for propagating topic metadata,
> but not other data.
> 8. Should we use SCRAM to send the token instead of DIGEST-MD5 since it's
> deprecated?
>
> Also, the images in the wiki seem broken.
>
> Thanks,
>
> Jun
>
> On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira  wrote:
>
>> From what I can see, remaining questions are:
>>
>> 1. Who / how are tokens renewed? By original requester only? or using
>> Kerberos auth only?
>> 2. Are tokens stored on each broker or in ZK?
>> 3. How are tokens invalidated / expired?
>> 4. Which encryption algorithm is used?
>> 5. What is the impersonation proposal (it wasn't in the KIP but was
>> discussed in this thread)?
>> 6. Do we need new ACLs, if so - for what actions?
>>
>> Gwen
>>
>> On Thu, Jun 9, 2016 at 7:48 PM, Harsha  wrote:
>> > Jun & Ismael,
>> >  Unfortunately I couldn't attend the KIP meeting
>> >  when delegation tokens discussed. Appreciate if
>> >  you can update the thread if you have any
>> >  further questions.
>> > Thanks,
>> > Harsha
>> >
>> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
>> >> It seems that the links to images in the KIP are broken.
>> >>
>> >> Liquan
>> >>
>> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
>> >> brahmbhatt.pa...@gmail.com> wrote:
>> >>
>> >> > 110. What does getDelegationTokenAs mean?
>> >> > In the current proposal we only allow a user to get delegation token
>> for
>> >> > the identity that it authenticated as using another mechanism, i.e.
>> A user
>> >> > that authenticate using a keytab for principal us...@example.com
>> will get
>> >> > delegation tokens for that user only. In future I think we will have
>> to
>> >> > extend support such that we allow some set of users (
>> >> > kafka-rest-u...@example.com, storm-nim...@example.com) to acquire
>> >> > delegation tokens on behalf of other users whose identity they have
>> >> > verified independently.  Kafka brokers will have ACLs to control
>> which
>> >> > users are allowed to impersonate other users and get tokens on
>> behalf of
>> >> > them. Overall Impersonation is a whole different problem in my
>> opinion and
>> >> > I think we can tackle it in separate KIP.
>> >> >
>> >> > 111. What's the typical rate of getting and renewing delegation
>> tokens?
>> >> > Typically this should be very very low, 1 request per minute is a
>> >> > relatively high estimate. However it depends on the token
>> expiration. I am
>> >> > less worried about the extra load it puts on controller vs the added
>> >> > complexity and the value it offers.
>> >> >
>> >> > Thanks
>> >> > Parth
>> >> >
>> >> >
>> >> >
>> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma 
>> wrote:
>> >> >
>> >> > > Thanks Rajini. It would probably require a separate KIP as it will
>> >> > > introduce user visible changes. We could also update KIP-48 to
>> have this
>> >> > > information, but it seems cleaner to do it separately. We can
>> discuss
>> >> > that
>> >> > > in the KIP call today.
>> >> > >
>> >> > > Ismael
>> >> > >
>> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
>> >> > > rajinisiva...@googlemail.com> wrote:
>> >> > >
>> >> > > > Ismael,
>> >> > > >
>> >> > > > I have created a JIRA (
>> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
>> >> > > > for adding SCRAM as a SASL mechanism. Would that need another
>> KIP? If
>> >> > > > KIP-48 will use this mechanism, can this just be a JIRA that gets
>> >> > > reviewed
>> >> > > > when the PR is ready?
>> >> > > >
>> >> > > > Thank you,
>> >> > > >
>> >> > > > Rajini
>> >> > > >
>> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma 
>> >> > wrote:
>> >> > > >
>> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
>> >> > > > >
>> >> > > > > Gwen had independently mentioned this as a SASL mechanism that
>> might
>> >> > be
>> >> > > > > useful for Kafka and I have been meaning to check it in more
>> detail.
>> >> > > Good
>> >> > > > > to know that you are willing to contribute an implementation.
>> Maybe
>> >> > we
>> >> > > > > should file a separate JIRA for this?
>> >> 

Re: avro-console-producer issue for Kafka Stream example PageViewRegionLambdaExample

2016-06-16 Thread Guozhang Wang
Here is my understanding about Avro "union type (or optional field if one
of its union value is null)":

1. Even for such fields, you need to specify them as "null" in your data,
such as {"region" : "US", "experience" : null}.

2. And if you do have a value for these union fields, you need to specify
its type, such as {"region" : "US", "experience" : {"string": "scala"}}.


Guozhang


On Thu, Jun 16, 2016 at 11:33 AM, Philippe Derome 
wrote:

> I am able to produce Avro from producer for pageview.avsc but not for
> userprofile.avsc. Here are two samples below, first one leaves out Avro
> optional experience, second one specifies it.
>
> Note: I do not register using REST curl calls schemas to schema-registry as
> I assume that kafka-avro-console-producer takes care of that.
>
> First case has Avro disagreeing that experience can be optional with an
> exception:
>
> AvroTypeException: Expected field name not found: experience
>
> Second case generates another AvroTypeException complaining about the union
> type:
>
> Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
> VALUE_STRING
>
> ./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
> UserProfile --property value.schema='{
>
>  "type": "record",
>
>  "name": "UserProfile",
>
>  "fields": [
>
>  {"name": "experience", "type": ["string", "null"]},
>
>  {"name": "region", "type": "string"}
>
>  ]
>
> }'
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> {"region":"US"}
>
> org.apache.kafka.common.errors.SerializationException: Error deserializing
> json {"region":"US"} to Avro of schema
>
> {"type":"record","name":"UserProfile","fields":[{"name":"experience","type":["string","null"]},{"name":"region","type":"string"}]}
>
> Caused by: org.apache.avro.AvroTypeException: Expected field name not
> found: experience
>
> at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
>
> at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:429)
>
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
>
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
>
> at
>
> io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:189)
>
> at
>
> io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:157)
>
> at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)
>
> at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
>
>
> ./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
> UserProfile --property value.schema='{
>
>  "type": "record",
>
>  "name": "UserProfile",
>
>  "fields": [
>
>  {"name": "experience", "type": ["string", "null"]},
>
>  {"name": "region", "type": "string"}
>
>  ]
>
> }'
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
>
> [jar:file:/Applications/confluent-3.0.0/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> {"region":"US","experience":"scala"}
>
> org.apache.kafka.common.errors.SerializationException: Error deserializing
> json {"region":"US","experience":"scala"} to Avro of schema
>
> {"type":"recor

Re: [DISCUSS] Java 8 as a minimum requirement

2016-06-16 Thread Guozhang Wang
I think moving from Java 7 to 8 for Kafka Streams won't be of a big issue.
What specific concerns do you have Jeff?

On Thu, Jun 16, 2016 at 2:12 PM, Jeff Klukas  wrote:

> Would the move to Java 8 be for all modules? I'd have some concern about
> removing Java 7 compatibility for kafka-clients and for kafka streams
> (though less so since it's still so new). I don't know how hard it will be
> to transition a Scala 2.11 application to Scala 2.12. Are we comfortable
> with the idea of applications stuck on Scala 2.11 or otherwise unable to
> update to Java 8 not having access to new client releases?
>
> On Thu, Jun 16, 2016 at 5:05 PM, Philippe Derome 
> wrote:
>
> > I strongly support motion having difficulty running (Apache Kafka as
> > opposed to Confluent) Stream examples with JDK 8 today.
> > On 16 Jun 2016 4:46 p.m., "Ismael Juma"  wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a discussion on making Java 8 a minimum
> requirement
> > > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now).
> This
> > > is the first discussion on the topic so the idea is to understand how
> > > people feel about it. If people feel it's too soon, then we can pick up
> > the
> > > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > > positive, I will start a vote thread.
> > >
> > > Let's start with some dates. Java 7 hasn't received public updates
> since
> > > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> > scheduled
> > > to be released in March 2017[3].
> > >
> > > The first argument for dropping support for Java 7 is that the last
> > public
> > > release by Oracle contains a large number of known security
> > > vulnerabilities. The effectiveness of Kafka's security features is
> > reduced
> > > if the underlying runtime is not itself secure.
> > >
> > > The second argument for moving to Java 8 is that it adds a number of
> > > compelling features:
> > >
> > > * Lambda expressions and method references (particularly useful for the
> > > Kafka Streams DSL)
> > > * Default methods (very useful for maintaining compatibility when
> adding
> > > methods to interfaces)
> > > * java.util.stream (helpful for making collection transformations more
> > > concise)
> > > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder,
> LongAccumulator)
> > > * Other nice things: SplittableRandom, Optional (and many others I have
> > not
> > > mentioned)
> > >
> > > The third argument is that it will simplify our testing matrix, we
> won't
> > > have to test with Java 7 any longer (this is particularly useful for
> > system
> > > tests that take hours to run). It will also make it easier to support
> > Scala
> > > 2.12, which requires Java 8.
> > >
> > > The fourth argument is that many other open-source projects have taken
> > the
> > > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop
> 3[7],
> > > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android
> will
> > > support Java 8 in the next version (although it will take a while
> before
> > > most phones will use that version sadly). This reduces (but does not
> > > eliminate) the chance that we would be the first project that would
> > cause a
> > > user to consider a Java upgrade.
> > >
> > > The main argument for not making the change is that a reasonable number
> > of
> > > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > > More specifically, we care about the subset who would be able to
> upgrade
> > to
> > > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> > would
> > > be great if we could quantify this in some way.
> > >
> > > What do you think?
> > >
> > > Ismael
> > >
> > > [1] https://java.com/en/download/faq/java_7.xml
> > > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > > [3] http://openjdk.java.net/projects/jdk9/
> > > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > > [5]
> https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > > [8] https://webtide.com/jetty-9-3-features/
> > > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > > [10]
> > >
> > >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > > [11] http://markmail.org/message/l7s276y3xkga2eqf
> > >
> >
>



-- 
-- Guozhang


Re: [VOTE] KIP-4 Create Topics Schema

2016-06-16 Thread Guozhang Wang
+1.

On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma  wrote:

> +1 (binding)
>
> On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke  wrote:
>
> > I would like to initiate the voting process for the "KIP-4 Create Topics
> > Schema changes". This is not a vote for all of KIP-4, but specifically
> for
> > the create topics changes. I have included the exact changes below for
> > clarity:
> > >
> > > Create Topics Request (KAFKA-2945
> > > )
> > >
> > > CreateTopics Request (Version: 0) => [create_topic_requests] timeout
> > >   create_topic_requests => topic num_partitions replication_factor
> > [replica_assignment] [configs]
> > > topic => STRING
> > > num_partitions => INT32
> > > replication_factor => INT16
> > > replica_assignment => partition_id [replicas]
> > >   partition_id => INT32
> > >   replicas => INT32
> > > configs => config_key config_value
> > >   config_key => STRING
> > >   config_value => STRING
> > >   timeout => INT32
> > >
> > > CreateTopicsRequest is a batch request to initiate topic creation with
> > > either predefined or automatic replica assignment and optionally topic
> > > configuration.
> > >
> > > Request semantics:
> > >
> > >1. Must be sent to the controller broker
> > >2. If there are multiple instructions for the same topic in one
> > >request an InvalidRequestException will be logged on the broker and
> > the
> > >client will be disconnected.
> > >   - This is because the list of topics is modeled server side as a
> > >   map with TopicName as the key
> > >3. The principal must be authorized to the "Create" Operation on the
> > >"Cluster" resource to create topics.
> > >   - Unauthorized requests will receive a
> > ClusterAuthorizationException
> > >4.
> > >
> > >Only one from ReplicaAssignment or (num_partitions +
> > replication_factor
> > >), can be defined in one instruction.
> > >- If both parameters are specified an InvalidRequestException will
> be
> > >   logged on the broker and the client will be disconnected.
> > >   - In the case ReplicaAssignment is defined number of partitions
> and
> > >   replicas will be calculated from the supplied replica_assignment.
> > >   - In the case of defined (num_partitions + replication_factor)
> > >   replica assignment will be automatically generated by the server.
> > >   - One or the other must be defined. The existing broker side auto
> > >   create defaults will not be used
> > >   (default.replication.factor, num.partitions). The client
> > implementation can
> > >   have defaults for these options when generating the messages.
> > >   - The first replica in [replicas] is assumed to be the preferred
> > >   leader. This matches current behavior elsewhere.
> > >5. Setting a timeout > 0 will allow the request to block until the
> > >topic metadata is "complete" on the controller node.
> > >   - Complete means the local topic metadata cache been completely
> > >   populated and all partitions have leaders
> > >  - The topic metadata is updated when the controller sends out
> > >  update metadata requests to the brokers
> > >   - If a timeout error occurs, the topic could still be created
> > >   successfully at a later time. Its up to the client to query for
> > the state
> > >   at that point.
> > >6. Setting a timeout <= 0 will validate arguments and trigger the
> > >create topics and return immediately.
> > >   - This is essentially the fully asynchronous mode we have in the
> > >   Zookeeper tools today.
> > >   - The error code in the response will either contain an argument
> > >   validation exception or a timeout exception. If you receive a
> > timeout
> > >   exception, because you asked for 0 timeout, you can assume the
> > message was
> > >   valid and the topic creation was triggered.
> > >7. The request is not transactional.
> > >   1. If an error occurs on one topic, the others could still be
> > >   created.
> > >   2. Errors are reported independently.
> > >
> > > QA:
> > >
> > >- Why is CreateTopicsRequest a batch request?
> > >   - Scenarios where tools or admins want to create many topics
> should
> > >   be able to with fewer requests
> > >   - Example: MirrorMaker may want to create the topics downstream
> > >- What happens if some topics error immediately? Will it
> > >return immediately?
> > >   - The request will block until all topics have either been
> created,
> > >   errors, or the timeout has been hit
> > >   - There is no "short circuiting" where 1 error stops the other
> > >   topics from being created
> > >- Why implement "partial blocking" instead of fully async or fully
> > >consistent?
> > >   - See Cluster Consistent Blocking
> > >   <
> >
> https://cwiki.apache.org/conflue

[jira] [Commented] (KAFKA-3185) Allow users to cleanup internal data

2016-06-16 Thread Guozhang Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334926#comment-15334926
 ] 

Guozhang Wang commented on KAFKA-3185:
--

I think [~mjsax] has already started working on this, but did not yet assign 
this ticket to himself.

There is indeed a list of newbie tickets for Kafka Streams:

https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20streams%20AND%20labels%20in%20(newbie%2C%20%22newbie%2B%2B%22)%20ORDER%20BY%20priority%20DESC

Feel free to grab any that is not assigned yet.

> Allow users to cleanup internal data
> 
>
> Key: KAFKA-3185
> URL: https://issues.apache.org/jira/browse/KAFKA-3185
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Guozhang Wang
>Priority: Blocker
>  Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Currently the internal data is managed completely by Kafka Streams framework 
> and users cannot clean them up actively. This results in a bad out-of-the-box 
> user experience especially for running demo programs since it results 
> internal data (changelog topics, RocksDB files, etc) that need to be cleaned 
> manually. It will be better to add a
> {code}
> KafkaStreams.cleanup()
> {code}
> function call to clean up these internal data programmatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #1485: KAFKA-3805: Check if DB is null.

2016-06-16 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1485


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-3805) Running multiple instances of a Streams app on the same machine results in Java_org_rocksdb_RocksDB_write error

2016-06-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334935#comment-15334935
 ] 

ASF GitHub Bot commented on KAFKA-3805:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1485


> Running multiple instances of a Streams app on the same machine results in 
> Java_org_rocksdb_RocksDB_write error
> ---
>
> Key: KAFKA-3805
> URL: https://issues.apache.org/jira/browse/KAFKA-3805
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
> Environment: * Ubuntu Server 16.04 LTS
> - Java: OpenJDK Runtime Environment (build 
> 1.8.0_91-8u91-b14-0ubuntu4~16.04.1-b14)
> * OSX 10.11.5
>- Java: Java(TM) SE Runtime Environment (build 1.8.0_74-b02)
>Reporter: Andres Gomez Ferrer
>Assignee: Eno Thereska
>Priority: Critical
> Fix For: 0.10.0.1
>
> Attachments: hs_err_pid23047.log, kafka-streams-testing.zip
>
>
> I reproduce the error working with simple two instances of  my kafka streams 
> application reading from one topic writing on other and using a RocksDB store.
> I reproduce it starting one instance, sending some messages on the input 
> topic and later starting another Kafka streams instance. Then, the first 
> instance died and drop a core dump.
> --
>  A fatal error has been detected by the Java Runtime Environment:
>   SIGSEGV (0xb) at pc=0x7f9f488fd8e8, pid=23047, tid=140322171361024
>  JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 
> 1.8.0_91-8u91-b14-0ubuntu4~16.04.1-b14)
>  Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
>  Problematic frame:
>  C  [librocksdbjni712288404493406713..so+0x15b8e8]  
> Java_org_rocksdb_RocksDB_write0+0x48
>  Core dump written. Default location: /root/core or core.23047
>  An error report file with more information is saved as:
>  /root/hs_err_pid23047.log
>  If you would like to submit a bug report, please visit:
>http://bugreport.java.com/bugreport/crash.jsp
>  The crash happened outside the Java Virtual Machine in native code.
>  See problematic frame for where to report the bug.
> --
> The core dump is there: 
> https://s3-eu-west-1.amazonaws.com/arodriguezg-test/core
> Note: The core dump is collected at Ubuntu Server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (KAFKA-3805) Running multiple instances of a Streams app on the same machine results in Java_org_rocksdb_RocksDB_write error

2016-06-16 Thread Guozhang Wang (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-3805.
--
   Resolution: Fixed
Fix Version/s: 0.10.0.1

Issue resolved by pull request 1485
[https://github.com/apache/kafka/pull/1485]

> Running multiple instances of a Streams app on the same machine results in 
> Java_org_rocksdb_RocksDB_write error
> ---
>
> Key: KAFKA-3805
> URL: https://issues.apache.org/jira/browse/KAFKA-3805
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
> Environment: * Ubuntu Server 16.04 LTS
> - Java: OpenJDK Runtime Environment (build 
> 1.8.0_91-8u91-b14-0ubuntu4~16.04.1-b14)
> * OSX 10.11.5
>- Java: Java(TM) SE Runtime Environment (build 1.8.0_74-b02)
>Reporter: Andres Gomez Ferrer
>Assignee: Eno Thereska
>Priority: Critical
> Fix For: 0.10.0.1
>
> Attachments: hs_err_pid23047.log, kafka-streams-testing.zip
>
>
> I reproduce the error working with simple two instances of  my kafka streams 
> application reading from one topic writing on other and using a RocksDB store.
> I reproduce it starting one instance, sending some messages on the input 
> topic and later starting another Kafka streams instance. Then, the first 
> instance died and drop a core dump.
> --
>  A fatal error has been detected by the Java Runtime Environment:
>   SIGSEGV (0xb) at pc=0x7f9f488fd8e8, pid=23047, tid=140322171361024
>  JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 
> 1.8.0_91-8u91-b14-0ubuntu4~16.04.1-b14)
>  Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
>  Problematic frame:
>  C  [librocksdbjni712288404493406713..so+0x15b8e8]  
> Java_org_rocksdb_RocksDB_write0+0x48
>  Core dump written. Default location: /root/core or core.23047
>  An error report file with more information is saved as:
>  /root/hs_err_pid23047.log
>  If you would like to submit a bug report, please visit:
>http://bugreport.java.com/bugreport/crash.jsp
>  The crash happened outside the Java Virtual Machine in native code.
>  See problematic frame for where to report the bug.
> --
> The core dump is there: 
> https://s3-eu-west-1.amazonaws.com/arodriguezg-test/core
> Note: The core dump is collected at Ubuntu Server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-3185) Allow users to cleanup internal data

2016-06-16 Thread Ishita Mandhan (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334945#comment-15334945
 ] 

Ishita Mandhan commented on KAFKA-3185:
---

Ah okay, sounds good- I'll look for something else. thank you!

> Allow users to cleanup internal data
> 
>
> Key: KAFKA-3185
> URL: https://issues.apache.org/jira/browse/KAFKA-3185
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Guozhang Wang
>Priority: Blocker
>  Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Currently the internal data is managed completely by Kafka Streams framework 
> and users cannot clean them up actively. This results in a bad out-of-the-box 
> user experience especially for running demo programs since it results 
> internal data (changelog topics, RocksDB files, etc) that need to be cleaned 
> manually. It will be better to add a
> {code}
> KafkaStreams.cleanup()
> {code}
> function call to clean up these internal data programmatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


  1   2   >