[jira] [Resolved] (KAFKA-4337) Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the specified zookeeper)

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4337.
--
Resolution: Won't Fix

> Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA 
> API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the 
> specified zookeeper)
> ---
>
> Key: KAFKA-4337
> URL: https://issues.apache.org/jira/browse/KAFKA-4337
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Reporter: Bharat Patel
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  Want to use below code snippet to create topic in multiple zookeepers with 
> Kafka Java APIS. when I specify 2 zookeeprs IPs in zookeeperConnect variable 
> it only creates topic in anyone of zookeeper. Both the zookeeper are 2 
> different kafka clusters.
>  String zookeeperConnect = zookeeperIPs; // Multiple zookeeper IPs
>int sessionTimeoutMs = 10 * 1000;
>int connectionTimeoutMs = 8 * 1000;
>try {
> ZkClient zkClient = new ZkClient(
> zookeeperConnect,
> sessionTimeoutMs,
> connectionTimeoutMs,
> ZKStringSerializer$.MODULE$);
> boolean isSecureKafkaCluster = false;
> ZkUtils zkUtils = new ZkUtils(zkClient, 
> new ZkConnection(zookeeperConnect), isSecureKafkaCluster);
>  String topic1 = "nameofTopictobeCreated";
>  int partitions = 1;
>  int replication = 1;
>  Properties topicConfig = new Properties(); // add per-topic 
> configurations settings here
>  AdminUtils.createTopic(zkUtils, topic1, partitions, replication, 
> topicConfig);



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-4264) kafka-server-stop.sh fails is Kafka launched via kafka-server-start.sh

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4264.
--
Resolution: Duplicate

This is related to KAFKA-4931. PR is available for KAFKA-4931. 

> kafka-server-stop.sh fails is Kafka launched via kafka-server-start.sh
> --
>
> Key: KAFKA-4264
> URL: https://issues.apache.org/jira/browse/KAFKA-4264
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.10.0.1
> Environment: Tested in Debian Jessy
>Reporter: Alex Schmitz
>Priority: Trivial
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> kafka-server-stop.sh greps for the process ID to kill with the following: 
> bq. PIDS=$(ps ax | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk 
> '{print $1}')
> However, if Kafka is launched via the kafka-server-start.sh script, the 
> process doesn't include kafka.Kafka, the grep fails to find the process, and 
> it returns the failure message, No Kafka server to stop. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5819) Add Joined class and relevant KStream join overloads

[Damian Guy (JIRA)]

Damian Guy created KAFKA-5819:
-

 Summary: Add Joined class and relevant KStream join overloads
 Key: KAFKA-5819
 URL: https://issues.apache.org/jira/browse/KAFKA-5819
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Damian Guy
Assignee: Damian Guy
 Fix For: 1.0.0


Add the {{Joined}} class as defined in KIP-182 and the following overloads to 
{{KStream}}
{code}
 KStream join(final KStream other, final ValueJoiner joiner, final JoinWindows windows, final 
Joined options);
 
 KStream join(final KTable other, final ValueJoiner joiner, final Joined options);
 
 KStream leftJoin(final KStream other, final 
ValueJoiner joiner, final JoinWindows 
windows, final Joined options);
 
 KStream leftJoin(final KTable other, final ValueJoiner joiner, final Joined options);
 
 KStream outerJoin(final KStream other, final 
ValueJoiner joiner, final JoinWindows 
windows, final Joined options);
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3776: KAFKA-5819: Add Joined class and relevant KStream ...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3776

KAFKA-5819: Add Joined class and relevant KStream join overloads

Add the `Joined` class and the overloads to `KStream` that use it.
Deprecate existing methods that have `Serde` params

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kip-182-stream-join

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3776.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3776


commit 5be7a49c245f762479ec02c7e1625a1da882bde9
Author: Damian Guy 
Date:   2017-09-01T08:44:16Z

add Joined class and overloads for KStream#join

commit d3991b13e5fd6f9c926c1d6cf3d4e15e77eea7ba
Author: Damian Guy 
Date:   2017-09-01T09:05:39Z

KStream#leftJoin(KStream...)

commit d8b007304e462b7d0e59e3fb4d4f2fa10a78d05b
Author: Damian Guy 
Date:   2017-09-01T09:25:13Z

stream table leftJoin

commit eeed855e153df766f1341469132f45fff62a13ce
Author: Damian Guy 
Date:   2017-09-01T09:40:01Z

outerJoin




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Fault Injection

[Viktor Somogyi]

Hi Colin,

I'd be interested in this and also think it's a valuable thing to have this
for the community and would greatly increase the test coverage.
Saw you already have a PR, I'll give a review as I have time :).

Viktor

On Tue, Aug 22, 2017 at 9:36 PM, Timothy Chen  wrote:

> Hi Colin,
>
> The Kibosh code is just a README for now, is it going to be published soon?
>
> Tim
>
> On Tue, Aug 22, 2017 at 11:44 AM, Colin McCabe  wrote:
> > Hi all,
> >
> > I've been working on a fault injector for Apache Kafka.  The general
> > idea is to create faults such as network partitions or disk failures,
> > and see what happens in the cluster.  The fault injector can run as part
> > of a ducktape system test, or standalone.
> >
> > The fault injector has two processes: a coordinator, and an agent.  The
> > agent process is responsible for actually implementing the faults.  For
> > example, it might run iptables, send signals to processes, generate a
> > lot of load, or do something else to disrupt the computer it is running
> > on.  We run an agent process on each node where we would like to
> > potentially inject faults.  So it will run alongside the brokers,
> > zookeeper nodes, etc.
> >
> > The coordinator process is responsible for communicating with the agent
> > processes and for scheduling faults.  For example, the coordinator can
> > be instructed to create a fault immediately on several nodes.  Or it can
> > be instructed to create faults over time, based on a pseudorandom seed.
> > Both the coordinator and the agent expose a REST interface that accepts
> > objects serialized via JSON.
> >
> > I think two kinds of faults will be especially interesting: network
> > faults, and disk errors.  Simulating network faults in a Linux
> > environment is relatively straightforward using iptables.  Disk errors
> > are tougher to simulate, but I have written a FUSE filesystem to do
> > this.  The  filesystem essentially simulates a bind mount in most cases,
> > but it can take a JSON specification telling it to inject certain
> > faults.  (Disk errors seem especially relevant to the ongoing work on
> > JBOD.)
> >
> > Although it's not a user-visible component, I think having a fault
> > injector will be really great for Kafka users.  It will really help us
> > stress test Kafka in more situations.  I'm going to post some patches in
> > a day or two-- it would be great to get some feedback.  Check out
> > https://cwiki.apache.org/confluence/display/KAFKA/Fault+Injection
> >
> > best,
> > Colin
>

Re: [DISCUSS] 0.11.0.1 bug fix release

[Damian Guy]

Hi all,

There is one last outstanding issue in 0.11.0.1. This will hopefully be
resolved today. Assuming it is i will cut the first RC tomorrow.

Thanks for your patience.

Regards,
Damian

On Tue, 29 Aug 2017 at 10:53 Damian Guy  wrote:

> We still have 2 outstanding issues to close. They both have patches
> available and will hopefully be completed shortly.
>
> Thanks,
> Damian
>
> On Thu, 24 Aug 2017 at 10:05 Damian Guy  wrote:
>
>> A quick update. There are 2 remaining issues, that both have patches
>> available. Hopefully they will be merged soon and we can begin:
>> https://issues.apache.org/jira/projects/KAFKA/versions/12340632
>>
>> Thanks,
>> Damian
>>
>> On Tue, 22 Aug 2017 at 10:29 Damian Guy  wrote:
>>
>>> An update on the 0.11.0.1 release. We currently have 5 outstanding
>>> issues: https://issues.apache.org/jira/projects/KAFKA/versions/12340632
>>> 3 with patches available that we can hopefully get merged pretty soon (1
>>> is actually already on 0.11.0.1)
>>> 2 issues that are Open, 1 is unassigned.
>>> Hopefully we can get this cleaned up in the next day or two and then i
>>> can go about building an RC.
>>>
>>> Thanks,
>>> Damian
>>>
>>> On Thu, 17 Aug 2017 at 17:45 Damian Guy  wrote:
>>>
 Just a quick update.

 The list has reduced to 6 remaining issues:
 https://issues.apache.org/jira/projects/KAFKA/versions/12340632

 Thanks to everyone for completing and/or moving tickets to future
 releases.

 Damian


 On Thu, 17 Aug 2017 at 09:50 Damian Guy  wrote:

> Hi Srikanth,
> Optimistically i'm aiming for end of next week. Though it depends on
> how quickly the outstanding issues are closed and any other blockers that
> arise.
>
> Thanks,
> Damian
>
> On Thu, 17 Aug 2017 at 07:59 Srikanth Sampath <
> ssampath.apa...@gmail.com> wrote:
>
>> Thanks Damian.  What's the ballpark when 0.11.0.1 will be available?
>> -Srikanth
>>
>> On Wed, Aug 16, 2017 at 5:59 PM, Damian Guy 
>> wrote:
>>
>> > Hi,
>> >
>> > It seems like it must be time for 0.11.0.1 bug fix release!
>> >
>> > Since the 0.11.0.0 release we've fixed 30 JIRAs that
>> > are targeted for 0.11.0.1:
>> >
>> > https://issues.apache.org/jira/browse/KAFKA-5700?jql=
>> > project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Fixed%
>> > 20AND%20fixVersion%20%3D%200.11.0.1%20ORDER%20BY%
>> > 20priority%20DESC%2C%20key%20DESC
>> >
>> > We have 15 outstanding issues that are targeted at 0. <
>> http://0.10.2.1/>
>> > 11.0.1:
>> >
>> > https://issues.apache.org/jira/browse/KAFKA-5567?jql=
>> > project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Unresolved%20AND%
>> > 20fixVersion%20%3D%200.11.0.1%20ORDER%20BY%20priority%
>> > 20DESC%2C%20key%20DESC
>> >
>> > Can the owners of the remaining issues please resolve them or move
>> them to
>> > a future release.
>> >
>> > As soon as the remaining tasks for 0.11.0.1 reaches zero i'll
>> create the
>> > first RC.
>> >
>> > Thanks,
>> > Damian
>> >
>>
>

Dry-run option for new admin commands

[Viktor Somogyi]

Hi all,

I've been working on creating a KIP for one of the jiras that aim to
refactor the admin commands (KAFKA-3268
 -> KAFKA-5722
) and I see that the KIPs
related to these are still under discussion or voting (KIP-183

and
KIP-179

).

Hence I'd like to raise the question separately whether it'd be useful to
create a --dry-run option generally for every new admin command going
forward. I think that this would be very useful for provisioning changes,
especially if somebody wants to make sure they don't crash the prod system
with their change and the result of the command is what they really want.
Furthermore automatic provisioning tools would also benefit from this.

I imagine that in case of the dry-run the broker side wouldn't run the
command but only provide the changed state in some form. For example the
changed partitions in case of the partition reassignment tool or the
altered topics in case of the config-commands tool, however the format
really depends on the tool being used.

What do you guys think? Would such an addition make sense?

Regards,
Viktor

Re: Dry-run option for new admin commands

[Ted Yu]

State change on broker side may be subject to various factors (network,
disk, etc).

How would the dry-run report state change which is close to what actually
happens in non-dry-run mode ?

Thanks

On Fri, Sep 1, 2017 at 7:48 AM, Viktor Somogyi 
wrote:

> Hi all,
>
> I've been working on creating a KIP for one of the jiras that aim to
> refactor the admin commands (KAFKA-3268
>  -> KAFKA-5722
> ) and I see that the
> KIPs
> related to these are still under discussion or voting (KIP-183
>  PreferredReplicaLeaderElectionCommand+to+use+AdminClient>
> and
> KIP-179
>  ReassignPartitionsCommand+to+use+AdminClient>
> ).
>
> Hence I'd like to raise the question separately whether it'd be useful to
> create a --dry-run option generally for every new admin command going
> forward. I think that this would be very useful for provisioning changes,
> especially if somebody wants to make sure they don't crash the prod system
> with their change and the result of the command is what they really want.
> Furthermore automatic provisioning tools would also benefit from this.
>
> I imagine that in case of the dry-run the broker side wouldn't run the
> command but only provide the changed state in some form. For example the
> changed partitions in case of the partition reassignment tool or the
> altered topics in case of the config-commands tool, however the format
> really depends on the tool being used.
>
> What do you guys think? Would such an addition make sense?
>
> Regards,
> Viktor
>

Re: [VOTE] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

[Attila Kreiner]

Hi All,

Looks like accepted. Updating KIP page.

Thx,
Attila

2017-08-31 19:22 GMT+02:00 Matthias J. Sax :

> +1
>
> On 8/31/17 8:49 AM, Jason Gustafson wrote:
> > +1
> >
> > On Thu, Aug 31, 2017 at 8:36 AM, Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> >
> >> +1 non binding
> >> Thanks
> >>
> >> On Thu, Aug 31, 2017 at 8:35 AM, Guozhang Wang 
> wrote:
> >>> ＋1
> >>>
> >>> On Thu, Aug 31, 2017 at 7:57 AM, Bill Bejeck 
> wrote:
> >>>
>  +1
> 
>  Thanks,
>  Bill
> 
>  On Thu, Aug 31, 2017 at 10:31 AM, Vahid S Hashemian <
>  vahidhashem...@us.ibm.com> wrote:
> 
> > +1 (non-binding)
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From:   Molnár Bálint 
> > To: dev@kafka.apache.org
> > Date:   08/31/2017 02:13 AM
> > Subject:Re: [VOTE] KIP-191: KafkaConsumer.subscribe()
> overload
> > that takes just Pattern
> >
> >
> >
> > +1 (non-binding)
> >
> > 2017-08-31 10:33 GMT+02:00 Manikumar :
> >
> >> +1 (non-binding)
> >>
> >> On Thu, Aug 31, 2017 at 1:53 PM, Ismael Juma 
>  wrote:
> >>
> >>> Thanks for the KIP, +1 (binding).
> >>>
> >>> Ismael
> >>>
> >>> On 31 Aug 2017 8:38 am, "Attila Kreiner" 
> >> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> Thx for the comments, I pretty much see a consensus here. So I'd
> >> like
> > to
> >>> start the vote for:
> >>>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> > apache.org_confluence_display_KAFKA_KIP-2D&d=DwIBaQ&c=jf_
> > iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc&m=-
> > tCQqMGtotzV0UuQdwhdYJ745XQzBJp5lz9O1oM9QgA&s=Wq-_8a-
> > 94g2UGfy8hlJcx9WMdxK0WJRZ7V2ex2qKPpY&e=
> >
> >>> 191%3A+KafkaConsumer.
> >>> subscribe%28%29+overload+that+takes+just+Pattern
> >>>
> >>> Cheers,
> >>> Attila
> >>>
> >>
> >
> >
> >
> >
> >
> 
> >>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>
> >
>
>

[jira] [Created] (KAFKA-5820) Remove unneeded synchronized keyword in StreamThread

[Ted Yu (JIRA)]

Ted Yu created KAFKA-5820:
-

 Summary: Remove unneeded synchronized keyword in StreamThread
 Key: KAFKA-5820
 URL: https://issues.apache.org/jira/browse/KAFKA-5820
 Project: Kafka
  Issue Type: Improvement
Reporter: Ted Yu
Priority: Minor


There are three methods in StreamThread which have unnecessary synchronized 
keyword since the variable accessed, state, is volatile :

isRunningAndNotRebalancing
isRunning
shutdown

synchronized keyword can be dropped for these methods.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3777: KAFKA-5820 Remove unneeded synchronized keyword in...

[tedyu]

GitHub user tedyu opened a pull request:

https://github.com/apache/kafka/pull/3777

KAFKA-5820 Remove unneeded synchronized keyword in StreamThread

I removed synchronized keyword from 3 methods.

I ran the change thru streams module where test suite passed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tedyu/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3777.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3777


commit ef79c50e5e20236c589bbea2588fa7e11a892953
Author: tedyu 
Date:   2017-09-01T15:28:14Z

KAFKA-5820 Remove unneeded synchronized keyword in StreamThread




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5785) Always close connection if KafkaChannel.setSend throws exception

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5785.

Resolution: Duplicate

KAFKA-5607 includes a fix for this.

> Always close connection if KafkaChannel.setSend throws exception
> 
>
> Key: KAFKA-5785
> URL: https://issues.apache.org/jira/browse/KAFKA-5785
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> The code is currently:
> {code}
> try {
> channel.setSend(send);
> } catch (CancelledKeyException e) {
> this.failedSends.add(connectionId);
> close(channel, false);
> }
> {code}
> This is generally OK, but if another exception is thrown (typically due to a 
> bug), we leak the connection.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

[Jason Gustafson]

Hi Attila,

We should allow 3 days for the vote even if it is clear it will pass.

Thanks,
Jason

On Fri, Sep 1, 2017 at 8:16 AM, Attila Kreiner  wrote:

> Hi All,
>
> Looks like accepted. Updating KIP page.
>
> Thx,
> Attila
>
> 2017-08-31 19:22 GMT+02:00 Matthias J. Sax :
>
> > +1
> >
> > On 8/31/17 8:49 AM, Jason Gustafson wrote:
> > > +1
> > >
> > > On Thu, Aug 31, 2017 at 8:36 AM, Mickael Maison <
> > mickael.mai...@gmail.com>
> > > wrote:
> > >
> > >> +1 non binding
> > >> Thanks
> > >>
> > >> On Thu, Aug 31, 2017 at 8:35 AM, Guozhang Wang 
> > wrote:
> > >>> ＋1
> > >>>
> > >>> On Thu, Aug 31, 2017 at 7:57 AM, Bill Bejeck 
> > wrote:
> > >>>
> >  +1
> > 
> >  Thanks,
> >  Bill
> > 
> >  On Thu, Aug 31, 2017 at 10:31 AM, Vahid S Hashemian <
> >  vahidhashem...@us.ibm.com> wrote:
> > 
> > > +1 (non-binding)
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From:   Molnár Bálint 
> > > To: dev@kafka.apache.org
> > > Date:   08/31/2017 02:13 AM
> > > Subject:Re: [VOTE] KIP-191: KafkaConsumer.subscribe()
> > overload
> > > that takes just Pattern
> > >
> > >
> > >
> > > +1 (non-binding)
> > >
> > > 2017-08-31 10:33 GMT+02:00 Manikumar :
> > >
> > >> +1 (non-binding)
> > >>
> > >> On Thu, Aug 31, 2017 at 1:53 PM, Ismael Juma 
> >  wrote:
> > >>
> > >>> Thanks for the KIP, +1 (binding).
> > >>>
> > >>> Ismael
> > >>>
> > >>> On 31 Aug 2017 8:38 am, "Attila Kreiner" 
> > >> wrote:
> > >>>
> > >>> Hi All,
> > >>>
> > >>> Thx for the comments, I pretty much see a consensus here. So I'd
> > >> like
> > > to
> > >>> start the vote for:
> > >>>
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> > > apache.org_confluence_display_KAFKA_KIP-2D&d=DwIBaQ&c=jf_
> > > iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> > kjJc7uSVcviKUc&m=-
> > > tCQqMGtotzV0UuQdwhdYJ745XQzBJp5lz9O1oM9QgA&s=Wq-_8a-
> > > 94g2UGfy8hlJcx9WMdxK0WJRZ7V2ex2qKPpY&e=
> > >
> > >>> 191%3A+KafkaConsumer.
> > >>> subscribe%28%29+overload+that+takes+just+Pattern
> > >>>
> > >>> Cheers,
> > >>> Attila
> > >>>
> > >>
> > >
> > >
> > >
> > >
> > >
> > 
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>
> > >
> >
> >
>

Re: Dry-run option for new admin commands

[Viktor Somogyi]

Yea, I think we have to make a constraint that the dry-run is valid as far
as the environment doesn't change.

Also I think too that we can dismiss reporting admin operations (like
--describe) as they won't be that useful and focus only on those which are
altering.

(Hope I got your questions right :) )

On 1 Sep 2017 16:54, "Ted Yu"  wrote:

State change on broker side may be subject to various factors (network,
disk, etc).

How would the dry-run report state change which is close to what actually
happens in non-dry-run mode ?

Thanks

On Fri, Sep 1, 2017 at 7:48 AM, Viktor Somogyi 
wrote:

> Hi all,
>
> I've been working on creating a KIP for one of the jiras that aim to
> refactor the admin commands (KAFKA-3268
>  -> KAFKA-5722
> ) and I see that the
> KIPs
> related to these are still under discussion or voting (KIP-183
>  PreferredReplicaLeaderElectionCommand+to+use+AdminClient>
> and
> KIP-179
>  ReassignPartitionsCommand+to+use+AdminClient>
> ).
>
> Hence I'd like to raise the question separately whether it'd be useful to
> create a --dry-run option generally for every new admin command going
> forward. I think that this would be very useful for provisioning changes,
> especially if somebody wants to make sure they don't crash the prod system
> with their change and the result of the command is what they really want.
> Furthermore automatic provisioning tools would also benefit from this.
>
> I imagine that in case of the dry-run the broker side wouldn't run the
> command but only provide the changed state in some form. For example the
> changed partitions in case of the partition reassignment tool or the
> altered topics in case of the config-commands tool, however the format
> really depends on the tool being used.
>
> What do you guys think? Would such an addition make sense?
>
> Regards,
> Viktor
>

Request access to Wiki

[charly molter]

Hi,

I would like to open a KIP and start a discussion.

Could someone please grant me access.

My username is cmolter: https://cwiki.apache.org/confluence/display/~cmolter

Thanks!

-- 
Charly Molter

[GitHub] kafka pull request #3709: KAFKA-5759; Allow user to specify relative path as...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3709


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-5821) Intermittent test failure in SaslPlainSslEndToEndAuthorizationTest.testAcls

[Ted Yu (JIRA)]

Ted Yu created KAFKA-5821:
-

 Summary: Intermittent test failure in 
SaslPlainSslEndToEndAuthorizationTest.testAcls
 Key: KAFKA-5821
 URL: https://issues.apache.org/jira/browse/KAFKA-5821
 Project: Kafka
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


>From 
>https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/7245/testReport/junit/kafka.api/SaslPlainSslEndToEndAuthorizationTest/testAcls/
> :
{code}
java.lang.SecurityException: zookeeper.set.acl is true, but the verification of 
the JAAS login file failed.
at kafka.server.KafkaServer.initZk(KafkaServer.scala:329)
at kafka.server.KafkaServer.startup(KafkaServer.scala:192)
at kafka.utils.TestUtils$.createServer(TestUtils.scala:134)
at 
kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:94)
at 
kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:93)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at 
kafka.integration.KafkaServerTestHarness.setUp(KafkaServerTestHarness.scala:93)
at 
kafka.api.IntegrationTestHarness.setUp(IntegrationTestHarness.scala:66)
at 
kafka.api.EndToEndAuthorizationTest.setUp(EndToEndAuthorizationTest.scala:158)
at 
kafka.api.SaslEndToEndAuthorizationTest.setUp(SaslEndToEndAuthorizationTest.scala:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.gradle.internal.dispatch.Reflection

[jira] [Created] (KAFKA-5822) Consistent logging of topic partitions

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-5822:
--

 Summary: Consistent logging of topic partitions
 Key: KAFKA-5822
 URL: https://issues.apache.org/jira/browse/KAFKA-5822
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson


In some cases partitions are logged as "[topic,partition]" and in others as 
"topic-partition." It would be nice to standardize to make searching easier.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Fault Injection

[Colin McCabe]

Thanks, Victor.  Also check out the fault injection umbrella JIRA here:
https://issues.apache.org/jira/browse/KAFKA-5775 with more subtasks.

cheers,
Colin


On Fri, Sep 1, 2017, at 05:07, Viktor Somogyi wrote:
> Hi Colin,
> 
> I'd be interested in this and also think it's a valuable thing to have
> this
> for the community and would greatly increase the test coverage.
> Saw you already have a PR, I'll give a review as I have time :).
> 
> Viktor
> 
> On Tue, Aug 22, 2017 at 9:36 PM, Timothy Chen  wrote:
> 
> > Hi Colin,
> >
> > The Kibosh code is just a README for now, is it going to be published soon?
> >
> > Tim
> >
> > On Tue, Aug 22, 2017 at 11:44 AM, Colin McCabe  wrote:
> > > Hi all,
> > >
> > > I've been working on a fault injector for Apache Kafka.  The general
> > > idea is to create faults such as network partitions or disk failures,
> > > and see what happens in the cluster.  The fault injector can run as part
> > > of a ducktape system test, or standalone.
> > >
> > > The fault injector has two processes: a coordinator, and an agent.  The
> > > agent process is responsible for actually implementing the faults.  For
> > > example, it might run iptables, send signals to processes, generate a
> > > lot of load, or do something else to disrupt the computer it is running
> > > on.  We run an agent process on each node where we would like to
> > > potentially inject faults.  So it will run alongside the brokers,
> > > zookeeper nodes, etc.
> > >
> > > The coordinator process is responsible for communicating with the agent
> > > processes and for scheduling faults.  For example, the coordinator can
> > > be instructed to create a fault immediately on several nodes.  Or it can
> > > be instructed to create faults over time, based on a pseudorandom seed.
> > > Both the coordinator and the agent expose a REST interface that accepts
> > > objects serialized via JSON.
> > >
> > > I think two kinds of faults will be especially interesting: network
> > > faults, and disk errors.  Simulating network faults in a Linux
> > > environment is relatively straightforward using iptables.  Disk errors
> > > are tougher to simulate, but I have written a FUSE filesystem to do
> > > this.  The  filesystem essentially simulates a bind mount in most cases,
> > > but it can take a JSON specification telling it to inject certain
> > > faults.  (Disk errors seem especially relevant to the ongoing work on
> > > JBOD.)
> > >
> > > Although it's not a user-visible component, I think having a fault
> > > injector will be really great for Kafka users.  It will really help us
> > > stress test Kafka in more situations.  I'm going to post some patches in
> > > a day or two-- it would be great to get some feedback.  Check out
> > > https://cwiki.apache.org/confluence/display/KAFKA/Fault+Injection
> > >
> > > best,
> > > Colin
> >

Re: Request access to Wiki

[Jun Rao]

Hi, Charly,

Thanks for your interest. Just granted you the permission to the wiki.

Jun

On Fri, Sep 1, 2017 at 9:22 AM, charly molter 
wrote:

> Hi,
>
> I would like to open a KIP and start a discussion.
>
> Could someone please grant me access.
>
> My username is cmolter: https://cwiki.apache.org/
> confluence/display/~cmolter
>
> Thanks!
>
> --
> Charly Molter
>

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Bill Bejeck]

+1

On Thu, Aug 31, 2017 at 10:20 AM, Damian Guy  wrote:

> Thanks everyone for voting! Unfortunately i've had to make a bit of an
> update based on some issues found during implementation.
> The main changes are:
> BytesStoreSupplier -> StoreSupplier
> Addition of:
> WindowBytesStoreSupplier, KeyValueBytesStoreSupplier,
> SessionBytesStoreSupplier that will restrict store types to 
> 3 new overloads added to Materialized to enable developers to create a
> Materialized of the appropriate type, i..e, WindowStore etc
> Update DSL where Materialized is used such that the stores have generic
> types of 
> Some minor changes to the arguments to Store#persistentWindowStore and
> Store#persistentSessionStore
>
> Please take a look and recast the votes.
>
> Thanks for your time,
> Damian
>
> On Fri, 25 Aug 2017 at 17:05 Matthias J. Sax 
> wrote:
>
> > Thanks Damian. Great KIP!
> >
> > +1
> >
> >
> > -Matthias
> >
> > On 8/25/17 6:45 AM, Damian Guy wrote:
> > > Hi,
> > >
> > > I've just realised we need to add two methods to StateStoreBuilder or
> it
> > > isn't going to work:
> > >
> > > Map logConfig();
> > > boolean loggingEnabled();
> > >
> > > These are needed when we are building the topology and determining
> > > changelog topic names and configs.
> > >
> > >
> > > I've also update the KIP to add
> > >
> > > StreamBuilder#stream(String topic)
> > >
> > > StreamBuilder#stream(String topic, Consumed options)
> > >
> > >
> > > Thanks
> > >
> > >
> > > On Thu, 24 Aug 2017 at 22:11 Sriram Subramanian 
> > wrote:
> > >
> > >> +1
> > >>
> > >> On Thu, Aug 24, 2017 at 10:20 AM, Guozhang Wang 
> > >> wrote:
> > >>
> > >>> +1. Thanks Damian!
> > >>>
> > >>> On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck 
> > wrote:
> > >>>
> >  Thanks for the KIP!
> > 
> >  +1
> > 
> >  Thanks,
> >  Bill
> > 
> >  On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy 
> > >>> wrote:
> > 
> > > Hi,
> > >
> > > I'd like to kick off the voting thread for KIP-182:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
> > > use+of+custom+storage+engines
> > >
> > > Thanks,
> > > Damian
> > >
> > 
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>>
> > >>
> > >
> >
> >
>

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Matthias J. Sax]

+1

On 9/1/17 2:53 PM, Bill Bejeck wrote:
> +1
> 
> On Thu, Aug 31, 2017 at 10:20 AM, Damian Guy  wrote:
> 
>> Thanks everyone for voting! Unfortunately i've had to make a bit of an
>> update based on some issues found during implementation.
>> The main changes are:
>> BytesStoreSupplier -> StoreSupplier
>> Addition of:
>> WindowBytesStoreSupplier, KeyValueBytesStoreSupplier,
>> SessionBytesStoreSupplier that will restrict store types to 
>> 3 new overloads added to Materialized to enable developers to create a
>> Materialized of the appropriate type, i..e, WindowStore etc
>> Update DSL where Materialized is used such that the stores have generic
>> types of 
>> Some minor changes to the arguments to Store#persistentWindowStore and
>> Store#persistentSessionStore
>>
>> Please take a look and recast the votes.
>>
>> Thanks for your time,
>> Damian
>>
>> On Fri, 25 Aug 2017 at 17:05 Matthias J. Sax 
>> wrote:
>>
>>> Thanks Damian. Great KIP!
>>>
>>> +1
>>>
>>>
>>> -Matthias
>>>
>>> On 8/25/17 6:45 AM, Damian Guy wrote:
 Hi,

 I've just realised we need to add two methods to StateStoreBuilder or
>> it
 isn't going to work:

 Map logConfig();
 boolean loggingEnabled();

 These are needed when we are building the topology and determining
 changelog topic names and configs.


 I've also update the KIP to add

 StreamBuilder#stream(String topic)

 StreamBuilder#stream(String topic, Consumed options)


 Thanks


 On Thu, 24 Aug 2017 at 22:11 Sriram Subramanian 
>>> wrote:

> +1
>
> On Thu, Aug 24, 2017 at 10:20 AM, Guozhang Wang 
> wrote:
>
>> +1. Thanks Damian!
>>
>> On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck 
>>> wrote:
>>
>>> Thanks for the KIP!
>>>
>>> +1
>>>
>>> Thanks,
>>> Bill
>>>
>>> On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy 
>> wrote:
>>>
 Hi,

 I'd like to kick off the voting thread for KIP-182:
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-
 182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
 use+of+custom+storage+engines

 Thanks,
 Damian

>>>
>>
>>
>>
>> --
>> -- Guozhang
>>
>

>>>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

[GitHub] kafka pull request #3778: KAFKA-5822: Consistent log formatting of topic par...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/3778

KAFKA-5822: Consistent log formatting of topic partitions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-5822

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3778.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3778


commit a40373c229510fcadd6156253a59f6d7badf5d47
Author: Jason Gustafson 
Date:   2017-09-01T23:35:05Z

KAFKA-5822: Consistent log formatting of topic partitions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-5823) Update Docs

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-5823:
--

 Summary: Update Docs
 Key: KAFKA-5823
 URL: https://issues.apache.org/jira/browse/KAFKA-5823
 Project: Kafka
  Issue Type: Sub-task
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5824) Cannot write to key value store provided by ProcessorTopologyTestDriver

[Dmitry Minkovsky (JIRA)]

Dmitry Minkovsky created KAFKA-5824:
---

 Summary: Cannot write to key value store provided by 
ProcessorTopologyTestDriver
 Key: KAFKA-5824
 URL: https://issues.apache.org/jira/browse/KAFKA-5824
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.11.0.0
Reporter: Dmitry Minkovsky


I am trying to `put()` to a KeyValueStore that I got from 
ProcessorTopologyTestDriver#getKeyValueStore() as part of setup for a test. The 
JavaDoc endorses this use-case:

 * This is often useful in test cases to pre-populate the store before the 
test case instructs the topology to
 * {@link #process(String, byte[], byte[]) process an input message}, 
and/or to check the store afterward.

However, the `put()` results in the following error: 

{{
java.lang.IllegalStateException: This should not happen as offset() should only 
be called while a record is processed

at 
org.apache.kafka.streams.processor.internals.AbstractProcessorContext.offset(AbstractProcessorContext.java:139)
at 
org.apache.kafka.streams.state.internals.CachingKeyValueStore.put(CachingKeyValueStore.java:193)
at 
org.apache.kafka.streams.state.internals.CachingKeyValueStore.put(CachingKeyValueStore.java:188)
at 
pony.UserEntityTopologySupplierTest.confirm-settings-requests(UserEntityTopologySupplierTest.groovy:81)
}}

This error seems straightforward: I am not doing the `put` within the context 
of stream processing. How do I reconcile this with the fact that I am trying to 
populate the store for a test, which the JavaDoc endorses?

Thank you,
Dmitry



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

[Attila Kreiner]

Hi Jason,

Sorry, I didn't know that one. Then I assume we should wait a few days
before moving forward.

Regards,
Attila

2017-09-01 18:01 GMT+02:00 Jason Gustafson :

> Hi Attila,
>
> We should allow 3 days for the vote even if it is clear it will pass.
>
> Thanks,
> Jason
>
> On Fri, Sep 1, 2017 at 8:16 AM, Attila Kreiner  wrote:
>
> > Hi All,
> >
> > Looks like accepted. Updating KIP page.
> >
> > Thx,
> > Attila
> >
> > 2017-08-31 19:22 GMT+02:00 Matthias J. Sax :
> >
> > > +1
> > >
> > > On 8/31/17 8:49 AM, Jason Gustafson wrote:
> > > > +1
> > > >
> > > > On Thu, Aug 31, 2017 at 8:36 AM, Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > >> +1 non binding
> > > >> Thanks
> > > >>
> > > >> On Thu, Aug 31, 2017 at 8:35 AM, Guozhang Wang 
> > > wrote:
> > > >>> ＋1
> > > >>>
> > > >>> On Thu, Aug 31, 2017 at 7:57 AM, Bill Bejeck 
> > > wrote:
> > > >>>
> > >  +1
> > > 
> > >  Thanks,
> > >  Bill
> > > 
> > >  On Thu, Aug 31, 2017 at 10:31 AM, Vahid S Hashemian <
> > >  vahidhashem...@us.ibm.com> wrote:
> > > 
> > > > +1 (non-binding)
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From:   Molnár Bálint 
> > > > To: dev@kafka.apache.org
> > > > Date:   08/31/2017 02:13 AM
> > > > Subject:Re: [VOTE] KIP-191: KafkaConsumer.subscribe()
> > > overload
> > > > that takes just Pattern
> > > >
> > > >
> > > >
> > > > +1 (non-binding)
> > > >
> > > > 2017-08-31 10:33 GMT+02:00 Manikumar  >:
> > > >
> > > >> +1 (non-binding)
> > > >>
> > > >> On Thu, Aug 31, 2017 at 1:53 PM, Ismael Juma  >
> > >  wrote:
> > > >>
> > > >>> Thanks for the KIP, +1 (binding).
> > > >>>
> > > >>> Ismael
> > > >>>
> > > >>> On 31 Aug 2017 8:38 am, "Attila Kreiner" 
> > > >> wrote:
> > > >>>
> > > >>> Hi All,
> > > >>>
> > > >>> Thx for the comments, I pretty much see a consensus here. So
> I'd
> > > >> like
> > > > to
> > > >>> start the vote for:
> > > >>>
> > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> > > > apache.org_confluence_display_KAFKA_KIP-2D&d=DwIBaQ&c=jf_
> > > > iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> > > kjJc7uSVcviKUc&m=-
> > > > tCQqMGtotzV0UuQdwhdYJ745XQzBJp5lz9O1oM9QgA&s=Wq-_8a-
> > > > 94g2UGfy8hlJcx9WMdxK0WJRZ7V2ex2qKPpY&e=
> > > >
> > > >>> 191%3A+KafkaConsumer.
> > > >>> subscribe%28%29+overload+that+takes+just+Pattern
> > > >>>
> > > >>> Cheers,
> > > >>> Attila
> > > >>>
> > > >>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > 
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> -- Guozhang
> > > >>
> > > >
> > >
> > >
> >
>