Re: Build problems in Eclipse dev environment setup

2017-01-08 Thread Dhwani Katagade

Hi Ismael,

Thanks for your response. I agree this is not a Kafka issue, its about 
gradle and its eclipse plugin. I was able to resolve the compile errors 
in Eclipse by manually setting the Default output folder for core 
project to core/build/classes/main.


I understand Eclipse is not the popular choice for an IDE but, for me 
and my other Eclipse brethren, I looked into the problems I faced and 
noticed that the output folders for gradle and those for eclipse get 
configured differently. The output folders in eclipse go under 
/bin where as in gradle they are under /build. Due to 
this the projects in Eclipse don't work fine unless the command line 
build is completed at least once. Will it be fine if we could align 
these so that inter project dependencies fall well in place in either 
build? *I was looking into the possibilities of fixing that and I have a 
change available that I am testing out.*


There is another discussion on the pros and cons of this here 
https://discuss.gradle.org/t/eclipse-generated-files-should-be-put-in-the-same-place-as-the-gradle-generated-files/6986


People, please let me know your opinion and I will provide a PR for this 
if I get this working for myself.


thanks
-dhwani

On 01/07/2017 07:02 PM, Ismael Juma wrote:

Hi Dhwani,

This is more about Gradle and Eclipse than Kafka and many of us use
IntelliJ IDEA so won't be able to help you. I can confirm that IntelliJ is
able to import the project fine out of the box.

Ismael

On 3 Jan 2017 6:45 pm, "Dhwani Katagade" 
wrote:


Hi,

I am trying to setup the eclipse dev environment on trunk. I have followed
the instructions given here
https://github.com/apache/kafka/blob/trunk/README.md
https://cwiki.apache.org/confluence/display/KAFKA/Developer+
Setup#DeveloperSetup-Eclipsesetup

After I run ./gradlew eclipse in the command line followed by "Import
existing projects into workspace" in Eclipse, I am able to see the
projects. But I see the following build errors in Eclipse.

Description: The import kafka.utils.ShutdownableThread cannot be resolved
Resource: Consumer.java
Path: /kafka-examples/src/main/java/kafka/examples
Location: line 19
Type: Java Problem

There are other errors as well but they are of a similar nature and others
dependent on them. I can see the class ShutdownableThread in core and core
is compiled properly. The examples project has core setup as  dependency
project which seems correct. Also the command line build works just fine.

Also I don't see any .class files generated under core/bin. They are
present under core/build/classes/main/ though. Is this due to some kind of
mismatch between the command line build and Eclipse build?

Could you please suggest what am I missing here.

Environment:
- Eclipse Neon
- Java 1.7
- Scala 2.10.6
- CentOS 6.8

thanks
-dhwani


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is
the property of Persistent Systems Ltd. It is intended only for the use of
the individual or entity to which it is addressed. If you are not the
intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in
error, please notify the sender and delete all copies of this message.
Persistent Systems Ltd. does not accept any liability for virus infected
mails.





DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.



Re: [DISCUSS] KIP-104: Granular Sensors for Streams

2017-01-08 Thread Ismael Juma
Thanks for updating the KIP to include the Metrics API that we are now
exposing, very helpful! Looking at it, do we really want to expose it to
users? The API seems to have grown organically (as an internal API) and
doesn't look particularly user-friendly to me. Questions I have:

1. `metricName` and `sensor` have a huge number of overloads. We usually
try to avoid that in public APIs by using a class that encapsulates the
parameters. Also, `inactiveSensorExpirationTimeSeconds` doesn't make sense
for clients since the thread that purges inactive sensors is only enabled
in the broker.
2. Do we want to expose `removeSensor`?
3. Do we want to expose `addReporter`?
4. We typically have methods without `get` as accessors, but here we have
`getSensor` for the accessor and `sensor` is really `getOrCreateSensor`.
Maybe it's justified based the typical usage (but it would be good to
confirm)?
5. I didn't understand the comment about why an interface wouldn't work. If
it's a MetricsRegistry interface, it could live in the clients module
(outside Streams as you said), but that is not necessarily an issue as far
as I can see.

Thanks,
Ismael

P.S. Here's the list of methods grouped and without javadoc to make it
easier to see what I mean:

//metricName overloads
public MetricName metricName(String name, String group, String description,
Map tags);
public MetricName metricName(String name, String group, String description);
public MetricName metricName(String name, String group);
public MetricName metricName(String name, String group, String description,
String... keyValue);
public MetricName metricName(String name, String group, Map
tags);

//sensor overloads
public Sensor sensor(String name);
public Sensor sensor(String name, Sensor.RecordLevel recordLevel);
public Sensor sensor(String name, Sensor... parents);
public Sensor sensor(String name, Sensor.RecordLevel recordLevel, Sensor...
parents);
public synchronized Sensor sensor(String name, MetricConfig config,
Sensor... parents);
public synchronized Sensor sensor(String name, MetricConfig config,
Sensor.RecordLevel recordLevel, Sensor... parents);
public synchronized Sensor sensor(String name, MetricConfig config,
Sensor.RecordLevel recordLevel, Sensor... parents);
public synchronized Sensor sensor(String name, MetricConfig config, long
inactiveSensorExpirationTimeSeconds, Sensor.RecordLevel recordLevel,
Sensor... parents);

public MetricConfig config();

public Sensor getSensor(String name);

public void removeSensor(String name);

public void addMetric(MetricName metricName, Measurable measurable);
public synchronized void addMetric(MetricName metricName, MetricConfig
config, Measurable measurable);
public synchronized KafkaMetric removeMetric(MetricName metricName);

public synchronized void addReporter(MetricsReporter reporter);

public Map metrics();

public KafkaMetric metric(MetricName metricName);

On Sat, Jan 7, 2017 at 12:15 AM, Eno Thereska 
wrote:

> Ok, I'll list all the methods in the Metrics class for completion. An
> interface won't work since it will have to reside outside of streams
> unfortunately.
>
> Thanks
> Eno
> > On 6 Jan 2017, at 23:54, Ismael Juma  wrote:
> >
> > Hi Guozhang,
> >
> > I understand the requirement and I don't have an issue with that. My
> point
> > is that the `Metrics` registry API is becoming public via this KIP so we
> > should ensure that it's suitable. It may make sense to introduce an
> > interface (say MetricsRegistry) that exposes a reasonable subset (do we
> > want to expose `removeSensor` for example?). This is relatively
> > straightforward and little additional code.
> >
> > Ismael
> >
> > On Fri, Jan 6, 2017 at 11:29 PM, Guozhang Wang 
> wrote:
> >
> >> Unlike Producer and Consumer, Streams users may likely to add their own
> >> sensors depending on their apps and that is the main reason we added
> >> facilities to let them register customized "throughput" "latency" and
> any
> >> generic sensors.
> >>
> >> I think Eno has thought about just adding an API in StreamsMetrics to
> >> register any sensors, which will be directly translated into a
> >> `metrics.sensor` call. In the end he decided to just expose the registry
> >> itself since the functions itself seem just like duplicating the same
> >> `sensor` functions in `Metrics`.
> >>
> >>
> >>
> >> Guozhang
> >>
> >> On Fri, Jan 6, 2017 at 2:16 PM, Ismael Juma  wrote:
> >>
> >>> Thanks for the explanation Eno. The KIP did mention that the metrics
> >>> registry would be exposed, yes. What is missing is that the registry is
> >> not
> >>> currently exposed by anything else. Traditionally, we list all public
> >> APIs
> >>> created by a KIP, which is effectively true for the registry in this
> >> case.
> >>> Did we consider using an interface instead of the concrete class? It
> >> seems
> >>> that a lot of these things were discussed in the PR, so it would be
> good
> >> to
> >>> have a summary in the KIP too.
> >>>
> >>> Ismael
> >>>
> >>> On Fri, Jan 6, 2017 a

[VOTE] KIP-108: Create Topic Policy

2017-01-08 Thread Ismael Juma
Hi all,

As the discussion seems to have settled down, I would like to initiate the
voting process for KIP-108: Create Topic Policy:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-108
%3A+Create+Topic+Policy

The vote will run for a minimum of 72 hours.

Thanks,
Ismael


Re: [DISCUSS] KIP-104: Granular Sensors for Streams

2017-01-08 Thread Eno Thereska
Ismael,

Based on the streams user demand for registering their sensors and metrics we 
decided to expose the metrics registry. I'm not a fan of replicating a ton of 
code and add yet another layer/interface that does the same thing that the 
existing Metrics class and I think the methods are pretty basic (e.g., we do 
need 'removeSensor'). 

I can look into a subsequent JIRA to fix style issues such as your point 4 
(remove 'get') directly on the Metrics class.

Thanks
Eno


> On 8 Jan 2017, at 14:42, Ismael Juma  wrote:
> 
> Thanks for updating the KIP to include the Metrics API that we are now
> exposing, very helpful! Looking at it, do we really want to expose it to
> users? The API seems to have grown organically (as an internal API) and
> doesn't look particularly user-friendly to me. Questions I have:
> 
> 1. `metricName` and `sensor` have a huge number of overloads. We usually
> try to avoid that in public APIs by using a class that encapsulates the
> parameters. Also, `inactiveSensorExpirationTimeSeconds` doesn't make sense
> for clients since the thread that purges inactive sensors is only enabled
> in the broker.
> 2. Do we want to expose `removeSensor`?
> 3. Do we want to expose `addReporter`?
> 4. We typically have methods without `get` as accessors, but here we have
> `getSensor` for the accessor and `sensor` is really `getOrCreateSensor`.
> Maybe it's justified based the typical usage (but it would be good to
> confirm)?
> 5. I didn't understand the comment about why an interface wouldn't work. If
> it's a MetricsRegistry interface, it could live in the clients module
> (outside Streams as you said), but that is not necessarily an issue as far
> as I can see.
> 
> Thanks,
> Ismael
> 
> P.S. Here's the list of methods grouped and without javadoc to make it
> easier to see what I mean:
> 
> //metricName overloads
> public MetricName metricName(String name, String group, String description,
> Map tags);
> public MetricName metricName(String name, String group, String description);
> public MetricName metricName(String name, String group);
> public MetricName metricName(String name, String group, String description,
> String... keyValue);
> public MetricName metricName(String name, String group, Map
> tags);
> 
> //sensor overloads
> public Sensor sensor(String name);
> public Sensor sensor(String name, Sensor.RecordLevel recordLevel);
> public Sensor sensor(String name, Sensor... parents);
> public Sensor sensor(String name, Sensor.RecordLevel recordLevel, Sensor...
> parents);
> public synchronized Sensor sensor(String name, MetricConfig config,
> Sensor... parents);
> public synchronized Sensor sensor(String name, MetricConfig config,
> Sensor.RecordLevel recordLevel, Sensor... parents);
> public synchronized Sensor sensor(String name, MetricConfig config,
> Sensor.RecordLevel recordLevel, Sensor... parents);
> public synchronized Sensor sensor(String name, MetricConfig config, long
> inactiveSensorExpirationTimeSeconds, Sensor.RecordLevel recordLevel,
> Sensor... parents);
> 
> public MetricConfig config();
> 
> public Sensor getSensor(String name);
> 
> public void removeSensor(String name);
> 
> public void addMetric(MetricName metricName, Measurable measurable);
> public synchronized void addMetric(MetricName metricName, MetricConfig
> config, Measurable measurable);
> public synchronized KafkaMetric removeMetric(MetricName metricName);
> 
> public synchronized void addReporter(MetricsReporter reporter);
> 
> public Map metrics();
> 
> public KafkaMetric metric(MetricName metricName);
> 
> On Sat, Jan 7, 2017 at 12:15 AM, Eno Thereska 
> wrote:
> 
>> Ok, I'll list all the methods in the Metrics class for completion. An
>> interface won't work since it will have to reside outside of streams
>> unfortunately.
>> 
>> Thanks
>> Eno
>>> On 6 Jan 2017, at 23:54, Ismael Juma  wrote:
>>> 
>>> Hi Guozhang,
>>> 
>>> I understand the requirement and I don't have an issue with that. My
>> point
>>> is that the `Metrics` registry API is becoming public via this KIP so we
>>> should ensure that it's suitable. It may make sense to introduce an
>>> interface (say MetricsRegistry) that exposes a reasonable subset (do we
>>> want to expose `removeSensor` for example?). This is relatively
>>> straightforward and little additional code.
>>> 
>>> Ismael
>>> 
>>> On Fri, Jan 6, 2017 at 11:29 PM, Guozhang Wang 
>> wrote:
>>> 
 Unlike Producer and Consumer, Streams users may likely to add their own
 sensors depending on their apps and that is the main reason we added
 facilities to let them register customized "throughput" "latency" and
>> any
 generic sensors.
 
 I think Eno has thought about just adding an API in StreamsMetrics to
 register any sensors, which will be directly translated into a
 `metrics.sensor` call. In the end he decided to just expose the registry
 itself since the functions itself seem just like duplicating the same
 `sensor` functions in `Metrics`.
>

Re: [DISCUSS] KIP-104: Granular Sensors for Streams

2017-01-08 Thread Eno Thereska
I can see the point that all of a sudden exposing the internal Metrics class 
might be too big a bite to take in this KIP, since the Metrics class might have 
to be cleaned up further. I was perhaps naively assuming it's good since it has 
been reviewed when it was first introduced into Kafka. Furthermore, that class 
brings in a whole bunch of other stuff, such as Min/Avg. They are all simple 
things, but others might argue differently.

An intermediate alternative would be to not expose the Metrics class for now. 
Users can still get a read-only Map of all the streams metrics, as they do in 
the Producer/Consumer classes. Furthermore, we already have helper functions to 
help users add throughput and latency metrics and that could be a good enough 
step for now. So:

- Metrics registry()
+ public Map readOnlyRegistry()

Ideally either vote on the previous approach or add a note to this discuss 
thread please since we're getting close to the feature freeze deadline.

Thanks
Eno

> On 8 Jan 2017, at 15:06, Eno Thereska  wrote:
> 
> Ismael,
> 
> Based on the streams user demand for registering their sensors and metrics we 
> decided to expose the metrics registry. I'm not a fan of replicating a ton of 
> code and add yet another layer/interface that does the same thing that the 
> existing Metrics class and I think the methods are pretty basic (e.g., we do 
> need 'removeSensor'). 
> 
> I can look into a subsequent JIRA to fix style issues such as your point 4 
> (remove 'get') directly on the Metrics class.
> 
> Thanks
> Eno
> 
> 
>> On 8 Jan 2017, at 14:42, Ismael Juma  wrote:
>> 
>> Thanks for updating the KIP to include the Metrics API that we are now
>> exposing, very helpful! Looking at it, do we really want to expose it to
>> users? The API seems to have grown organically (as an internal API) and
>> doesn't look particularly user-friendly to me. Questions I have:
>> 
>> 1. `metricName` and `sensor` have a huge number of overloads. We usually
>> try to avoid that in public APIs by using a class that encapsulates the
>> parameters. Also, `inactiveSensorExpirationTimeSeconds` doesn't make sense
>> for clients since the thread that purges inactive sensors is only enabled
>> in the broker.
>> 2. Do we want to expose `removeSensor`?
>> 3. Do we want to expose `addReporter`?
>> 4. We typically have methods without `get` as accessors, but here we have
>> `getSensor` for the accessor and `sensor` is really `getOrCreateSensor`.
>> Maybe it's justified based the typical usage (but it would be good to
>> confirm)?
>> 5. I didn't understand the comment about why an interface wouldn't work. If
>> it's a MetricsRegistry interface, it could live in the clients module
>> (outside Streams as you said), but that is not necessarily an issue as far
>> as I can see.
>> 
>> Thanks,
>> Ismael
>> 
>> P.S. Here's the list of methods grouped and without javadoc to make it
>> easier to see what I mean:
>> 
>> //metricName overloads
>> public MetricName metricName(String name, String group, String description,
>> Map tags);
>> public MetricName metricName(String name, String group, String description);
>> public MetricName metricName(String name, String group);
>> public MetricName metricName(String name, String group, String description,
>> String... keyValue);
>> public MetricName metricName(String name, String group, Map
>> tags);
>> 
>> //sensor overloads
>> public Sensor sensor(String name);
>> public Sensor sensor(String name, Sensor.RecordLevel recordLevel);
>> public Sensor sensor(String name, Sensor... parents);
>> public Sensor sensor(String name, Sensor.RecordLevel recordLevel, Sensor...
>> parents);
>> public synchronized Sensor sensor(String name, MetricConfig config,
>> Sensor... parents);
>> public synchronized Sensor sensor(String name, MetricConfig config,
>> Sensor.RecordLevel recordLevel, Sensor... parents);
>> public synchronized Sensor sensor(String name, MetricConfig config,
>> Sensor.RecordLevel recordLevel, Sensor... parents);
>> public synchronized Sensor sensor(String name, MetricConfig config, long
>> inactiveSensorExpirationTimeSeconds, Sensor.RecordLevel recordLevel,
>> Sensor... parents);
>> 
>> public MetricConfig config();
>> 
>> public Sensor getSensor(String name);
>> 
>> public void removeSensor(String name);
>> 
>> public void addMetric(MetricName metricName, Measurable measurable);
>> public synchronized void addMetric(MetricName metricName, MetricConfig
>> config, Measurable measurable);
>> public synchronized KafkaMetric removeMetric(MetricName metricName);
>> 
>> public synchronized void addReporter(MetricsReporter reporter);
>> 
>> public Map metrics();
>> 
>> public KafkaMetric metric(MetricName metricName);
>> 
>> On Sat, Jan 7, 2017 at 12:15 AM, Eno Thereska 
>> wrote:
>> 
>>> Ok, I'll list all the methods in the Metrics class for completion. An
>>> interface won't work since it will have to reside outside of streams
>>> unfortunately.
>>> 
>>> Thanks
>>> Eno
 On 6 Jan 2017, at 23:

Re: [VOTE] KIP-104: Granular Sensors for Streams

2017-01-08 Thread Bill Bejeck
+1

On Sat, Jan 7, 2017 at 1:06 PM, Matthias J. Sax 
wrote:

> +1
>
> On 1/6/17 4:48 PM, Sriram Subramanian wrote:
> > +1
> >
> > On Fri, Jan 6, 2017 at 4:41 PM, Eno Thereska 
> wrote:
> >
> >> I light of the recent discussion on KIP-104 I've made changes to the KIP
> >> to explicitly list all APIs of the Metrics class for completion. I think
> >> it's fair to re-start the vote on the KIP in light of the new discussion
> >> for transparency:
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 104%3A+Granular+Sensors+for+Streams  >> confluence/display/KAFKA/KIP-104:+Granular+Sensors+for+Streams>
> >>
> >> So could you please re-vote even if you've previously voted?
> >>
> >> Thank you
> >> no
> >>
> >>> On 6 Jan 2017, at 17:31, Sriram Subramanian  wrote:
> >>>
> >>> +1
> >>>
> >>> On Fri, Jan 6, 2017 at 9:12 AM, Matthias J. Sax  >
> >>> wrote:
> >>>
>  +1
> 
>  On 1/6/17 8:01 AM, Guozhang Wang wrote:
> > +1
> >
> > On Fri, Jan 6, 2017 at 5:05 AM, Bill Bejeck 
> wrote:
> >
> >> +1
> >>
> >> On Fri, Jan 6, 2017 at 5:57 AM, Damian Guy 
>  wrote:
> >>
> >>> +1
> >>>
> >>> On Fri, 6 Jan 2017 at 09:37 Eno Thereska 
>  wrote:
> >>>
>  The discussion points for KIP-104 are addressed. At this point
> we'd
> >> like
>  to start the vote for it:
> 
> 
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 104%3A+Granular+Sensors+for+Streams
>  <
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 104:+Granular+Sensors+for+Streams
> >
> 
>  Thanks,
>  Eno and Aarti
> >>>
> >>
> >
> >
> >
> 
> 
> >>
> >>
> >
>
>


[jira] [Updated] (KAFKA-4465) Create docker image and scripts for running tests locally

2017-01-08 Thread Ewen Cheslack-Postava (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4465:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2197
[https://github.com/apache/kafka/pull/2197]

> Create docker image and scripts for running tests locally
> -
>
> Key: KAFKA-4465
> URL: https://issues.apache.org/jira/browse/KAFKA-4465
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Raghav Kumar Gautam
>Assignee: Raghav Kumar Gautam
> Fix For: 0.10.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #2197: KAFKA-4465: Create docker image and scripts for ru...

2017-01-08 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2197


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-4465) Create docker image and scripts for running tests locally

2017-01-08 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15810267#comment-15810267
 ] 

ASF GitHub Bot commented on KAFKA-4465:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2197


> Create docker image and scripts for running tests locally
> -
>
> Key: KAFKA-4465
> URL: https://issues.apache.org/jira/browse/KAFKA-4465
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Raghav Kumar Gautam
>Assignee: Raghav Kumar Gautam
> Fix For: 0.10.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #2333: KAFKA-3452 Follow-up: Refactoring StateStore hiera...

2017-01-08 Thread guozhangwang
GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/2333

KAFKA-3452 Follow-up: Refactoring StateStore hierarchies

This is a refactoring follow-up of 
https://github.com/apache/kafka/pull/2166.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka 
K3452-followup-state-store-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2333.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2333


commit 575226cbbf84ae52a1962e338ae5ab0108dce2eb
Author: Guozhang Wang 
Date:   2017-01-09T00:48:26Z

a first pass of the refactoring




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-3452) Support session windows

2017-01-08 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15810319#comment-15810319
 ] 

ASF GitHub Bot commented on KAFKA-3452:
---

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/2333

KAFKA-3452 Follow-up: Refactoring StateStore hierarchies

This is a refactoring follow-up of 
https://github.com/apache/kafka/pull/2166.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka 
K3452-followup-state-store-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2333.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2333


commit 575226cbbf84ae52a1962e338ae5ab0108dce2eb
Author: Guozhang Wang 
Date:   2017-01-09T00:48:26Z

a first pass of the refactoring




> Support session windows
> ---
>
> Key: KAFKA-3452
> URL: https://issues.apache.org/jira/browse/KAFKA-3452
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Damian Guy
>  Labels: api, kip
> Fix For: 0.10.2.0
>
>
> The Streams DSL currently does not provide session window as in the DataFlow 
> model. We have seen some common use cases for this feature and it's better 
> adding this support asap.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-94+Session+Windows



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Jenkins build is back to normal : kafka-trunk-jdk7 #1810

2017-01-08 Thread Apache Jenkins Server
See 



Build failed in Jenkins: kafka-trunk-jdk8 #1156

2017-01-08 Thread Apache Jenkins Server
See 

Changes:

[me] KAFKA-4465: Create docker image and scripts for running tests locally

--
[...truncated 18062 lines...]

org.apache.kafka.streams.KafkaStreamsTest > testCleanup STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCleanup PASSED

org.apache.kafka.streams.KafkaStreamsTest > testStartAndClose STARTED

org.apache.kafka.streams.KafkaStreamsTest > testStartAndClose PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning 
STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice PASSED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[0] STARTED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[0] PASSED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[1] STARTED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[0] FAILED
java.lang.AssertionError: Condition not met within timeout 6. Did not 
receive 1 number of records
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:259)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinValuesRecordsReceived(IntegrationTestUtils.java:253)
at 
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.waitUntilAtLeastNumRecordProcessed(QueryableStateIntegrationTest.java:669)
at 
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.queryOnRebalance(QueryableStateIntegrationTest.java:350)

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[1] PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns PASSED

org.apache.kafka.streams.integration.KStreamRepartiti

[jira] [Commented] (KAFKA-4558) throttling_test fails if the producer starts too fast.

2017-01-08 Thread Ewen Cheslack-Postava (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15810593#comment-15810593
 ] 

Ewen Cheslack-Postava commented on KAFKA-4558:
--

[~apurva] I couldn't remember when lag metrics are actually updated, but I 
guess if they are 0 even if the topic doesn't exist then it doesn't help. 
You're right that at least we'll be a lot closer to the correct condition by 
checking that we at least have an assignment.

Using JMX seems fine to me, I didn't look carefully through the JmxMixin 
implementation but thought I remembered it being pretty messy (I think 
partially since it has to run an extra process that does the data collection). 
If it's easy to get from that, that's great. I was also selfishly considering 
other future uses of metrics where we'd probably like there to be easier (and 
cheaper) ways to capture this info :)

Re: HttpMetricsReporter -- nothing in tests requires a KIP since it'd be 
unsupported, although I have no doubt once it existed some folks would want it 
to be supported.

> throttling_test fails if the producer starts too fast.
> --
>
> Key: KAFKA-4558
> URL: https://issues.apache.org/jira/browse/KAFKA-4558
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>
> As described in https://issues.apache.org/jira/browse/KAFKA-4526, the 
> throttling test will fail if the producer in the produce-consume-validate 
> loop starts up before the consumer is fully initialized.
> We need to block the start of the producer until the consumer is ready to go. 
> The current plan is to poll the consumer for a particular metric (like, for 
> instance, partition assignment) which will act as a good proxy for successful 
> initialization. Currently, we just check for the existence of a process with 
> the PID, which is not a strong enough check, causing the test to fail 
> intermittently. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (KAFKA-4402) Kafka Producer's DefaultPartitioner is actually not round robin as said in the code comments "If no partition or key is present choose a partition in a round-robin fashi

2017-01-08 Thread Ewen Cheslack-Postava (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-4402.
--
   Resolution: Fixed
Fix Version/s: 0.10.2.0

Issue resolved by pull request 2128
[https://github.com/apache/kafka/pull/2128]

> Kafka Producer's DefaultPartitioner is actually not round robin as said in 
> the code comments "If no partition or key is present choose a partition in a 
> round-robin fashion"
> 
>
> Key: KAFKA-4402
> URL: https://issues.apache.org/jira/browse/KAFKA-4402
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Yao
>Priority: Minor
> Fix For: 0.10.2.0
>
>
> From this code comments, it is said that Kafka client  Producer's 
> DefaultPartitioner will do round robin if "no partition or key is present", 
> https://github.com/apache/kafka/blob/41e676d29587042994a72baa5000a8861a075c8c/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L34
> from the code it looks trying to do round robin as well, as it maintained a 
> counter and try to increase it every time and then will decide which 
> partition to go to; 
> However the issue here is the counter is a global counter that is shared by 
> all the topics, so  it is actually not round robin per topic and sometimes 
> caused unbalanced routing among different partitions. 
> Although we can pass a custom implementation of interface 
> "org.apache.kafka.clients.producer.Partitioner", it might be still good to 
> make the default implementation true round robin as comment. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request #2128: KAFKA-4402: make the KafkaProducer true round robi...

2017-01-08 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2128


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-4402) Kafka Producer's DefaultPartitioner is actually not round robin as said in the code comments "If no partition or key is present choose a partition in a round-robin fash

2017-01-08 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15810616#comment-15810616
 ] 

ASF GitHub Bot commented on KAFKA-4402:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2128


> Kafka Producer's DefaultPartitioner is actually not round robin as said in 
> the code comments "If no partition or key is present choose a partition in a 
> round-robin fashion"
> 
>
> Key: KAFKA-4402
> URL: https://issues.apache.org/jira/browse/KAFKA-4402
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Yao
>Priority: Minor
> Fix For: 0.10.2.0
>
>
> From this code comments, it is said that Kafka client  Producer's 
> DefaultPartitioner will do round robin if "no partition or key is present", 
> https://github.com/apache/kafka/blob/41e676d29587042994a72baa5000a8861a075c8c/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L34
> from the code it looks trying to do round robin as well, as it maintained a 
> counter and try to increase it every time and then will decide which 
> partition to go to; 
> However the issue here is the counter is a global counter that is shared by 
> all the topics, so  it is actually not round robin per topic and sometimes 
> caused unbalanced routing among different partitions. 
> Although we can pass a custom implementation of interface 
> "org.apache.kafka.clients.producer.Partitioner", it might be still good to 
> make the default implementation true round robin as comment. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2017-01-08 Thread Jun Rao
Hi, Jason,

100. Yes, AppId level security is mainly for protecting the shared
transaction log. We could also include AppId in produce request (not in
message format) so that we could protect writes at the AppId level. I agree
that we need to support prefix matching on AppId for applications like
stream to use this conveniently.

A couple of other comments.

122. Earlier, Becket asked for the use case of knowing the number of
messages in a message set. One potential use case is KAFKA-4293. Currently,
since we don't know the number of messages in a compressed set, to finish
the iteration, we rely on catching EOF in the decompressor, which adds a
bit overhead in the consumer.

123. I am wondering if the coordinator needs to add a "BEGIN transaction
message" on a BeginTxnRequest
.
Could we just wait until an AddPartitionsToTxnRequest

?

Thanks,

Jun


On Thu, Jan 5, 2017 at 11:05 AM, Jason Gustafson  wrote:

> Hi Jun,
>
> Let me start picking off a some of your questions (we're giving the shadow
> log suggestion a bit more thought before responding).
>
> 100. Security: It seems that if an app is mistakenly configured with the
> > appId of an existing producer, it can take over the pid and prevent the
> > existing app from publishing. So, I am wondering if it makes sense to add
> > ACLs at the TransactionResource level just like we do for
> > ConsumerGroupResource. So, a user can only do transactions under a
> > particular appId if he/she has the write permission to the
> > TransactionResource
> > associated with the appId.
>
>
> I think this makes sense in general. There are a couple points worth
> mentioning:
>
> 1. Because we only use the AppID in requests to the transaction
> coordinator, that's the only point at which we can do authorization in the
> current proposal. It is possible for a malicious producer to hijack another
> producer's PID and use it to write data. It wouldn't be able to commit or
> abort transactions, but it could effectively fence the legitimate producer
> from a partition by forcing an epoch bump. We could add the AppID to the
> ProduceRequest schema, but we would still need to protect its binding to
> the PID somehow. This is one argument in favor of dropping the PID and
> using the AppID in the log message format. However, there are still ways in
> the current proposal to give better protection if we added the AppID
> authorization at the transaction coordinator as you suggest. Note that a
> malicious producer would have to be authorized to write to the same topics
> used by the transactional producer. So one way to protect those topics is
> to only allow write access by the authorized transactional producers. The
> transactional producers could still interfere with each other, but perhaps
> that's a smaller concern (it's similar in effect to the limitations of
> consumer group authorization).
>
> 2. It's a bit unfortunate that we don't have something like the consumer's
> groupId to use for authorization. The AppID is really more of an instance
> ID (we were reluctant to introduce any formal notion of a producer group).
> I guess distributed applications could use a common prefix and a wildcard
> authorization policy. I don't think we currently support general wildcards,
> but that might be helpful for this use case.
>
> -Jason
>
> On Wed, Jan 4, 2017 at 12:55 PM, Jay Kreps  wrote:
>
> > Hey Jun,
> >
> > We had a proposal like this previously. The suppression scheme was
> slightly
> > different. Rather than than attempting to recopy or swap, there was
> instead
> > an aborted offset index maintained along with each segment containing a
> > sequential list of aborted offsets. The filtering would happen at fetch
> > time and would just ensure that fetch requests never span an aborted
> > transaction. That is, if you did a fetch request which would include
> > offsets 7,8,9,10,11, but offsets 7 and 10 appears in the index of aborted
> > transactions, then the fetch would return 8,9 only even if there was more
> > space in the fetch response. This leads to minimal overhead, but
> > potentially would give back smaller fetch responses if transactions are
> > being continually aborted.
> >
> > One downside to this approach (both your proposal and the variation that
> I
> > just described is that it does not allow the possibility of consuming in
> > transaction commit order. Consuming in transaction commit order means
> that
> > the only delay you incur is the delay in committing a given transaction.
> > Consuming in offset order means you cannot consume a given offset until
> ALL
> > previously begun transactions are committed or aborted. KIP-98 doesn't
> > propose making this change now, but since it is consumer side it is
> > possible.
> >
> > -Jay
> >
> > On Tue, Jan 3, 

[jira] [Created] (KAFKA-4608) RocksDBWindowStore.fetch() is inefficient for large ranges

2017-01-08 Thread Elias Levy (JIRA)
Elias Levy created KAFKA-4608:
-

 Summary: RocksDBWindowStore.fetch() is inefficient for large ranges
 Key: KAFKA-4608
 URL: https://issues.apache.org/jira/browse/KAFKA-4608
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Affects Versions: 0.10.1.1
Reporter: Elias Levy


It is not unreasonable for a user to call {{RocksDBWindowStore.fetch}} to scan 
for a key across a large time range.  For instance, someone may call it with a 
{{timeFrom}} of zero or a {{timeTo}} of max long in an attempt to fetch keys 
matching across all time forwards or backwards.  

But if you do so, {{fetch}} will peg the CPU, as it attempts to iterate over 
every single segment id in the range. That is obviously very inefficient.  

{{fetch}} should trim the {{timeFrom}}/{{timeTo}} range based on the available 
time range in the {{segments}} hash map, so that it only iterates over the 
available time range.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Build failed in Jenkins: kafka-trunk-jdk8 #1157

2017-01-08 Thread Apache Jenkins Server
See 

Changes:

[me] KAFKA-4402: make the KafkaProducer true round robin per topic

--
[...truncated 31151 lines...]
kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testR

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression

2017-01-08 Thread Dongjin Lee
It seems like no one needs any update on this KIP. If then, May I start the
vote?

Regards,
Dongjin

On Fri, Jan 6, 2017 at 11:40 AM, Dongjin Lee  wrote:

> Hi all,
>
> I've just posted a new KIP "KIP-110: Add Codec for ZStandard Compression"
> for
> discussion:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 110%3A+Add+Codec+for+ZStandard+Compression
>
> Please have a look when you are free.
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
>
> *Software developer in Line+.So interested in massive-scale machine
> learning.facebook: www.facebook.com/dongjin.lee.kr
> linkedin: 
> kr.linkedin.com/in/dongjinleekr
> github:
> github.com/dongjinleekr
> twitter: www.twitter.com/dongjinleekr
> *
>



-- 
*Dongjin Lee*


*Software developer in Line+.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
linkedin:
kr.linkedin.com/in/dongjinleekr
github:
github.com/dongjinleekr
twitter: www.twitter.com/dongjinleekr
*


[jira] [Assigned] (KAFKA-3745) Consider adding join key to ValueJoiner interface

2017-01-08 Thread Sreepathi Prasanna (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreepathi Prasanna reassigned KAFKA-3745:
-

Assignee: Sreepathi Prasanna

> Consider adding join key to ValueJoiner interface
> -
>
> Key: KAFKA-3745
> URL: https://issues.apache.org/jira/browse/KAFKA-3745
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Assignee: Sreepathi Prasanna
>Priority: Minor
>  Labels: api, newbie
>
> In working with Kafka Stream joining, it's sometimes the case that a join key 
> is not actually present in the values of the joins themselves (if, for 
> example, a previous transform generated an ephemeral join key.) In such 
> cases, the actual key of the join is not available in the ValueJoiner 
> implementation to be used to construct the final joined value. This can be 
> worked around by explicitly threading the join key into the value if needed, 
> but it seems like extending the interface to pass the join key along as well 
> would be helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)