Please add me to the contributor list

2015-05-17 Thread Manasvi Gupta
Hello,

Please add me to contributor list & on confluence so I can self-assign &
work on "newbie" jira tickets.

username (Jira & Confluence) - manasvigupta

Thanks,
Manasvi Gupta


Re: Please add me to the contributor list

2015-05-17 Thread Jun Rao
Added to both.

Thanks,

Jun

On Sun, May 17, 2015 at 12:56 PM, Manasvi Gupta  wrote:

> Hello,
>
> Please add me to contributor list & on confluence so I can self-assign &
> work on "newbie" jira tickets.
>
> username (Jira & Confluence) - manasvigupta
>
> Thanks,
> Manasvi Gupta
>


Re: Review Request 33049: Patch for KAFKA-2084

2015-05-17 Thread Dong Lin


> On May 12, 2015, 7:38 p.m., Dong Lin wrote:
> > clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java, lines 
> > 122-127
> > 
> >
> > The function quits on the first quota violaation, and calculate 
> > delayTime based on this violation. 
> > 
> > Should we enumerate all metric in this.metrics() and set delayTime of 
> > the QuotaViolationException to be the largest delayTime of all violatoins?
> 
> Aditya Auradkar wrote:
> Also a good point. Perhaps you can fix it in the patch you will submit? 
> For quotas, we effectively have only one quota per sensor, so it's ok (for 
> now).

Sure! Will do it.


> On May 12, 2015, 7:38 p.m., Dong Lin wrote:
> > clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java, line 117
> > 
> >
> > metric.value(timeMs), which translates to Rate.measure(config, timeMs), 
> > may return Infinity. This may introduce infinite delay.
> > 
> > I think this bug is rooted either in class Rate or  class SampledStat. 
> > In short, SampledStat.purgeObsoleteSamples will remove all Samples, such 
> > that SampledStat.oldest(now) == now.
> > 
> > Should I open a ticket and submit a patch for it?
> 
> Aditya Auradkar wrote:
> Hey dong, yeah you should submit a patch for it.

Sure! I will do it.


- Dong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/#review83441
---


On May 11, 2015, 11:17 p.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33049/
> ---
> 
> (Updated May 11, 2015, 11:17 p.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2084
> https://issues.apache.org/jira/browse/KAFKA-2084
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Updated patch for quotas. This patch does the following: 
> 1. Add per-client metrics for both producer and consumers 
> 2. Add configuration for quotas 
> 3. Compute delay times in the metrics package and return the delay times in 
> QuotaViolationException 
> 4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
> request. Implemented request throttling for produce and fetch requests. 
> 5. Added unit and integration test cases.
> 6. This doesn't include a system test. There is a separate ticket for that
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.java 
> dfa1b0a11042ad9d127226f0e0cec8b1d42b8441 
>   clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
> d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
>   
> clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
>  a451e5385c9eca76b38b425e8ac856b2715fcffe 
>   clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
> ca823fd4639523018311b814fde69b6177e73b97 
>   clients/src/test/java/org/apache/kafka/common/utils/MockTime.java  
>   core/src/main/scala/kafka/server/ClientQuotaMetrics.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 417960dd1ab407ebebad8fdb0e97415db3e91a2f 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> b7d2a2842e17411a823b93bdedc84657cbd62be1 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
>   core/src/main/scala/kafka/server/ThrottledRequest.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
> fc226c863095b7761290292cd8755cd7ad0f155c 
>   core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/ClientQuotaMetricsTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
> 8014a5a6c362785539f24eb03d77278434614fe6 
>   core/src/test/scala/unit/kafka/server/ThrottledRequestExpirationTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33049/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>



Re: Review Request 34170: Patch for KAFKA-2191

2015-05-17 Thread Dong Lin


> On May 13, 2015, 11:50 p.m., Aditya Auradkar wrote:
> > clients/src/main/java/org/apache/kafka/common/metrics/stats/SampledStat.java,
> >  line 45
> > 
> >
> > I think this is a good catch.
> > 
> > Just so I understand, this is adding new samples for blank intervals of 
> > time when record was not called. As an optimization, I think you should 
> > only add "MetricConfig.samples" number of samples. In the rare case, there 
> > is no activity for 10 mins (say), this will add 10sec*6*10 = 600 samples 
> > which will be purged immediately on the next record call.
> 
> Jay Kreps wrote:
> This is a really good point. It is totally possible for a metric to track 
> activity on a topic that has no writes for a month, the first write would 
> then cause you to cycle through a month of samples. The logic around 
> correctly skipping these windows and calculating the correct window boundry 
> needs to be carefully worked out.
> 
> Jay Kreps wrote:
> Actually maybe you can elaborate on why this is needed at all? In the 
> current code if the current sample is complete we add a new sample. Your code 
> adds lots of samples. But why do we need that? Isn't purging obsolete samples 
> handled on measurement already? I think that is more elegant. You must see 
> some issue there, maybe you can explain?
> 
> Dong Lin wrote:
> Adi: Good catch! I will update the patch to fix the problem.
> 
> Jay: Sure. Here is the problem I find with the current code.
> 
> - The current code implements SampledStat.advance in a way that is 
> probably different from users' expectation: typically when rate is sampled, 
> time will be divided into consecutive slots of equal length and samples are 
> expired in unit of these time slots, which assures the user the rate is 
> measured on samples collected in the past expireAge period. The current 
> implementation doesn't provide such an easy-to-understand interpretations, 
> since the effective sampled period can be anywhere between 0 and expireAge.
> 
> - As one extreme example, say we call rate.measure after expireAge (i.e. 
> config.samples() * config.timeWindowMs()) has passed since last sample. Then 
> purgeObsoleteSamples will reset all samples and elapsed will be 0 as a 
> result. This is problematic: rate should instead keep the information that we 
> haven’t observed any sample during the past expireAge.
> 
> Does it make sense?
> 
> Jay Kreps wrote:
> Hey Dong, I think what you are saying is that the last sample is partial, 
> that is it doesn't cover a full window yet. This is true and is by design. 
> That is the whole reason there are multiple samples--to stabalize the partial 
> estimate. The ideal window would be a backwards looking sliding window of N 
> ms from the time of measurement but that would be computationally unfeasible.
> 
> I don't understand your example. If no events have occurred in 
> config.samples() * config.timeWindowMs() then the observed rate is 0, right? 
> I think you are saying that the time estimate should be the full window, but 
> that is fixed by changing the other formula as I suggested.
> 
> Dong Lin wrote:
> Hey Jay, thanks for reply. The problem I wanted to explain with this 
> example is that, if no events have occurred in config.samples() * 
> config.timeWindowMs(), oldest == now and the measured rate will be infinite 
> (which should be 0 instead).
> 
> The example explained above is only relevant to the current code. As you 
> said, using the other formula you suggested will fix the problem and avoid 
> the need to make this change here.
> 
> Just a couple of questions towards an acceptable patch. How about we use 
> the other formula you suggested in Rate.measure()? When the user calls 
> rate.measure() with number of samples <= 1 (i.e. ellapsed <= timeWindowMs), 
> should we throw exception or return n/timeWindowMs? And should we keep the 
> existing SampledStat.oldest()?
> 
> Thank much,
> 
> Jay Kreps wrote:
> Ah gotcha, yeah i think it is easiest to fix in measure() rather than in 
> advance.
> 
> You definitely don't want to throw an exception in measure because it is 
> totally correct to have any number of samples when measure is called. I think 
> the proposed change was require the configured num samples (i.e. the max 
> samples) to be at least 2 and then always use the formula we discussed in 
> measure.
> 
> But heres the thing I think this logic is simply very precise and has to 
> be carefully thought out with each change. So don't listen to me too much, 
> work it through and see if you agree that that makes sense and if you don't 
> let's talk about alternatives.

Thanks Jay for all your helpful comments! I will try to come up with the best 
approach. I will submit the patch after I test it with Adi's quota 
implementation (as part of KA

Re: Review Request 30403: Patch for KAFKA-1906

2015-05-17 Thread Jaikiran Pai

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30403/
---

(Updated May 18, 2015, 4:42 a.m.)


Review request for kafka.


Bugs: KAFKA-1906
https://issues.apache.org/jira/browse/KAFKA-1906


Repository: kafka


Description (updated)
---

KAFKA-1906 Default the Kafka data log directory to $KAFKA_HOME/data/kafka-logs 
directory, where KAFKA_HOME is the Kafka installation directory


Diffs (updated)
-

  bin/kafka-run-class.sh 8c3fa286145344fb527307012c1d1000d855aa18 
  bin/windows/kafka-run-class.bat 4aa2ab8ddb9bf56c26fc1c292f359e50b40461e5 
  config/server.properties 80ee2fc6e94a114e7710ae4df3f4e2b83e06f080 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
2428dbd7197a58cf4cad42ef82b385dab3a2b15e 

Diff: https://reviews.apache.org/r/30403/diff/


Testing
---

The change here involves updating the Kafka scripts (for Windows and * nix) to 
infer and setup KAFKA_HOME environment variable. This value is then used by the 
KafkaConfig to decide what path to default to for the Kafka data logs, in the 
absence of any explicitly set log.dirs (or log.dir) properties.

Care has been taken to ensure that other mechanism which might presently be 
bypassing the Kafka scripts, will still continue to function, since in the 
absence of KAFKA_HOME environment property value, we fall back to 
/tmp/kafka-logs (the present default) as the default data log directory

Existing tests have been run to ensure that this change maintains backward 
compatibility (i.e. doesn't fail when KAFKA_HOME isn't available/set) and 2 new 
test methods have been added to the KafkaConfigTest to ensure that this change 
works.

Although the change has been made to both .sh and .bat files, to support this, 
I haven't actually tested this change on a Windows OS and would appreciate if 
someone can test this there and let me know if they run into any issues.


Thanks,

Jaikiran Pai



[jira] [Updated] (KAFKA-1906) Default the Kafka log directory to a relative path within the Kafka installation directory

2015-05-17 Thread jaikiran pai (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jaikiran pai updated KAFKA-1906:

Attachment: KAFKA-1906_2015-05-18_10:11:12.patch

> Default the Kafka log directory to a relative path within the Kafka 
> installation directory
> --
>
> Key: KAFKA-1906
> URL: https://issues.apache.org/jira/browse/KAFKA-1906
> Project: Kafka
>  Issue Type: Improvement
>  Components: core, log
>Reporter: jaikiran pai
>Assignee: jaikiran pai
> Attachments: KAFKA-1906.patch, KAFKA-1906_2015-05-18_10:11:12.patch
>
>
> During one of the discussions on kafka-dev mailing list, a point was raised 
> that the current /tmp/kafka-logs default path for storing the Kafka data 
> wasn't probably a good idea and instead we should default it to a relative 
> path within the Kafka installation directory:
> {quote}
> Having a relative path and keeping data under /data in the kafka distro
> would make sense. This would require some reworking of the shell scripts,
> though, as I think right now you an actually run Kafka from any directory
> and the cwd of the process will be whatever directory you start from. If we
> have a relative path in the config then the working directory will HAVE to
> be the kafka directory. This works for the simple download case but may
> making some packaging stuff harder for other use cases.
> {quote}
> The discussion is here 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201501.mbox/%3ccaoejijga38ktwh4jyfsu4w8idxybf-_ze_b748txsorp8yz...@mail.gmail.com%3E
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-1906) Default the Kafka log directory to a relative path within the Kafka installation directory

2015-05-17 Thread jaikiran pai (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jaikiran pai updated KAFKA-1906:

Status: Patch Available  (was: In Progress)

> Default the Kafka log directory to a relative path within the Kafka 
> installation directory
> --
>
> Key: KAFKA-1906
> URL: https://issues.apache.org/jira/browse/KAFKA-1906
> Project: Kafka
>  Issue Type: Improvement
>  Components: core, log
>Reporter: jaikiran pai
>Assignee: jaikiran pai
> Attachments: KAFKA-1906.patch, KAFKA-1906_2015-05-18_10:11:12.patch
>
>
> During one of the discussions on kafka-dev mailing list, a point was raised 
> that the current /tmp/kafka-logs default path for storing the Kafka data 
> wasn't probably a good idea and instead we should default it to a relative 
> path within the Kafka installation directory:
> {quote}
> Having a relative path and keeping data under /data in the kafka distro
> would make sense. This would require some reworking of the shell scripts,
> though, as I think right now you an actually run Kafka from any directory
> and the cwd of the process will be whatever directory you start from. If we
> have a relative path in the config then the working directory will HAVE to
> be the kafka directory. This works for the simple download case but may
> making some packaging stuff harder for other use cases.
> {quote}
> The discussion is here 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201501.mbox/%3ccaoejijga38ktwh4jyfsu4w8idxybf-_ze_b748txsorp8yz...@mail.gmail.com%3E
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-1906) Default the Kafka log directory to a relative path within the Kafka installation directory

2015-05-17 Thread jaikiran pai (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547526#comment-14547526
 ] 

jaikiran pai commented on KAFKA-1906:
-

Updated reviewboard https://reviews.apache.org/r/30403/diff/
 against branch origin/trunk

> Default the Kafka log directory to a relative path within the Kafka 
> installation directory
> --
>
> Key: KAFKA-1906
> URL: https://issues.apache.org/jira/browse/KAFKA-1906
> Project: Kafka
>  Issue Type: Improvement
>  Components: core, log
>Reporter: jaikiran pai
>Assignee: jaikiran pai
> Attachments: KAFKA-1906.patch, KAFKA-1906_2015-05-18_10:11:12.patch
>
>
> During one of the discussions on kafka-dev mailing list, a point was raised 
> that the current /tmp/kafka-logs default path for storing the Kafka data 
> wasn't probably a good idea and instead we should default it to a relative 
> path within the Kafka installation directory:
> {quote}
> Having a relative path and keeping data under /data in the kafka distro
> would make sense. This would require some reworking of the shell scripts,
> though, as I think right now you an actually run Kafka from any directory
> and the cwd of the process will be whatever directory you start from. If we
> have a relative path in the config then the working directory will HAVE to
> be the kafka directory. This works for the simple download case but may
> making some packaging stuff harder for other use cases.
> {quote}
> The discussion is here 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201501.mbox/%3ccaoejijga38ktwh4jyfsu4w8idxybf-_ze_b748txsorp8yz...@mail.gmail.com%3E
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 30403: Patch for KAFKA-1906

2015-05-17 Thread Jaikiran Pai


> On Feb. 7, 2015, 4:41 p.m., Neha Narkhede wrote:
> > config/server.properties, line 58
> > 
> >
> > Why leave the value here to point at /tmp/kafka-logs? The way I see is 
> > that we default the log directory to data/ in the kafka installation 
> > directory and possibly encourage storing kafka data under var/ if it must 
> > be overridden for production, by changing this commented out value to 
> > /var/kafka-logs?

This has now been addressed to leave out the reference to the /tmp folder.


- Jaikiran


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30403/#review71557
---


On May 18, 2015, 4:42 a.m., Jaikiran Pai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30403/
> ---
> 
> (Updated May 18, 2015, 4:42 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1906
> https://issues.apache.org/jira/browse/KAFKA-1906
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1906 Default the Kafka data log directory to 
> $KAFKA_HOME/data/kafka-logs directory, where KAFKA_HOME is the Kafka 
> installation directory
> 
> 
> Diffs
> -
> 
>   bin/kafka-run-class.sh 8c3fa286145344fb527307012c1d1000d855aa18 
>   bin/windows/kafka-run-class.bat 4aa2ab8ddb9bf56c26fc1c292f359e50b40461e5 
>   config/server.properties 80ee2fc6e94a114e7710ae4df3f4e2b83e06f080 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
>   core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
> 2428dbd7197a58cf4cad42ef82b385dab3a2b15e 
> 
> Diff: https://reviews.apache.org/r/30403/diff/
> 
> 
> Testing
> ---
> 
> The change here involves updating the Kafka scripts (for Windows and * nix) 
> to infer and setup KAFKA_HOME environment variable. This value is then used 
> by the KafkaConfig to decide what path to default to for the Kafka data logs, 
> in the absence of any explicitly set log.dirs (or log.dir) properties.
> 
> Care has been taken to ensure that other mechanism which might presently be 
> bypassing the Kafka scripts, will still continue to function, since in the 
> absence of KAFKA_HOME environment property value, we fall back to 
> /tmp/kafka-logs (the present default) as the default data log directory
> 
> Existing tests have been run to ensure that this change maintains backward 
> compatibility (i.e. doesn't fail when KAFKA_HOME isn't available/set) and 2 
> new test methods have been added to the KafkaConfigTest to ensure that this 
> change works.
> 
> Although the change has been made to both .sh and .bat files, to support 
> this, I haven't actually tested this change on a Windows OS and would 
> appreciate if someone can test this there and let me know if they run into 
> any issues.
> 
> 
> Thanks,
> 
> Jaikiran Pai
> 
>



Re: Review Request 30403: Patch for KAFKA-1906

2015-05-17 Thread Jaikiran Pai

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30403/#review84107
---


It took me a while to get back to contributing. I have now updated this patch 
to incorporate Neha's review comments. This is now ready to be reviewed again.

- Jaikiran Pai


On May 18, 2015, 4:42 a.m., Jaikiran Pai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30403/
> ---
> 
> (Updated May 18, 2015, 4:42 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1906
> https://issues.apache.org/jira/browse/KAFKA-1906
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1906 Default the Kafka data log directory to 
> $KAFKA_HOME/data/kafka-logs directory, where KAFKA_HOME is the Kafka 
> installation directory
> 
> 
> Diffs
> -
> 
>   bin/kafka-run-class.sh 8c3fa286145344fb527307012c1d1000d855aa18 
>   bin/windows/kafka-run-class.bat 4aa2ab8ddb9bf56c26fc1c292f359e50b40461e5 
>   config/server.properties 80ee2fc6e94a114e7710ae4df3f4e2b83e06f080 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 9efa15ca5567b295ab412ee9eea7c03eb4cdc18b 
>   core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
> 2428dbd7197a58cf4cad42ef82b385dab3a2b15e 
> 
> Diff: https://reviews.apache.org/r/30403/diff/
> 
> 
> Testing
> ---
> 
> The change here involves updating the Kafka scripts (for Windows and * nix) 
> to infer and setup KAFKA_HOME environment variable. This value is then used 
> by the KafkaConfig to decide what path to default to for the Kafka data logs, 
> in the absence of any explicitly set log.dirs (or log.dir) properties.
> 
> Care has been taken to ensure that other mechanism which might presently be 
> bypassing the Kafka scripts, will still continue to function, since in the 
> absence of KAFKA_HOME environment property value, we fall back to 
> /tmp/kafka-logs (the present default) as the default data log directory
> 
> Existing tests have been run to ensure that this change maintains backward 
> compatibility (i.e. doesn't fail when KAFKA_HOME isn't available/set) and 2 
> new test methods have been added to the KafkaConfigTest to ensure that this 
> change works.
> 
> Although the change has been made to both .sh and .bat files, to support 
> this, I haven't actually tested this change on a Windows OS and would 
> appreciate if someone can test this there and let me know if they run into 
> any issues.
> 
> 
> Thanks,
> 
> Jaikiran Pai
> 
>



Next is 0.8.3 or is it 0.9?

2015-05-17 Thread Jaikiran Pai

Hello everyone,

What is the next planned version of Kafka going to be? I vaguely 
remember reading in some mail that it will be 0.9 and "trunk" is where 
it would be released from. I was just updating my local workspace for 
contributing and noticed that the gradle.properties in trunk notes the 
next version as 0.8.3-SNAPSHOT.


-Jaikiran


[jira] [Updated] (KAFKA-1888) Add a "rolling upgrade" system test

2015-05-17 Thread Abhishek Nigam (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Nigam updated KAFKA-1888:
--
Assignee: (was: Abhishek Nigam)

> Add a "rolling upgrade" system test
> ---
>
> Key: KAFKA-1888
> URL: https://issues.apache.org/jira/browse/KAFKA-1888
> Project: Kafka
>  Issue Type: Improvement
>  Components: system tests
>Reporter: Gwen Shapira
> Fix For: 0.9.0
>
> Attachments: KAFKA-1888_2015-03-23_11:54:25.patch
>
>
> To help test upgrades and compatibility between versions, it will be cool to 
> add a rolling-upgrade test to system tests:
> Given two versions (just a path to the jars?), check that you can do a
> rolling upgrade of the brokers from one version to another (using clients 
> from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-1888) Add a "rolling upgrade" system test

2015-05-17 Thread Abhishek Nigam (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547566#comment-14547566
 ] 

Abhishek Nigam commented on KAFKA-1888:
---

Geoffrey,
Thanks for the heads up. I saw a related article that you are planning to work 
on the API compatibility testing as well.
I am taking myself off of this ticket as it looks like this ticket will be 
subsumed by your work.

-Abhishek

> Add a "rolling upgrade" system test
> ---
>
> Key: KAFKA-1888
> URL: https://issues.apache.org/jira/browse/KAFKA-1888
> Project: Kafka
>  Issue Type: Improvement
>  Components: system tests
>Reporter: Gwen Shapira
>Assignee: Abhishek Nigam
> Fix For: 0.9.0
>
> Attachments: KAFKA-1888_2015-03-23_11:54:25.patch
>
>
> To help test upgrades and compatibility between versions, it will be cool to 
> add a rolling-upgrade test to system tests:
> Given two versions (just a path to the jars?), check that you can do a
> rolling upgrade of the brokers from one version to another (using clients 
> from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Can someone review ticket 1778

2015-05-17 Thread Abhishek Nigam
Hi,
For pinning the controller to a broker I have proposed a design. Can someone 
review the design and let me know if it looks ok.
I can then submit a patch for this ticket within the next couple of weeks.

-Abhishek



[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

2015-05-17 Thread Vamsi Subhash Achanta (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547606#comment-14547606
 ] 

Vamsi Subhash Achanta commented on KAFKA-1977:
--

can this be merged in the old consumer code in 0.8.2.x branch? We also have the 
same kind of requirement.

> Make logEndOffset available in the Zookeeper consumer
> -
>
> Key: KAFKA-1977
> URL: https://issues.apache.org/jira/browse/KAFKA-1977
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Reporter: Will Funnell
>Priority: Minor
> Attachments: 
> Make_logEndOffset_available_in_the_Zookeeper_consumer.patch
>
>
> The requirement is to create a snapshot from the Kafka topic but NOT do 
> continual reads after that point. For example you might be creating a backup 
> of the data to a file.
> In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps 
> was to expose the high watermark, as maxEndOffset, from the FetchResponse 
> object through to each MessageAndMetadata object in order to be aware when 
> the consumer has reached the end of each partition.
> The submitted patch achieves this by adding the maxEndOffset to the 
> PartitionTopicInfo, which is updated when a new message arrives in the 
> ConsumerFetcherThread and then exposed in MessageAndMetadata.
> See here for discussion:
> http://search-hadoop.com/m/4TaT4TpJy71



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)