[jira] [Commented] (KAFKA-1510) Force offset commits when migrating consumer offsets from zookeeper to kafka

[nicu marasoiu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118294#comment-14118294
 ] 

nicu marasoiu commented on KAFKA-1510:
--

[~jjkoshy] Hi, can you check my patch + comments & provide feedback pls?

> Force offset commits when migrating consumer offsets from zookeeper to kafka
> 
>
> Key: KAFKA-1510
> URL: https://issues.apache.org/jira/browse/KAFKA-1510
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Joel Koshy
>Assignee: Joel Koshy
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: 
> Patch_to_push_unfiltered_offsets_to_both_Kafka_and_potentially_Zookeeper_when_Kafka_is_con.patch,
>  Unfiltered_to_kafka,_Incremental_to_Zookeeper.patch
>
>
> When migrating consumer offsets from ZooKeeper to kafka, we have to turn on 
> dual-commit (i.e., the consumers will commit offsets to both zookeeper and 
> kafka) in addition to setting offsets.storage to kafka. However, when we 
> commit offsets we only commit offsets if they have changed (since the last 
> commit). For low-volume topics or for topics that receive data in bursts 
> offsets may not move for a long period of time. Therefore we may want to 
> force the commit (even if offsets have not changed) when migrating (i.e., 
> when dual-commit is enabled) - we can add a minimum interval threshold (say 
> force commit after every 10 auto-commits) as well as on rebalance and 
> shutdown.
> Also, I think it is safe to switch the default for offsets.storage from 
> zookeeper to kafka and set the default to dual-commit (for people who have 
> not migrated yet). We have deployed this to the largest consumers at linkedin 
> and have not seen any issues so far (except for the migration caveat that 
> this jira will resolve).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1282) Disconnect idle socket connection in Selector

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118317#comment-14118317
 ] 

Jun Rao commented on KAFKA-1282:


Thanks for the latest patch. I was trying to do some local testing. The 
following are my observations.

1. I first started a local ZK and broker (setting connections.max.idle.ms 
10secs). I then started a console-producer and a console-consumer. Then, I 
typed in sth in console-producer every 15 secs. However, I don't see the 
producer connection gets killed. I added sth instrumentation. It doesn't seem 
that removeEldestEntry() is called on every fetch request.

2. As I was debugging this, I realized that it's kind of weird to kill idle 
connections only when there is another non-idle connection. This makes 
debugging harder since one can't just test this out with a single connection. 
It's much simpler to understand if the idle connection can just be killed after 
the connection idle time, independent of other connections to the broker. To 
address the concern of closing many sockets in one iteration of the selector, 
we can calculate the time that a socket entry is expected to be killed (this is 
the access time of the oldest entry + maxIdleTime, or maxIdleTime if no entry 
exists). When that time comes during the iteration of the selector, we can just 
check the oldest entry and see if it needs to be closed.

3. It would be good to check if our clients (especially the producer, both old 
and new) can handle a closed idle connection properly. For example, when 
detecting an already closed socket, the producer should be able to resend the 
message and therefore we shouldn't see any data loss.

> Disconnect idle socket connection in Selector
> -
>
> Key: KAFKA-1282
> URL: https://issues.apache.org/jira/browse/KAFKA-1282
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Neha Narkhede
>  Labels: newbie++
> Fix For: 0.9.0
>
> Attachments: 
> KAFKA-1282_Disconnect_idle_socket_connection_in_Selector.patch, 
> idleDisconnect.patch
>
>
> To reduce # socket connections, it would be useful for the new producer to 
> close socket connections that are idle. We can introduce a new producer 
> config for the idle time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1507) Using GetOffsetShell against non-existent topic creates the topic unintentionally

[Sriharsha Chintalapani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriharsha Chintalapani updated KAFKA-1507:
--
Reviewer: Jay Kreps

> Using GetOffsetShell against non-existent topic creates the topic 
> unintentionally
> -
>
> Key: KAFKA-1507
> URL: https://issues.apache.org/jira/browse/KAFKA-1507
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
> Environment: centos
>Reporter: Luke Forehand
>Assignee: Sriharsha Chintalapani
>Priority: Minor
>  Labels: newbie
> Attachments: KAFKA-1507.patch, KAFKA-1507.patch, 
> KAFKA-1507_2014-07-22_10:27:45.patch, KAFKA-1507_2014-07-23_17:07:20.patch, 
> KAFKA-1507_2014-08-12_18:09:06.patch, KAFKA-1507_2014-08-22_11:06:38.patch, 
> KAFKA-1507_2014-08-22_11:08:51.patch
>
>
> A typo in using GetOffsetShell command can cause a
> topic to be created which cannot be deleted (because deletion is still in
> progress)
> ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
> kafka10:9092,kafka11:9092,kafka12:9092,kafka13:9092 --topic typo --time 1
> ./kafka-topics.sh --zookeeper stormqa1/kafka-prod --describe --topic typo
> Topic:typo  PartitionCount:8ReplicationFactor:1 Configs:
>  Topic: typo Partition: 0Leader: 10  Replicas: 10
>   Isr: 10
> ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1507) Using GetOffsetShell against non-existent topic creates the topic unintentionally

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118324#comment-14118324
 ] 

Sriharsha Chintalapani commented on KAFKA-1507:
---

2 weeks ping. [~jkreps] [~junrao] Can you please take a look at this patch. Lot 
of files changed so hard to keep it merge conflict free :).

> Using GetOffsetShell against non-existent topic creates the topic 
> unintentionally
> -
>
> Key: KAFKA-1507
> URL: https://issues.apache.org/jira/browse/KAFKA-1507
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
> Environment: centos
>Reporter: Luke Forehand
>Assignee: Sriharsha Chintalapani
>Priority: Minor
>  Labels: newbie
> Attachments: KAFKA-1507.patch, KAFKA-1507.patch, 
> KAFKA-1507_2014-07-22_10:27:45.patch, KAFKA-1507_2014-07-23_17:07:20.patch, 
> KAFKA-1507_2014-08-12_18:09:06.patch, KAFKA-1507_2014-08-22_11:06:38.patch, 
> KAFKA-1507_2014-08-22_11:08:51.patch
>
>
> A typo in using GetOffsetShell command can cause a
> topic to be created which cannot be deleted (because deletion is still in
> progress)
> ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
> kafka10:9092,kafka11:9092,kafka12:9092,kafka13:9092 --topic typo --time 1
> ./kafka-topics.sh --zookeeper stormqa1/kafka-prod --describe --topic typo
> Topic:typo  PartitionCount:8ReplicationFactor:1 Configs:
>  Topic: typo Partition: 0Leader: 10  Replicas: 10
>   Isr: 10
> ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25155: Fix KAFKA-1616

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25155/#review52036
---



core/src/main/scala/kafka/server/RequestPurgatory.scala


It's probably better to expose an api that returns the size of the queue, 
instead of exposing the queue itself.



core/src/main/scala/kafka/server/RequestPurgatory.scala


It probably would be clearer if we use RequestPurgatory.this.size().



core/src/main/scala/kafka/server/RequestPurgatory.scala


I thought that we only want to call purgeSatisfied if delayed.size >= 
purgeIntveral and similarly, only call watcherForKey...purgeSatisfied() if 
RequestPurgatory.this.size() >= purgeInterval?



core/src/test/scala/unit/kafka/server/RequestPurgatoryTest.scala


one of the request => one of the requests



core/src/test/scala/unit/kafka/server/RequestPurgatoryTest.scala


Instead of using sleep, would it be better to wrap the test using 
waitUntilTrue()?


- Jun Rao


On Sept. 1, 2014, 9:41 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25155/
> ---
> 
> (Updated Sept. 1, 2014, 9:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1616
> https://issues.apache.org/jira/browse/KAFKA-1616
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Purgatory size to be the sum of watched list sizes; delayed request to be the 
> expiry queue length; remove atomic integers for metrics; add a unit test for 
> watched list sizes and enqueued requests
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala 
> ce06d2c381348deef8559374869fcaed923da1d1 
>   core/src/test/scala/unit/kafka/server/RequestPurgatoryTest.scala 
> 168712de241125982d556c188c76514fceb93779 
> 
> Diff: https://reviews.apache.org/r/25155/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Offset Request with timestamp

[Manjunath Shivakumar]

Hi,

My usecase is to fetch the offsets for a given topic from X milliseconds ago.
If I use the offset api

https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI

to do this and pass in a timestamp of (now() - X), I get the earliest offset in 
the current log segment and not the offset from X milliseconds ago.

Is this the correct usage or behaviour?

Thanks,
Manju


In order to protect our email recipients, Betfair Group use SkyScan from 
MessageLabs to scan all Incoming and Outgoing mail for viruses.

Re: Review Request 24676: Fix KAFKA-1583

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24676/#review52044
---


Are you including the changes in kafka-1616 too? That would be fine. However, 
the comments in the other jira also need to be addressed in this patch.


core/src/main/scala/kafka/cluster/Partition.scala


may now in => may not be in



core/src/main/scala/kafka/server/RequestPurgatory.scala


Force completing the dealyed operation, if not already compeleted.



core/src/main/scala/kafka/server/RequestPurgatory.scala


and will be called exactly once.



core/src/main/scala/kafka/server/RequestPurgatory.scala


Not sure if we need the while loop any more. The comments may also need to 
be adjusted.


- Jun Rao


On Sept. 2, 2014, 1:09 a.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24676/
> ---
> 
> (Updated Sept. 2, 2014, 1:09 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1583
> https://issues.apache.org/jira/browse/KAFKA-1583
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Incorporated Jun's comments round two.
> 
> Incorporated Jun's comments.
> 
> 1. I left some cases in Log since they are return values for some of their 
> APIs.
> 2. I kept the fetch info in the delayed fetch metadata since it needs to be 
> used for re-reading the log.
> 3. I kept the name of "callbackOnComplete" by following the principle that 
> only the caller knows what the callback is used for, and hence they can name 
> the callback as reponseCallback (from KafkaApi) and putCacheCallback (from 
> OffsetManager), all the callee will take the callback as "callbackOnComplete".
> 
> Unit test passed, with some other notes:
> 
> 1. Found and fix a bug in the current delayed fetch satisifaction logic: 
> previously when we calculate the bytes, we do not take in the fetchMaxBytes 
> into consideration as an upper limit for a single partition's log, but simply 
> get the diff between the current HW/LEO and the fetch offset.
> 2. Found and fix a bug in the unit tests: we used to create replica manager 
> on the fly but did not shut it down upon completing the test, which will leak 
> the background thread (i.e. reaper thread of purgatory).
> 3. Changed the RequestPurgatory API a bit with Jun's comments. Now it has two 
> implemented functions: forceComplete() and isCompleted(), and two functions 
> that need to be instantiated in the subclasses: tryComplete() and complete(). 
> Please let me know if people have more comments on the current API.
> 4. Cleaned the SimpleFetch test, previously this test is too complicate but 
> it actually just test a simple logic of the replica manager.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/api/FetchRequest.scala 
> 51cdccf7f90eb530cc62b094ed822b8469d50b12 
>   core/src/main/scala/kafka/api/FetchResponse.scala 
> af9308737bf7832eca018c2b3ede703f7d1209f1 
>   core/src/main/scala/kafka/api/OffsetCommitRequest.scala 
> 861a6cf11dc6b6431fcbbe9de00c74a122f204bd 
>   core/src/main/scala/kafka/api/ProducerRequest.scala 
> b2366e7eedcac17f657271d5293ff0bef6f3cbe6 
>   core/src/main/scala/kafka/api/ProducerResponse.scala 
> a286272c834b6f40164999ff8b7f8998875f2cfe 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> ff106b47e6ee194cea1cf589474fef975b9dd7e2 
>   core/src/main/scala/kafka/common/ErrorMapping.scala 
> 3fae7910e4ce17bc8325887a046f383e0c151d44 
>   core/src/main/scala/kafka/log/Log.scala 
> 0ddf97bd30311b6039e19abade41d2fbbad2f59b 
>   core/src/main/scala/kafka/network/BoundedByteBufferSend.scala 
> a624359fb2059340bb8dc1619c5b5f226e26eb9b 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> e0f14e25af03e6d4344386dcabc1457ee784d345 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 9481508fc2d6140b36829840c337e557f3d090da 
>   core/src/main/scala/kafka/server/FetchRequestPurgatory.scala 
> ed1318891253556cdf4d908033b704495acd5724 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> c584b559416b3ee4bcbec5966be4891e0a03eefb 
>   core/src/main/scala/kafka/server/OffsetManager.scala 
> 43eb2a35bb54d32c66cdb94772df657b3a104d1a 
>   core/src/main/scala/kafka/server/ProducerRequestPurgatory.scala 
> d4a7d4a79b44263a1f7e1a92874dd36aa06e5a3f 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 68758e35d496a4659819960ae8e809d6e215568e 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala 
> ce06d2c381348deef8559374869fcaed923da1d1 
>   core/src/main/scala/kafka/utils/DelayedItem.scala 
> d72764940

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system tests can run

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118449#comment-14118449
 ] 

Guozhang Wang commented on KAFKA-1485:
--

Hello Joe, it seems the file kafka.tools.ZookeeperMainWrapper is not checked 
in, causing system test to fail again. Could you help double check?

> Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system tests can run
> --
>
> Key: KAFKA-1485
> URL: https://issues.apache.org/jira/browse/KAFKA-1485
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1.1
>Reporter: Machiel Groeneveld
>Assignee: Gwen Shapira
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1485.2.patch, KAFKA-1485.3.patch, 
> KAFKA-1485.4.patch, KAFKA-1485.patch
>
>
> I can't run projects alongside Kafka that use zookeeper 3.4 jars. 3.4 has 
> been out for 2.5 years and seems to be ready for adoption.
> In particular Apache Storm will upgrade to Zookeeper 3.4.x in their next 
> 0.9.2 release. I can't run both versions in my tests at the same time. 
> The only compile problem I saw was in EmbeddedZookeeper.scala 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system tests can run

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118459#comment-14118459
 ] 

Joe Stein commented on KAFKA-1485:
--

yup, sorry about that, git pull and try again please

> Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system tests can run
> --
>
> Key: KAFKA-1485
> URL: https://issues.apache.org/jira/browse/KAFKA-1485
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1.1
>Reporter: Machiel Groeneveld
>Assignee: Gwen Shapira
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1485.2.patch, KAFKA-1485.3.patch, 
> KAFKA-1485.4.patch, KAFKA-1485.patch
>
>
> I can't run projects alongside Kafka that use zookeeper 3.4 jars. 3.4 has 
> been out for 2.5 years and seems to be ready for adoption.
> In particular Apache Storm will upgrade to Zookeeper 3.4.x in their next 
> 0.9.2 release. I can't run both versions in my tests at the same time. 
> The only compile problem I saw was in EmbeddedZookeeper.scala 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: How to debug unit test in Eclipse

[Neha Narkhede]

Could you give more details about what errors/issues you ran into while
trying to run the unit tests?


On Mon, Sep 1, 2014 at 12:34 AM, Abhishek Sharma 
wrote:

> Hi,
>
> Recently, I setup Kafka workspace using Eclipse and did try to run several
> unit test cases from Eclipse only.
>
> Hard luck, not able to run and debug them.
>
> Anyone, please suggest me over this. How to run and debug unit test cases
> and Kafka application using Eclipse.
>
> Thanks
> Abhishek Sharma
>

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system tests can run

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118483#comment-14118483
 ] 

Guozhang Wang commented on KAFKA-1485:
--

Confirmed it is in now. Thanks.

> Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system tests can run
> --
>
> Key: KAFKA-1485
> URL: https://issues.apache.org/jira/browse/KAFKA-1485
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1.1
>Reporter: Machiel Groeneveld
>Assignee: Gwen Shapira
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1485.2.patch, KAFKA-1485.3.patch, 
> KAFKA-1485.4.patch, KAFKA-1485.patch
>
>
> I can't run projects alongside Kafka that use zookeeper 3.4 jars. 3.4 has 
> been out for 2.5 years and seems to be ready for adoption.
> In particular Apache Storm will upgrade to Zookeeper 3.4.x in their next 
> 0.9.2 release. I can't run both versions in my tests at the same time. 
> The only compile problem I saw was in EmbeddedZookeeper.scala 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: How to debug unit test in Eclipse

[Abhishek Sharma]

Thanks for the reply.

Everything is working fine. I just wish to execute unit test cases from 
eclipse IDE and want to debug them. Is this possible??


On Tuesday 02 September 2014 11:53 PM, Neha Narkhede wrote:

Could you give more details about what errors/issues you ran into while
trying to run the unit tests?


On Mon, Sep 1, 2014 at 12:34 AM, Abhishek Sharma 
wrote:


Hi,

Recently, I setup Kafka workspace using Eclipse and did try to run several
unit test cases from Eclipse only.

Hard luck, not able to run and debug them.

Anyone, please suggest me over this. How to run and debug unit test cases
and Kafka application using Eclipse.

Thanks
Abhishek Sharma

Re: Review Request 25155: Fix KAFKA-1616

[Guozhang Wang]

> On Sept. 2, 2014, 4:51 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/server/RequestPurgatory.scala, line 251
> > 
> >
> > It's probably better to expose an api that returns the size of the 
> > queue, instead of exposing the queue itself.

I made this change since the queue size needs to be accessed outside 
RequestPurgatory in the unit tests, and I thought since the reaper is a private 
class of the purgatory already, it is OK to expose its queue to the purgatory 
anyways. Let me know if you have a strong preference.


> On Sept. 2, 2014, 4:51 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/server/RequestPurgatory.scala, lines 268-270
> > 
> >
> > I thought that we only want to call purgeSatisfied if delayed.size >= 
> > purgeIntveral and similarly, only call watcherForKey...purgeSatisfied() if 
> > RequestPurgatory.this.size() >= purgeInterval?

Previously the logic checks on sum of watch list sizes and queue size, but it 
never check them separately. I thought since watch list sizes >= queue size in 
theory, and it is >> queue size in practice, we can just check on watch list 
sizes, and always purge both when the condition satisfies for simplicity. Let 
me know if you have a strong preference.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25155/#review52036
---


On Sept. 1, 2014, 9:41 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25155/
> ---
> 
> (Updated Sept. 1, 2014, 9:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1616
> https://issues.apache.org/jira/browse/KAFKA-1616
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Purgatory size to be the sum of watched list sizes; delayed request to be the 
> expiry queue length; remove atomic integers for metrics; add a unit test for 
> watched list sizes and enqueued requests
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala 
> ce06d2c381348deef8559374869fcaed923da1d1 
>   core/src/test/scala/unit/kafka/server/RequestPurgatoryTest.scala 
> 168712de241125982d556c188c76514fceb93779 
> 
> Diff: https://reviews.apache.org/r/25155/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Build failed in Jenkins: Kafka-trunk #259

[Apache Jenkins Server]

See 

Changes:

[joe.stein] KAFKA-1485 Upgrade to Zookeeper 3.4.6 missing file

--
[...truncated 1311 lines...]
at 
kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:36)
at 
kafka.api.ProducerFailureHandlingTest.setUp(ProducerFailureHandlingTest.scala:64)

kafka.api.ApiUtilsTest > testShortStringNonASCII PASSED

kafka.api.ApiUtilsTest > testShortStringASCII PASSED

kafka.api.test.ProducerCompressionTest > testCompression[0] FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:34)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at 
kafka.api.test.ProducerCompressionTest.setUp(ProducerCompressionTest.scala:54)

java.lang.NullPointerException
at 
kafka.api.test.ProducerCompressionTest.tearDown(ProducerCompressionTest.scala:60)

kafka.api.test.ProducerCompressionTest > testCompression[1] FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:34)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at 
kafka.api.test.ProducerCompressionTest.setUp(ProducerCompressionTest.scala:54)

java.lang.NullPointerException
at 
kafka.api.test.ProducerCompressionTest.tearDown(ProducerCompressionTest.scala:60)

kafka.api.test.ProducerSendTest > testAutoCreateTopic FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:34)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at 
kafka.api.test.ProducerSendTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerSendTest.scala:35)
at 
kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:36)
at kafka.api.test.ProducerSendTest.setUp(ProducerSendTest.scala:51)

kafka.api.test.ProducerSendTest > testSendOffset FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:34)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at 
kafka.api.test.ProducerSendTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerSendTest.scala:35)
at 
kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:36)
at kafka.api.test.ProducerSendTest.setUp(ProducerSendTest.scala:51)

kafka.api.test.ProducerSendTest > testClose FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:34)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at 
kafka.api.test.ProducerSendTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerSendTest.scala:35)
at 
kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerT

Re: Review Request 25155: Fix KAFKA-1616

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25155/
---

(Updated Sept. 2, 2014, 7:57 p.m.)


Review request for kafka.


Bugs: KAFKA-1616
https://issues.apache.org/jira/browse/KAFKA-1616


Repository: kafka


Description
---

Purgatory size to be the sum of watched list sizes; delayed request to be the 
expiry queue length; remove atomic integers for metrics; add a unit test for 
watched list sizes and enqueued requests


Diffs (updated)
-

  core/src/main/scala/kafka/server/RequestPurgatory.scala 
ce06d2c381348deef8559374869fcaed923da1d1 
  core/src/test/scala/unit/kafka/server/RequestPurgatoryTest.scala 
168712de241125982d556c188c76514fceb93779 

Diff: https://reviews.apache.org/r/25155/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1616) Purgatory Size and Num.Delayed.Request metrics are incorrect

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118628#comment-14118628
 ] 

Guozhang Wang commented on KAFKA-1616:
--

Updated reviewboard https://reviews.apache.org/r/25155/diff/
 against branch origin/trunk

> Purgatory Size and Num.Delayed.Request metrics are incorrect
> 
>
> Key: KAFKA-1616
> URL: https://issues.apache.org/jira/browse/KAFKA-1616
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1616.patch, KAFKA-1616_2014-08-28_10:12:17.patch, 
> KAFKA-1616_2014-09-01_14:41:56.patch, KAFKA-1616_2014-09-02_12:58:07.patch
>
>
> The request purgatory used two atomic integers "watched" and "unsatisfied" to 
> record the purgatory size ( = watched + unsatisfied) and number of delayed 
> requests ( = unsatisfied). But due to some race conditions these two atomic 
> integers are not updated correctly, result in incorrect metrics.
> Proposed solution: to have a cleaner semantics, we can define the "purgatory 
> size" to be just the number of elements in the watched lists, and the "number 
> of delayed requests" to be just the length of the expiry queue. And instead 
> of using two atomic integeres we just compute the size of the lists / queue 
> on the fly each time the metrics are pulled. This may use some more CPU 
> cycles for these two metrics but should be minor, and the correctness is 
> guaranteed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1616) Purgatory Size and Num.Delayed.Request metrics are incorrect

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1616:
-
Attachment: KAFKA-1616_2014-09-02_12:58:07.patch

> Purgatory Size and Num.Delayed.Request metrics are incorrect
> 
>
> Key: KAFKA-1616
> URL: https://issues.apache.org/jira/browse/KAFKA-1616
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1616.patch, KAFKA-1616_2014-08-28_10:12:17.patch, 
> KAFKA-1616_2014-09-01_14:41:56.patch, KAFKA-1616_2014-09-02_12:58:07.patch
>
>
> The request purgatory used two atomic integers "watched" and "unsatisfied" to 
> record the purgatory size ( = watched + unsatisfied) and number of delayed 
> requests ( = unsatisfied). But due to some race conditions these two atomic 
> integers are not updated correctly, result in incorrect metrics.
> Proposed solution: to have a cleaner semantics, we can define the "purgatory 
> size" to be just the number of elements in the watched lists, and the "number 
> of delayed requests" to be just the length of the expiry queue. And instead 
> of using two atomic integeres we just compute the size of the lists / queue 
> on the fly each time the metrics are pulled. This may use some more CPU 
> cycles for these two metrics but should be minor, and the correctness is 
> guaranteed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1616) Purgatory Size and Num.Delayed.Request metrics are incorrect

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118674#comment-14118674
 ] 

Guozhang Wang commented on KAFKA-1616:
--

Updated reviewboard https://reviews.apache.org/r/25155/diff/
 against branch origin/trunk

> Purgatory Size and Num.Delayed.Request metrics are incorrect
> 
>
> Key: KAFKA-1616
> URL: https://issues.apache.org/jira/browse/KAFKA-1616
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1616.patch, KAFKA-1616_2014-08-28_10:12:17.patch, 
> KAFKA-1616_2014-09-01_14:41:56.patch, KAFKA-1616_2014-09-02_12:58:07.patch, 
> KAFKA-1616_2014-09-02_13:23:13.patch
>
>
> The request purgatory used two atomic integers "watched" and "unsatisfied" to 
> record the purgatory size ( = watched + unsatisfied) and number of delayed 
> requests ( = unsatisfied). But due to some race conditions these two atomic 
> integers are not updated correctly, result in incorrect metrics.
> Proposed solution: to have a cleaner semantics, we can define the "purgatory 
> size" to be just the number of elements in the watched lists, and the "number 
> of delayed requests" to be just the length of the expiry queue. And instead 
> of using two atomic integeres we just compute the size of the lists / queue 
> on the fly each time the metrics are pulled. This may use some more CPU 
> cycles for these two metrics but should be minor, and the correctness is 
> guaranteed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25155: Fix KAFKA-1616

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25155/
---

(Updated Sept. 2, 2014, 8:22 p.m.)


Review request for kafka.


Bugs: KAFKA-1616
https://issues.apache.org/jira/browse/KAFKA-1616


Repository: kafka


Description
---

Purgatory size to be the sum of watched list sizes; delayed request to be the 
expiry queue length; remove atomic integers for metrics; add a unit test for 
watched list sizes and enqueued requests


Diffs (updated)
-

  core/src/main/scala/kafka/server/RequestPurgatory.scala 
ce06d2c381348deef8559374869fcaed923da1d1 
  core/src/test/scala/unit/kafka/server/RequestPurgatoryTest.scala 
168712de241125982d556c188c76514fceb93779 

Diff: https://reviews.apache.org/r/25155/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1616) Purgatory Size and Num.Delayed.Request metrics are incorrect

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1616:
-
Attachment: KAFKA-1616_2014-09-02_13:23:13.patch

> Purgatory Size and Num.Delayed.Request metrics are incorrect
> 
>
> Key: KAFKA-1616
> URL: https://issues.apache.org/jira/browse/KAFKA-1616
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1616.patch, KAFKA-1616_2014-08-28_10:12:17.patch, 
> KAFKA-1616_2014-09-01_14:41:56.patch, KAFKA-1616_2014-09-02_12:58:07.patch, 
> KAFKA-1616_2014-09-02_13:23:13.patch
>
>
> The request purgatory used two atomic integers "watched" and "unsatisfied" to 
> record the purgatory size ( = watched + unsatisfied) and number of delayed 
> requests ( = unsatisfied). But due to some race conditions these two atomic 
> integers are not updated correctly, result in incorrect metrics.
> Proposed solution: to have a cleaner semantics, we can define the "purgatory 
> size" to be just the number of elements in the watched lists, and the "number 
> of delayed requests" to be just the length of the expiry queue. And instead 
> of using two atomic integeres we just compute the size of the lists / queue 
> on the fly each time the metrics are pulled. This may use some more CPU 
> cycles for these two metrics but should be minor, and the correctness is 
> guaranteed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 24676: Fix KAFKA-1583

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24676/
---

(Updated Sept. 2, 2014, 8:37 p.m.)


Review request for kafka.


Bugs: KAFKA-1583
https://issues.apache.org/jira/browse/KAFKA-1583


Repository: kafka


Description (updated)
---

Incorporated Jun's comments round three


Diffs (updated)
-

  core/src/main/scala/kafka/api/FetchRequest.scala 
51cdccf7f90eb530cc62b094ed822b8469d50b12 
  core/src/main/scala/kafka/api/FetchResponse.scala 
af9308737bf7832eca018c2b3ede703f7d1209f1 
  core/src/main/scala/kafka/api/OffsetCommitRequest.scala 
861a6cf11dc6b6431fcbbe9de00c74a122f204bd 
  core/src/main/scala/kafka/api/ProducerRequest.scala 
b2366e7eedcac17f657271d5293ff0bef6f3cbe6 
  core/src/main/scala/kafka/api/ProducerResponse.scala 
a286272c834b6f40164999ff8b7f8998875f2cfe 
  core/src/main/scala/kafka/cluster/Partition.scala 
ff106b47e6ee194cea1cf589474fef975b9dd7e2 
  core/src/main/scala/kafka/common/ErrorMapping.scala 
3fae7910e4ce17bc8325887a046f383e0c151d44 
  core/src/main/scala/kafka/log/Log.scala 
0ddf97bd30311b6039e19abade41d2fbbad2f59b 
  core/src/main/scala/kafka/network/BoundedByteBufferSend.scala 
a624359fb2059340bb8dc1619c5b5f226e26eb9b 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
e0f14e25af03e6d4344386dcabc1457ee784d345 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
9481508fc2d6140b36829840c337e557f3d090da 
  core/src/main/scala/kafka/server/FetchRequestPurgatory.scala 
ed1318891253556cdf4d908033b704495acd5724 
  core/src/main/scala/kafka/server/KafkaApis.scala 
c584b559416b3ee4bcbec5966be4891e0a03eefb 
  core/src/main/scala/kafka/server/OffsetManager.scala 
43eb2a35bb54d32c66cdb94772df657b3a104d1a 
  core/src/main/scala/kafka/server/ProducerRequestPurgatory.scala 
d4a7d4a79b44263a1f7e1a92874dd36aa06e5a3f 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
68758e35d496a4659819960ae8e809d6e215568e 
  core/src/main/scala/kafka/server/RequestPurgatory.scala 
ce06d2c381348deef8559374869fcaed923da1d1 
  core/src/main/scala/kafka/utils/DelayedItem.scala 
d7276494072f14f1cdf7d23f755ac32678c5675c 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
03a424d45215e1e7780567d9559dae4d0ae6fc29 
  core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
cd302aa51eb8377d88b752d48274e403926439f2 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
a9c4ddc78df0b3695a77a12cf8cf25521a203122 
  core/src/test/scala/unit/kafka/server/RequestPurgatoryTest.scala 
168712de241125982d556c188c76514fceb93779 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
ab60e9b3a4d063c838bdc7f97b3ac7d2ede87072 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
09ed8f5a7a414ae139803bf82d336c2d80bf4ac5 

Diff: https://reviews.apache.org/r/24676/diff/


Testing
---

Unit tests


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1583) Kafka API Refactoring

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1583:
-
Attachment: KAFKA-1583_2014-09-02_13:37:47.patch

> Kafka API Refactoring
> -
>
> Key: KAFKA-1583
> URL: https://issues.apache.org/jira/browse/KAFKA-1583
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, 
> KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, 
> KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch
>
>
> This is the next step of KAFKA-1430. Details can be found at this page:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1583) Kafka API Refactoring

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118696#comment-14118696
 ] 

Guozhang Wang commented on KAFKA-1583:
--

Updated reviewboard https://reviews.apache.org/r/24676/diff/
 against branch origin/trunk

> Kafka API Refactoring
> -
>
> Key: KAFKA-1583
> URL: https://issues.apache.org/jira/browse/KAFKA-1583
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, 
> KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, 
> KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch
>
>
> This is the next step of KAFKA-1430. Details can be found at this page:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25136: Patch for KAFKA-1610

[Mayuresh Gharat]

> On Aug. 29, 2014, 9:01 p.m., Guozhang Wang wrote:
> > LGTM. One minor thing about comments is that we do not need to say 
> > "Changing mapValues to map" since we do not need to leave comments 
> > indicating code change, but just comment on the purpose of coding. We can 
> > generally say sth. like "Create a new collection with map since it (maybe 
> > modified outside this block / in another function call, etc)".

Cool. I will change those comments and upload a new patch. Thanks.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/#review51909
---


On Aug. 29, 2014, 5:04 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25136/
> ---
> 
> (Updated Aug. 29, 2014, 5:04 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1610
> https://issues.apache.org/jira/browse/KAFKA-1610
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Added comments explaining the changes and reverted back some changes as per 
> comments on the reviewboard
> 
> 
> Removed the unnecessary import
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> 691d69a49a240f38883d2025afaec26fd61281b5 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 8ab4a1b8072c9dd187a9a6e94138b725d1f1b153 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> e0f14e25af03e6d4344386dcabc1457ee784d345 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 9481508fc2d6140b36829840c337e557f3d090da 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> c584b559416b3ee4bcbec5966be4891e0a03eefb 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
> af4783646803e58714770c21f8c3352370f26854 
>   core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
> c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e 
> 
> Diff: https://reviews.apache.org/r/25136/diff/
> 
> 
> Testing
> ---
> 
> Ran the unit tests and everything passed and the build succeeeded
> 
> 
> Thanks,
> 
> Mayuresh Gharat
> 
>

Re: Offset Request with timestamp

[Guozhang Wang]

The semantic of the offset API is to "return the latest possible offset of
the message that is appended no later than the given timestamp". For
implementation, it will get the starting offset of the log segment that is
created no later than the given timestamp, and hence if your log segment
contains data for a long period of time, then the offset API may return you
just the starting offset of the current log segment.

If your traffic is small and you still want a finer grained offset
response, you can try to reduce the log segment size (default to 1 GB);
however doing so will increase the number of file handlers with more
frequent log segment rolling.

Guozhang


On Tue, Sep 2, 2014 at 10:21 AM, Manjunath Shivakumar <
manjunath.shivaku...@betfair.com> wrote:

> Hi,
>
> My usecase is to fetch the offsets for a given topic from X milliseconds
> ago.
> If I use the offset api
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI
>
> to do this and pass in a timestamp of (now() - X), I get the earliest
> offset in the current log segment and not the offset from X milliseconds
> ago.
>
> Is this the correct usage or behaviour?
>
> Thanks,
> Manju
>
> 
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> 




-- 
-- Guozhang

Re: Review Request 24676: Fix KAFKA-1583

[Guozhang Wang]

> On Sept. 2, 2014, 5:53 p.m., Jun Rao wrote:
> > Are you including the changes in kafka-1616 too? That would be fine. 
> > However, the comments in the other jira also need to be addressed in this 
> > patch.

I was not intending to include the changes of KAFKA-1616. The plan is to first 
check int K-1616, then rebase K-1583 on that. However some of the changes may 
refect some review comments in K-1616 just for ease of rebasing. I can revert 
these back if you want.


> On Sept. 2, 2014, 5:53 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/cluster/Partition.scala, line 282
> > 
> >
> > may now in => may not be in

I think it should be "may now be in"?


> On Sept. 2, 2014, 5:53 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/server/RequestPurgatory.scala, lines 343-349
> > 
> >
> > Not sure if we need the while loop any more. The comments may also need 
> > to be adjusted.

Good point. For case "curr != null && cur.forceComplete() == false" we can just 
return as well.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24676/#review52044
---


On Sept. 2, 2014, 8:37 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24676/
> ---
> 
> (Updated Sept. 2, 2014, 8:37 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1583
> https://issues.apache.org/jira/browse/KAFKA-1583
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Incorporated Jun's comments round three
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/api/FetchRequest.scala 
> 51cdccf7f90eb530cc62b094ed822b8469d50b12 
>   core/src/main/scala/kafka/api/FetchResponse.scala 
> af9308737bf7832eca018c2b3ede703f7d1209f1 
>   core/src/main/scala/kafka/api/OffsetCommitRequest.scala 
> 861a6cf11dc6b6431fcbbe9de00c74a122f204bd 
>   core/src/main/scala/kafka/api/ProducerRequest.scala 
> b2366e7eedcac17f657271d5293ff0bef6f3cbe6 
>   core/src/main/scala/kafka/api/ProducerResponse.scala 
> a286272c834b6f40164999ff8b7f8998875f2cfe 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> ff106b47e6ee194cea1cf589474fef975b9dd7e2 
>   core/src/main/scala/kafka/common/ErrorMapping.scala 
> 3fae7910e4ce17bc8325887a046f383e0c151d44 
>   core/src/main/scala/kafka/log/Log.scala 
> 0ddf97bd30311b6039e19abade41d2fbbad2f59b 
>   core/src/main/scala/kafka/network/BoundedByteBufferSend.scala 
> a624359fb2059340bb8dc1619c5b5f226e26eb9b 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> e0f14e25af03e6d4344386dcabc1457ee784d345 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 9481508fc2d6140b36829840c337e557f3d090da 
>   core/src/main/scala/kafka/server/FetchRequestPurgatory.scala 
> ed1318891253556cdf4d908033b704495acd5724 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> c584b559416b3ee4bcbec5966be4891e0a03eefb 
>   core/src/main/scala/kafka/server/OffsetManager.scala 
> 43eb2a35bb54d32c66cdb94772df657b3a104d1a 
>   core/src/main/scala/kafka/server/ProducerRequestPurgatory.scala 
> d4a7d4a79b44263a1f7e1a92874dd36aa06e5a3f 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 68758e35d496a4659819960ae8e809d6e215568e 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala 
> ce06d2c381348deef8559374869fcaed923da1d1 
>   core/src/main/scala/kafka/utils/DelayedItem.scala 
> d7276494072f14f1cdf7d23f755ac32678c5675c 
>   core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
> 03a424d45215e1e7780567d9559dae4d0ae6fc29 
>   core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
> cd302aa51eb8377d88b752d48274e403926439f2 
>   core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
> a9c4ddc78df0b3695a77a12cf8cf25521a203122 
>   core/src/test/scala/unit/kafka/server/RequestPurgatoryTest.scala 
> 168712de241125982d556c188c76514fceb93779 
>   core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
> ab60e9b3a4d063c838bdc7f97b3ac7d2ede87072 
>   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
> 09ed8f5a7a414ae139803bf82d336c2d80bf4ac5 
> 
> Diff: https://reviews.apache.org/r/24676/diff/
> 
> 
> Testing
> ---
> 
> Unit tests
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Re: Review Request 25236: Patch for KAFKA-1619

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25236/#review52093
---

Ship it!


Ship It!

- Neha Narkhede


On Sept. 1, 2014, 5:55 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25236/
> ---
> 
> (Updated Sept. 1, 2014, 5:55 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1619
> https://issues.apache.org/jira/browse/KAFKA-1619
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> fix readme
> 
> 
> fix typo in readme
> 
> 
> Diffs
> -
> 
>   README.md 8cd5cfd1e04dbef3c0878ff477eebea9ac749233 
>   build.gradle 6d6f1a4349d71bd1b56b7cc5c450046a75a83d6e 
>   settings.gradle 6041784d6f84c66bb1e9df2bc112e1b06c0bb000 
> 
> Diff: https://reviews.apache.org/r/25236/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

Re: How to debug unit test in Eclipse

[Neha Narkhede]

Yes, it is possible. Looks like you are hitting some issues, so if you can
be more specific about the details, someone can help you out.


On Tue, Sep 2, 2014 at 2:34 PM, Abhishek Sharma 
wrote:

> Thanks for the reply.
>
> Everything is working fine. I just wish to execute unit test cases from
> eclipse IDE and want to debug them. Is this possible??
>
>
> On Tuesday 02 September 2014 11:53 PM, Neha Narkhede wrote:
>
>> Could you give more details about what errors/issues you ran into while
>> trying to run the unit tests?
>>
>>
>> On Mon, Sep 1, 2014 at 12:34 AM, Abhishek Sharma 
>> wrote:
>>
>>  Hi,
>>>
>>> Recently, I setup Kafka workspace using Eclipse and did try to run
>>> several
>>> unit test cases from Eclipse only.
>>>
>>> Hard luck, not able to run and debug them.
>>>
>>> Anyone, please suggest me over this. How to run and debug unit test cases
>>> and Kafka application using Eclipse.
>>>
>>> Thanks
>>> Abhishek Sharma
>>>
>>>
>

[jira] [Commented] (KAFKA-998) Producer should not retry on non-recoverable error codes

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118925#comment-14118925
 ] 

Guozhang Wang commented on KAFKA-998:
-

This ticket can be closed as won't fix since we are moving to new producer now.

> Producer should not retry on non-recoverable error codes
> 
>
> Key: KAFKA-998
> URL: https://issues.apache.org/jira/browse/KAFKA-998
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Joel Koshy
>Assignee: Guozhang Wang
> Attachments: KAFKA-998.v1.patch
>
>
> Based on a discussion with Guozhang. The producer currently retries on all 
> error codes (including messagesizetoolarge which is pointless to retry on). 
> This can slow down the producer unnecessarily.
> If at all we want to retry on that error code we would need to retry with a 
> smaller batch size, but that's a separate discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-998) Producer should not retry on non-recoverable error codes

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-998:

Resolution: Won't Fix
Status: Resolved  (was: Patch Available)

> Producer should not retry on non-recoverable error codes
> 
>
> Key: KAFKA-998
> URL: https://issues.apache.org/jira/browse/KAFKA-998
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Joel Koshy
>Assignee: Guozhang Wang
> Attachments: KAFKA-998.v1.patch
>
>
> Based on a discussion with Guozhang. The producer currently retries on all 
> error codes (including messagesizetoolarge which is pointless to retry on). 
> This can slow down the producer unnecessarily.
> If at all we want to retry on that error code we would need to retry with a 
> smaller batch size, but that's a separate discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1498) new producer performance and bug improvements

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-1498.
--
Resolution: Fixed

> new producer performance and bug improvements
> -
>
> Key: KAFKA-1498
> URL: https://issues.apache.org/jira/browse/KAFKA-1498
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1498.patch, KAFKA-1498.patch, 
> KAFKA-1498_2014-06-25_16:44:51.patch, KAFKA-1498_2014-06-30_10:47:17.patch, 
> KAFKA-1498_2014-06-30_15:47:56.patch, KAFKA-1498_2014-07-01_11:12:41.patch, 
> kafka-1498.patch
>
>
> We have seen the following issues with the new producer.
> 1. The producer request can be significantly larger than the configured batch 
> size.
> 2. The bottleneck in mirrormaker when there are keyed messages and 
> compression is turned on.
> 3. The selector is woken up on every message in the new producer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1061) Break-down sendTime to multipleSendTime

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-1061.
--
Resolution: Won't Fix

> Break-down sendTime to multipleSendTime
> ---
>
> Key: KAFKA-1061
> URL: https://issues.apache.org/jira/browse/KAFKA-1061
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.8.2
>
>
> After KAFKA-1060 is done we would also like to break the sendTime to each 
> MultiSend's time and its corresponding send data size.
> This is related to KAFKA-1043



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

High Level Consumer and Commit

[Bhavesh Mistry]

Hi Kafka Group,

I have to pull the data from the Topic and index into Elastic Search with
Bulk API and wanted to commit only batch that has been committed and still
continue to read from topic further on same topic.  I have auto commit to
be off.


List  batch .

while (iterator.hasNext()) {
 batch.add(iterator.next().message());
 if(batch size is 50 ){
  //===  Once the bulk API is successful it will commit the offset
to zookeeper...
  executor.submit(new Thread() process batch and commit batch,
cconsumerConnector)
  batch = new batch buffer
   }
}

This commitOffset API commits all messages that have been read so far.
 What is best way to continue reading and only commit another thread finish
batch process is successful.  This will lead to fragmentation of the
Consumer offset so what is best way to implement continuous reading stream
and commit the rage offset.

Is Simple Consumer a better approach for this.


Thanks,

Bhavesh







Thanks,
Bhavesh

Re: High Level Consumer and Commit

[Philip O'Toole]

Either use the SimpleConsumer which gives you much finer-grained control, or 
(this worked with 0.7) spin up a ConsumerConnection (this is a HighLevel 
consumer concept) per partition, turn off auto-commit.

Philip

 
-
http://www.philipotoole.com 


On Tuesday, September 2, 2014 4:38 PM, Bhavesh Mistry 
 wrote:
 


Hi Kafka Group,

I have to pull the data from the Topic and index into Elastic Search with
Bulk API and wanted to commit only batch that has been committed and still
continue to read from topic further on same topic.  I have auto commit to
be off.


List  batch .

while (iterator.hasNext()) {
batch.add(iterator.next().message());
if(batch size is 50 ){
  //===  Once the bulk API is successful it will commit the offset
to zookeeper...
  executor.submit(new Thread() process batch and commit batch,
cconsumerConnector)
  batch = new batch buffer
   }
}

This commitOffset API commits all messages that have been read so far.
What is best way to continue reading and only commit another thread finish
batch process is successful.  This will lead to fragmentation of the
Consumer offset so what is best way to implement continuous reading stream
and commit the rage offset.

Is Simple Consumer a better approach for this.


Thanks,

Bhavesh







Thanks,
Bhavesh

[jira] [Updated] (KAFKA-1591) Clean-up Unnecessary stack trace in error/warn logs

[Abhishek Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Sharma updated KAFKA-1591:
---
Status: Patch Available  (was: Open)

> Clean-up Unnecessary stack trace in error/warn logs
> ---
>
> Key: KAFKA-1591
> URL: https://issues.apache.org/jira/browse/KAFKA-1591
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>  Labels: newbie
> Fix For: 0.9.0
>
>
> Some of the unnecessary stack traces in error / warning log entries can 
> easily pollute the log files. Examples include KAFKA-1066, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1591) Clean-up Unnecessary stack trace in error/warn logs

[Abhishek Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Sharma updated KAFKA-1591:
---
Attachment: Jira-1591-SocketConnection-Warning.patch

First Patch File - Changed SocketConnection logging from INFO to TRACE.

> Clean-up Unnecessary stack trace in error/warn logs
> ---
>
> Key: KAFKA-1591
> URL: https://issues.apache.org/jira/browse/KAFKA-1591
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: Jira-1591-SocketConnection-Warning.patch
>
>
> Some of the unnecessary stack traces in error / warning log entries can 
> easily pollute the log files. Examples include KAFKA-1066, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)