[jira] [Comment Edited] (KAFKA-1745) Each new thread creates a PIPE and KQUEUE as open files during producer.send() and does not get cleared when the thread that creates them is cleared.

[Vishal (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199817#comment-14199817
 ] 

Vishal edited comment on KAFKA-1745 at 11/6/14 8:07 AM:


[~junrao] Sorry, I should have been more clear.
[~ewencp] I checked and I don't think I'm leaking producers. Anyway, this is a 
sample test case which can replicate the problem (check lsof every 10 seconds 
to notice the increase in KQUEUEs and PIPESs though the producer object is 
being reused):

{code}
public class Test {

private static Queue> producerPool = new 
ConcurrentLinkedQueue>();

private static ProducerConfig config;

static
{
Properties props = new Properties();
props.put("metadata.broker.list", "IP:Port");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
config = new ProducerConfig(props);
}

public static void main(String[] args) throws InterruptedException {

ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 10, 5, 
TimeUnit.SECONDS, new LinkedBlockingQueue());
tpe.allowCoreThreadTimeOut(true);
Thread run = new Thread(new Runnable() {

@Override
public void run() {
Producer producer = 
producerPool.poll();
if(producer == null)
{
producer = new Producer(config);
}
KeyedMessage data = new 
KeyedMessage("SaSMQ", "0", "test");
producer.send(data);
producerPool.add(producer);
}
});

while(true) //To make sure that the main program does not 
terminate
{
for(int i = 0;i<100; i++)
{
tpe.submit(run);
}
Thread.sleep(1); //10 seconds So that keepalive 
time is exceeded by the thread pool and threads are cleared. 
}
}

}
{code}


Though KQUEUE and PIPE does get cleared after some time (in this test case, it 
was after 1-2 minutes), why does it have to create a new one when a new thread 
accesses the producer object? 


was (Author: vishal m):
[~junrao] Sorry, I should have been more clear.
[~ewencp] I checked and I don't think I'm leaking producers. Anyway, this is a 
sample test case which can replicate the problem (check lsof every 10 seconds 
to notice the increase in KQUEUEs and PIPESs though the producer object is 
being reused):

{code}
public class Test {

private static Queue> producerPool = new 
ConcurrentLinkedQueue>();

private static ProducerConfig config;

static
{
Properties props = new Properties();
props.put("metadata.broker.list", "IP:Port");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
config = new ProducerConfig(props);
}

public static void main(String[] args) throws InterruptedException {

ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 10, 5, 
TimeUnit.SECONDS, new LinkedBlockingQueue());
tpe.allowCoreThreadTimeOut(true);
Thread run = new Thread(new Runnable() {

@Override
public void run() {
Producer producer = 
producerPool.poll();
if(producer == null)
{
producer = new Producer(config);
}
KeyedMessage data = new 
KeyedMessage("SaSMQ", "0", "test");
producer.send(data);
producerPool.add(producer);
}
});

while(true) //To make sure that the main program does not 
terminate
{
for(int i = 0;i<100; i++)
{
tpe.submit(run);
}
Thread.sleep(1); //10 seconds So that keepalive 
time is exceeded by the thread pool and threads are cleared. 
}
}

}
{code}


Though KQUEUE and PIPE does get cleared after some time, why 

[jira] [Commented] (KAFKA-1754) KOYA - Kafka on YARN

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200095#comment-14200095
 ] 

Steve Loughran commented on KAFKA-1754:
---

This is great! YARN is getting better and better at long-lived services; Kafka 
should fit in nicely —and allocation of disks would be a very interesting 
feature to add here.

# As arun says, your AM gets to choose whether to spawn a new container -and 
where. You could try to ask for one back on the old machine, though if all data 
was on in the container's workspace, you've lost it all.
# YARN will restart the AM, and if you ask it at launch time, it will keep the 
containers up. Your AM has the task of rebuilding all its state. Hadoop 2.6+ 
adds sliding windows on restarts, so long lived apps aren't considered failures 
if their AM fails once a fortnight.
# Security is an issue with long-lived apps. Have you tried this on secure 
clusters yet?

> KOYA - Kafka on YARN
> 
>
> Key: KAFKA-1754
> URL: https://issues.apache.org/jira/browse/KAFKA-1754
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Thomas Weise
> Attachments: DT-KOYA-Proposal- JIRA.pdf
>
>
> YARN (Hadoop 2.x) has enabled clusters to be used for a variety of workloads, 
> emerging as distributed operating system for big data applications. 
> Initiatives are on the way to bring long running services under the YARN 
> umbrella, leveraging it for centralized resource management and operations 
> ([YARN-896] and examples such as HBase, Accumulo or Memcached through 
> Slider). This JIRA is to propose KOYA (Kafka On Yarn), a YARN application 
> master to launch and manage Kafka clusters running on YARN. Brokers will use 
> resources allocated through YARN with support for recovery, monitoring etc. 
> Please see attached for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1745) Each new thread creates a PIPE and KQUEUE as open files during producer.send() and does not get cleared when the thread that creates them is cleared.

[Vishal (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199817#comment-14199817
 ] 

Vishal edited comment on KAFKA-1745 at 11/6/14 12:00 PM:
-

[~junrao] Sorry, I should have been more clear.
[~ewencp] I checked and I don't think I'm leaking producers. Anyway, this is a 
sample test case which can replicate the problem (check lsof every 10 seconds 
to notice the increase in KQUEUEs and PIPESs though the producer object is 
being reused):

{code}
public class Test {

private static Queue> producerPool = new 
ConcurrentLinkedQueue>();

private static ProducerConfig config;

static
{
Properties props = new Properties();
props.put("metadata.broker.list", "IP:Port");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
config = new ProducerConfig(props);
}

public static void main(String[] args) throws InterruptedException {

ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 10, 5, 
TimeUnit.SECONDS, new LinkedBlockingQueue());
tpe.allowCoreThreadTimeOut(true);
Thread run = new Thread(new Runnable() {

@Override
public void run() {
Producer producer = 
producerPool.poll();
if(producer == null)
{
producer = new Producer(config);
}
KeyedMessage data = new 
KeyedMessage("SaSMQ", "0", "test");
producer.send(data);
producerPool.add(producer);
}
});

while(true) //To make sure that the main program does not 
terminate
{
for(int i = 0;i<100; i++)
{
tpe.submit(run);
}
Thread.sleep(1); //10 seconds So that keepalive 
time is exceeded by the thread pool and threads are cleared. 
}
}

}
{code}


Though KQUEUE and PIPE does get cleared after some time (in this test case, it 
was after 1-2 minutes), why does kafka's api create a new one when a new thread 
accesses the producer object? 


was (Author: vishal m):
[~junrao] Sorry, I should have been more clear.
[~ewencp] I checked and I don't think I'm leaking producers. Anyway, this is a 
sample test case which can replicate the problem (check lsof every 10 seconds 
to notice the increase in KQUEUEs and PIPESs though the producer object is 
being reused):

{code}
public class Test {

private static Queue> producerPool = new 
ConcurrentLinkedQueue>();

private static ProducerConfig config;

static
{
Properties props = new Properties();
props.put("metadata.broker.list", "IP:Port");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
config = new ProducerConfig(props);
}

public static void main(String[] args) throws InterruptedException {

ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 10, 5, 
TimeUnit.SECONDS, new LinkedBlockingQueue());
tpe.allowCoreThreadTimeOut(true);
Thread run = new Thread(new Runnable() {

@Override
public void run() {
Producer producer = 
producerPool.poll();
if(producer == null)
{
producer = new Producer(config);
}
KeyedMessage data = new 
KeyedMessage("SaSMQ", "0", "test");
producer.send(data);
producerPool.add(producer);
}
});

while(true) //To make sure that the main program does not 
terminate
{
for(int i = 0;i<100; i++)
{
tpe.submit(run);
}
Thread.sleep(1); //10 seconds So that keepalive 
time is exceeded by the thread pool and threads are cleared. 
}
}

}
{code}


Though KQUEUE and PIPE does get cleared after some time (i

[jira] [Comment Edited] (KAFKA-1745) Each new thread creates a PIPE and KQUEUE as open files during producer.send() and does not get cleared when the thread that creates them is cleared.

[Vishal (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199817#comment-14199817
 ] 

Vishal edited comment on KAFKA-1745 at 11/6/14 12:00 PM:
-

[~junrao] Sorry, I should have been more clear.
[~ewencp] I checked and I don't think I'm leaking producers. Anyway, this is a 
sample test case which can replicate the problem (check lsof every 10 seconds 
to notice the increase in KQUEUEs and PIPESs though the producer object is 
being reused):

{code}
public class Test {

private static Queue> producerPool = new 
ConcurrentLinkedQueue>();

private static ProducerConfig config;

static
{
Properties props = new Properties();
props.put("metadata.broker.list", "IP:Port");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
config = new ProducerConfig(props);
}

public static void main(String[] args) throws InterruptedException {

ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 10, 5, 
TimeUnit.SECONDS, new LinkedBlockingQueue());
tpe.allowCoreThreadTimeOut(true);
Thread run = new Thread(new Runnable() {

@Override
public void run() {
Producer producer = 
producerPool.poll();
if(producer == null)
{
producer = new Producer(config);
}
KeyedMessage data = new 
KeyedMessage("SaSMQ", "0", "test");
producer.send(data);
producerPool.add(producer);
}
});

while(true) //To make sure that the main program does not 
terminate
{
for(int i = 0;i<100; i++)
{
tpe.submit(run);
}
Thread.sleep(1); //10 seconds So that keepalive 
time is exceeded by the thread pool and threads are cleared. 
}
}

}
{code}


Though KQUEUE and PIPE does get cleared after some time (in this test case, it 
was after 1-2 minutes), why does Kafka's api create a new one when a new thread 
accesses the producer object? 


was (Author: vishal m):
[~junrao] Sorry, I should have been more clear.
[~ewencp] I checked and I don't think I'm leaking producers. Anyway, this is a 
sample test case which can replicate the problem (check lsof every 10 seconds 
to notice the increase in KQUEUEs and PIPESs though the producer object is 
being reused):

{code}
public class Test {

private static Queue> producerPool = new 
ConcurrentLinkedQueue>();

private static ProducerConfig config;

static
{
Properties props = new Properties();
props.put("metadata.broker.list", "IP:Port");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
config = new ProducerConfig(props);
}

public static void main(String[] args) throws InterruptedException {

ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 10, 5, 
TimeUnit.SECONDS, new LinkedBlockingQueue());
tpe.allowCoreThreadTimeOut(true);
Thread run = new Thread(new Runnable() {

@Override
public void run() {
Producer producer = 
producerPool.poll();
if(producer == null)
{
producer = new Producer(config);
}
KeyedMessage data = new 
KeyedMessage("SaSMQ", "0", "test");
producer.send(data);
producerPool.add(producer);
}
});

while(true) //To make sure that the main program does not 
terminate
{
for(int i = 0;i<100; i++)
{
tpe.submit(run);
}
Thread.sleep(1); //10 seconds So that keepalive 
time is exceeded by the thread pool and threads are cleared. 
}
}

}
{code}


Though KQUEUE and PIPE does get cleared after some time (i

[jira] [Comment Edited] (KAFKA-1745) Each new thread creates a PIPE and KQUEUE as open files during producer.send() and does not get cleared when the thread that creates them is cleared.

[Vishal (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199817#comment-14199817
 ] 

Vishal edited comment on KAFKA-1745 at 11/6/14 12:04 PM:
-

[~junrao] Sorry, I should have been more clear.
[~ewencp] I checked and I don't think I'm leaking producers. Anyway, this is a 
sample test case which can replicate the problem (check lsof every 10 seconds 
to notice the increase in KQUEUEs and PIPESs though the producer object is 
being reused):

{code}
public class Test {

private static Queue> producerPool = new 
ConcurrentLinkedQueue>();

private static ProducerConfig config;

static
{
Properties props = new Properties();
props.put("metadata.broker.list", "IP:Port");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
config = new ProducerConfig(props);
}

public static void main(String[] args) throws InterruptedException {

ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 10, 5, 
TimeUnit.SECONDS, new LinkedBlockingQueue());
tpe.allowCoreThreadTimeOut(true);
Thread run = new Thread(new Runnable() {

@Override
public void run() {
Producer producer = 
producerPool.poll();
if(producer == null)
{
producer = new Producer(config);
}
KeyedMessage data = new 
KeyedMessage("SaSMQ", "0", "test");
producer.send(data);
producerPool.add(producer);
}
});

while(true) //To make sure that the main program does not 
terminate
{
for(int i = 0;i<100; i++)
{
tpe.submit(run);
}
Thread.sleep(1); //10 seconds So that the keep 
alive time set on the thread pool will be exceeded and the threads in the 
threadpool will be cleared after each iteration. 
}
}

}
{code}


Though KQUEUE and PIPE does get cleared after some time (in this test case, it 
was after 1-2 minutes), why does Kafka's api create a new one when a new thread 
accesses the producer object? 


was (Author: vishal m):
[~junrao] Sorry, I should have been more clear.
[~ewencp] I checked and I don't think I'm leaking producers. Anyway, this is a 
sample test case which can replicate the problem (check lsof every 10 seconds 
to notice the increase in KQUEUEs and PIPESs though the producer object is 
being reused):

{code}
public class Test {

private static Queue> producerPool = new 
ConcurrentLinkedQueue>();

private static ProducerConfig config;

static
{
Properties props = new Properties();
props.put("metadata.broker.list", "IP:Port");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
config = new ProducerConfig(props);
}

public static void main(String[] args) throws InterruptedException {

ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 10, 5, 
TimeUnit.SECONDS, new LinkedBlockingQueue());
tpe.allowCoreThreadTimeOut(true);
Thread run = new Thread(new Runnable() {

@Override
public void run() {
Producer producer = 
producerPool.poll();
if(producer == null)
{
producer = new Producer(config);
}
KeyedMessage data = new 
KeyedMessage("SaSMQ", "0", "test");
producer.send(data);
producerPool.add(producer);
}
});

while(true) //To make sure that the main program does not 
terminate
{
for(int i = 0;i<100; i++)
{
tpe.submit(run);
}
Thread.sleep(1); //10 seconds So that the keep 
alive time set on the thread pool will be exceeded and the threads in the 
threadpool will be clear

[jira] [Comment Edited] (KAFKA-1745) Each new thread creates a PIPE and KQUEUE as open files during producer.send() and does not get cleared when the thread that creates them is cleared.

[Vishal (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199817#comment-14199817
 ] 

Vishal edited comment on KAFKA-1745 at 11/6/14 12:03 PM:
-

[~junrao] Sorry, I should have been more clear.
[~ewencp] I checked and I don't think I'm leaking producers. Anyway, this is a 
sample test case which can replicate the problem (check lsof every 10 seconds 
to notice the increase in KQUEUEs and PIPESs though the producer object is 
being reused):

{code}
public class Test {

private static Queue> producerPool = new 
ConcurrentLinkedQueue>();

private static ProducerConfig config;

static
{
Properties props = new Properties();
props.put("metadata.broker.list", "IP:Port");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
config = new ProducerConfig(props);
}

public static void main(String[] args) throws InterruptedException {

ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 10, 5, 
TimeUnit.SECONDS, new LinkedBlockingQueue());
tpe.allowCoreThreadTimeOut(true);
Thread run = new Thread(new Runnable() {

@Override
public void run() {
Producer producer = 
producerPool.poll();
if(producer == null)
{
producer = new Producer(config);
}
KeyedMessage data = new 
KeyedMessage("SaSMQ", "0", "test");
producer.send(data);
producerPool.add(producer);
}
});

while(true) //To make sure that the main program does not 
terminate
{
for(int i = 0;i<100; i++)
{
tpe.submit(run);
}
Thread.sleep(1); //10 seconds So that the keep 
alive time set on the thread pool will be exceeded and the threads in the 
threadpool will be cleared. 
}
}

}
{code}


Though KQUEUE and PIPE does get cleared after some time (in this test case, it 
was after 1-2 minutes), why does Kafka's api create a new one when a new thread 
accesses the producer object? 


was (Author: vishal m):
[~junrao] Sorry, I should have been more clear.
[~ewencp] I checked and I don't think I'm leaking producers. Anyway, this is a 
sample test case which can replicate the problem (check lsof every 10 seconds 
to notice the increase in KQUEUEs and PIPESs though the producer object is 
being reused):

{code}
public class Test {

private static Queue> producerPool = new 
ConcurrentLinkedQueue>();

private static ProducerConfig config;

static
{
Properties props = new Properties();
props.put("metadata.broker.list", "IP:Port");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
config = new ProducerConfig(props);
}

public static void main(String[] args) throws InterruptedException {

ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 10, 5, 
TimeUnit.SECONDS, new LinkedBlockingQueue());
tpe.allowCoreThreadTimeOut(true);
Thread run = new Thread(new Runnable() {

@Override
public void run() {
Producer producer = 
producerPool.poll();
if(producer == null)
{
producer = new Producer(config);
}
KeyedMessage data = new 
KeyedMessage("SaSMQ", "0", "test");
producer.send(data);
producerPool.add(producer);
}
});

while(true) //To make sure that the main program does not 
terminate
{
for(int i = 0;i<100; i++)
{
tpe.submit(run);
}
Thread.sleep(1); //10 seconds So that keepalive 
time is exceeded by the thread pool and threads are cleared. 
}
}

}
{code}


Though KQUEUE and

[jira] [Created] (KAFKA-1757) Can not delete Topic index on Windows

[JIRA]

Lukáš Vyhlídka created KAFKA-1757:
-

 Summary: Can not delete Topic index on Windows
 Key: KAFKA-1757
 URL: https://issues.apache.org/jira/browse/KAFKA-1757
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 0.8.2
Reporter: Lukáš Vyhlídka
Assignee: Jay Kreps
Priority: Minor


When running the Kafka 0.8.2-Beta (Scala 2.10) on Windows, an attempt to delete 
the Topic throwed an error:

ERROR [KafkaApi-1] error when handling request Name: StopReplicaRequest; 
Version: 0; CorrelationId: 38; ClientId: ; DeletePartitions: true; 
ControllerId: 0; ControllerEpoch: 3; Partitions: [test,0] 
(kafka.server.KafkaApis)
kafka.common.KafkaStorageException: Delete of index .index 
failed.
at kafka.log.LogSegment.delete(LogSegment.scala:283)
at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.log.Log.delete(Log.scala:608)
at kafka.log.LogManager.deleteLog(LogManager.scala:375)
at 
kafka.cluster.Partition$$anonfun$delete$1.apply$mcV$sp(Partition.scala:144)
at kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
at kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at kafka.utils.Utils$.inWriteLock(Utils.scala:543)
at kafka.cluster.Partition.delete(Partition.scala:139)
at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:158)
at 
kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:191)
at 
kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:190)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:190)
at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:96)
at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
at java.lang.Thread.run(Thread.java:744)



When I have investigated the issue I figured out that the index file (in my 
environment it was 
C:\tmp\kafka-logs\----0014-0\.index)
 was locked by the kafka process and the OS did not allow to delete that file.

I tried to fix the problem in source codes and when I added close() method call 
into LogSegment.delete(), the Topic deletion started to work.

I will add here (not sure how to upload the file during issue creation) a diff 
with the changes I have made so You can take a look on that whether it is 
reasonable or not. It would be perfect if it could make it into the product...

In the end I would like to say that on Linux the deletion works just fine...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1757) Can not delete Topic index on Windows

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukáš Vyhlídka updated KAFKA-1757:
--
Fix Version/s: 0.8.2
   Status: Patch Available  (was: Open)

> Can not delete Topic index on Windows
> -
>
> Key: KAFKA-1757
> URL: https://issues.apache.org/jira/browse/KAFKA-1757
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.2
>Reporter: Lukáš Vyhlídka
>Assignee: Jay Kreps
>Priority: Minor
> Fix For: 0.8.2
>
>
> When running the Kafka 0.8.2-Beta (Scala 2.10) on Windows, an attempt to 
> delete the Topic throwed an error:
> ERROR [KafkaApi-1] error when handling request Name: StopReplicaRequest; 
> Version: 0; CorrelationId: 38; ClientId: ; DeletePartitions: true; 
> ControllerId: 0; ControllerEpoch: 3; Partitions: [test,0] 
> (kafka.server.KafkaApis)
> kafka.common.KafkaStorageException: Delete of index 
> .index failed.
> at kafka.log.LogSegment.delete(LogSegment.scala:283)
> at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
> at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.log.Log.delete(Log.scala:608)
> at kafka.log.LogManager.deleteLog(LogManager.scala:375)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply$mcV$sp(Partition.scala:144)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.utils.Utils$.inWriteLock(Utils.scala:543)
> at kafka.cluster.Partition.delete(Partition.scala:139)
> at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:158)
> at 
> kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:191)
> at 
> kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:190)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
> at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:190)
> at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:96)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
> at java.lang.Thread.run(Thread.java:744)
> When I have investigated the issue I figured out that the index file (in my 
> environment it was 
> C:\tmp\kafka-logs\----0014-0\.index)
>  was locked by the kafka process and the OS did not allow to delete that file.
> I tried to fix the problem in source codes and when I added close() method 
> call into LogSegment.delete(), the Topic deletion started to work.
> I will add here (not sure how to upload the file during issue creation) a 
> diff with the changes I have made so You can take a look on that whether it 
> is reasonable or not. It would be perfect if it could make it into the 
> product...
> In the end I would like to say that on Linux the deletion works just fine...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1757) Can not delete Topic index on Windows

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukáš Vyhlídka updated KAFKA-1757:
--
Status: Open  (was: Patch Available)

> Can not delete Topic index on Windows
> -
>
> Key: KAFKA-1757
> URL: https://issues.apache.org/jira/browse/KAFKA-1757
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.2
>Reporter: Lukáš Vyhlídka
>Assignee: Jay Kreps
>Priority: Minor
> Fix For: 0.8.2
>
>
> When running the Kafka 0.8.2-Beta (Scala 2.10) on Windows, an attempt to 
> delete the Topic throwed an error:
> ERROR [KafkaApi-1] error when handling request Name: StopReplicaRequest; 
> Version: 0; CorrelationId: 38; ClientId: ; DeletePartitions: true; 
> ControllerId: 0; ControllerEpoch: 3; Partitions: [test,0] 
> (kafka.server.KafkaApis)
> kafka.common.KafkaStorageException: Delete of index 
> .index failed.
> at kafka.log.LogSegment.delete(LogSegment.scala:283)
> at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
> at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.log.Log.delete(Log.scala:608)
> at kafka.log.LogManager.deleteLog(LogManager.scala:375)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply$mcV$sp(Partition.scala:144)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.utils.Utils$.inWriteLock(Utils.scala:543)
> at kafka.cluster.Partition.delete(Partition.scala:139)
> at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:158)
> at 
> kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:191)
> at 
> kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:190)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
> at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:190)
> at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:96)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
> at java.lang.Thread.run(Thread.java:744)
> When I have investigated the issue I figured out that the index file (in my 
> environment it was 
> C:\tmp\kafka-logs\----0014-0\.index)
>  was locked by the kafka process and the OS did not allow to delete that file.
> I tried to fix the problem in source codes and when I added close() method 
> call into LogSegment.delete(), the Topic deletion started to work.
> I will add here (not sure how to upload the file during issue creation) a 
> diff with the changes I have made so You can take a look on that whether it 
> is reasonable or not. It would be perfect if it could make it into the 
> product...
> In the end I would like to say that on Linux the deletion works just fine...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1757) Can not delete Topic index on Windows

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukáš Vyhlídka updated KAFKA-1757:
--
Attachment: lucky-v.patch

A patch that looks like a fix for the issue.

> Can not delete Topic index on Windows
> -
>
> Key: KAFKA-1757
> URL: https://issues.apache.org/jira/browse/KAFKA-1757
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.2
>Reporter: Lukáš Vyhlídka
>Assignee: Jay Kreps
>Priority: Minor
> Fix For: 0.8.2
>
> Attachments: lucky-v.patch
>
>
> When running the Kafka 0.8.2-Beta (Scala 2.10) on Windows, an attempt to 
> delete the Topic throwed an error:
> ERROR [KafkaApi-1] error when handling request Name: StopReplicaRequest; 
> Version: 0; CorrelationId: 38; ClientId: ; DeletePartitions: true; 
> ControllerId: 0; ControllerEpoch: 3; Partitions: [test,0] 
> (kafka.server.KafkaApis)
> kafka.common.KafkaStorageException: Delete of index 
> .index failed.
> at kafka.log.LogSegment.delete(LogSegment.scala:283)
> at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
> at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.log.Log.delete(Log.scala:608)
> at kafka.log.LogManager.deleteLog(LogManager.scala:375)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply$mcV$sp(Partition.scala:144)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.utils.Utils$.inWriteLock(Utils.scala:543)
> at kafka.cluster.Partition.delete(Partition.scala:139)
> at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:158)
> at 
> kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:191)
> at 
> kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:190)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
> at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:190)
> at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:96)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
> at java.lang.Thread.run(Thread.java:744)
> When I have investigated the issue I figured out that the index file (in my 
> environment it was 
> C:\tmp\kafka-logs\----0014-0\.index)
>  was locked by the kafka process and the OS did not allow to delete that file.
> I tried to fix the problem in source codes and when I added close() method 
> call into LogSegment.delete(), the Topic deletion started to work.
> I will add here (not sure how to upload the file during issue creation) a 
> diff with the changes I have made so You can take a look on that whether it 
> is reasonable or not. It would be perfect if it could make it into the 
> product...
> In the end I would like to say that on Linux the deletion works just fine...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1757) Can not delete Topic index on Windows

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukáš Vyhlídka updated KAFKA-1757:
--
Status: Patch Available  (was: Open)

> Can not delete Topic index on Windows
> -
>
> Key: KAFKA-1757
> URL: https://issues.apache.org/jira/browse/KAFKA-1757
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.2
>Reporter: Lukáš Vyhlídka
>Assignee: Jay Kreps
>Priority: Minor
> Fix For: 0.8.2
>
> Attachments: lucky-v.patch
>
>
> When running the Kafka 0.8.2-Beta (Scala 2.10) on Windows, an attempt to 
> delete the Topic throwed an error:
> ERROR [KafkaApi-1] error when handling request Name: StopReplicaRequest; 
> Version: 0; CorrelationId: 38; ClientId: ; DeletePartitions: true; 
> ControllerId: 0; ControllerEpoch: 3; Partitions: [test,0] 
> (kafka.server.KafkaApis)
> kafka.common.KafkaStorageException: Delete of index 
> .index failed.
> at kafka.log.LogSegment.delete(LogSegment.scala:283)
> at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
> at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.log.Log.delete(Log.scala:608)
> at kafka.log.LogManager.deleteLog(LogManager.scala:375)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply$mcV$sp(Partition.scala:144)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.utils.Utils$.inWriteLock(Utils.scala:543)
> at kafka.cluster.Partition.delete(Partition.scala:139)
> at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:158)
> at 
> kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:191)
> at 
> kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:190)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
> at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:190)
> at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:96)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
> at java.lang.Thread.run(Thread.java:744)
> When I have investigated the issue I figured out that the index file (in my 
> environment it was 
> C:\tmp\kafka-logs\----0014-0\.index)
>  was locked by the kafka process and the OS did not allow to delete that file.
> I tried to fix the problem in source codes and when I added close() method 
> call into LogSegment.delete(), the Topic deletion started to work.
> I will add here (not sure how to upload the file during issue creation) a 
> diff with the changes I have made so You can take a look on that whether it 
> is reasonable or not. It would be perfect if it could make it into the 
> product...
> In the end I would like to say that on Linux the deletion works just fine...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1667) topic-level configuration not validated

[Dmytro Kostiuchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmytro Kostiuchenko updated KAFKA-1667:
---
Attachment: KAFKA-1667_2014-11-06_17:10:14.patch

>  topic-level configuration not validated
> 
>
> Key: KAFKA-1667
> URL: https://issues.apache.org/jira/browse/KAFKA-1667
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Ryan Berdeen
>  Labels: newbie
> Attachments: KAFKA-1667.patch, KAFKA-1667_2014-11-05_19:43:53.patch, 
> KAFKA-1667_2014-11-06_17:10:14.patch
>
>
> I was able to set the configuration for a topic to these invalid values:
> {code}
> Topic:topic-config-test  PartitionCount:1ReplicationFactor:2 
> Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
> {code}
> It seems that the values are saved as long as they are the correct type, but 
> are not validated like the corresponding broker-level properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27634: Patch for KAFKA-1667

[Dmytro Kostiuchenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27634/
---

(Updated Nov. 6, 2014, 4:10 p.m.)


Review request for kafka.


Bugs: KAFKA-1667
https://issues.apache.org/jira/browse/KAFKA-1667


Repository: kafka


Description (updated)
---

KAFKA-1667 Fixed bugs in LogConfig. Added test and documentation


KAFKA-1667 Updated tests to reflect new boolean property parsing logic


KAFKA-1677 renamed methods to match naming convention


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java 
c4cea2cc072f4db4ce014b63d226431d3766bef1 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
0b2735e7fc42ef9894bef1997b1f06a8ebee5439 
  core/src/main/scala/kafka/log/LogConfig.scala 
e48922a97727dd0b98f3ae630ebb0af3bef2373d 
  core/src/main/scala/kafka/utils/Utils.scala 
23aefb4715b177feae1d2f83e8b910653ea10c5f 
  core/src/test/scala/kafka/log/LogConfigTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala 
f44568cb25edf25db857415119018fd4c9922f61 

Diff: https://reviews.apache.org/r/27634/diff/


Testing
---


Thanks,

Dmytro Kostiuchenko

[jira] [Commented] (KAFKA-1667) topic-level configuration not validated

[Dmytro Kostiuchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200370#comment-14200370
 ] 

Dmytro Kostiuchenko commented on KAFKA-1667:


Updated reviewboard https://reviews.apache.org/r/27634/diff/
 against branch origin/trunk

>  topic-level configuration not validated
> 
>
> Key: KAFKA-1667
> URL: https://issues.apache.org/jira/browse/KAFKA-1667
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Ryan Berdeen
>  Labels: newbie
> Attachments: KAFKA-1667.patch, KAFKA-1667_2014-11-05_19:43:53.patch, 
> KAFKA-1667_2014-11-06_17:10:14.patch
>
>
> I was able to set the configuration for a topic to these invalid values:
> {code}
> Topic:topic-config-test  PartitionCount:1ReplicationFactor:2 
> Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
> {code}
> It seems that the values are saved as long as they are the correct type, but 
> are not validated like the corresponding broker-level properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27634: Patch for KAFKA-1667

[Dmytro Kostiuchenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27634/
---

(Updated Nov. 6, 2014, 4:12 p.m.)


Review request for kafka.


Bugs: KAFKA-1667
https://issues.apache.org/jira/browse/KAFKA-1667


Repository: kafka


Description (updated)
---

KAFKA-1667 Fixed bugs in LogConfig. Added test and documentation


KAFKA-1667 Updated tests to reflect new boolean property parsing logic


KAFKA-1667 renamed methods to match naming convention


Diffs
-

  clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java 
c4cea2cc072f4db4ce014b63d226431d3766bef1 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
0b2735e7fc42ef9894bef1997b1f06a8ebee5439 
  core/src/main/scala/kafka/log/LogConfig.scala 
e48922a97727dd0b98f3ae630ebb0af3bef2373d 
  core/src/main/scala/kafka/utils/Utils.scala 
23aefb4715b177feae1d2f83e8b910653ea10c5f 
  core/src/test/scala/kafka/log/LogConfigTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala 
f44568cb25edf25db857415119018fd4c9922f61 

Diff: https://reviews.apache.org/r/27634/diff/


Testing
---


Thanks,

Dmytro Kostiuchenko

[jira] [Commented] (KAFKA-1667) topic-level configuration not validated

[Dmytro Kostiuchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200381#comment-14200381
 ] 

Dmytro Kostiuchenko commented on KAFKA-1667:


The main issue with the old patch was handling {{Properties}} defaults.
When properties were passed to {{fromProps(Properties)}} method, {{Properties}} 
object has been exposed as a {{Map}}. {{Map.get()}}, though is implemented in 
{{Hashtable}} and thus knows nothing about {{Properties}} defaults. Fixed that, 
added few trivial tests to check general correctness.

There is an impact on the client code though: boolean values are now parsed via 
{{Boolean.parseBoolean}} throwing no exception but instead falling back to 
{{false}} for invalid input.

>  topic-level configuration not validated
> 
>
> Key: KAFKA-1667
> URL: https://issues.apache.org/jira/browse/KAFKA-1667
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Ryan Berdeen
>  Labels: newbie
> Attachments: KAFKA-1667.patch, KAFKA-1667_2014-11-05_19:43:53.patch, 
> KAFKA-1667_2014-11-06_17:10:14.patch
>
>
> I was able to set the configuration for a topic to these invalid values:
> {code}
> Topic:topic-config-test  PartitionCount:1ReplicationFactor:2 
> Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
> {code}
> It seems that the values are saved as long as they are the correct type, but 
> are not validated like the corresponding broker-level properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1754) KOYA - Kafka on YARN

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200409#comment-14200409
 ] 

Thomas Weise commented on KAFKA-1754:
-

[~ste...@apache.org] Yes, we are running DataTorrent on secure clusters. You 
are probably thinking about the token max life time issue as addressed in 
[SLIDER-474]?   

> KOYA - Kafka on YARN
> 
>
> Key: KAFKA-1754
> URL: https://issues.apache.org/jira/browse/KAFKA-1754
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Thomas Weise
> Attachments: DT-KOYA-Proposal- JIRA.pdf
>
>
> YARN (Hadoop 2.x) has enabled clusters to be used for a variety of workloads, 
> emerging as distributed operating system for big data applications. 
> Initiatives are on the way to bring long running services under the YARN 
> umbrella, leveraging it for centralized resource management and operations 
> ([YARN-896] and examples such as HBase, Accumulo or Memcached through 
> Slider). This JIRA is to propose KOYA (Kafka On Yarn), a YARN application 
> master to launch and manage Kafka clusters running on YARN. Brokers will use 
> resources allocated through YARN with support for recovery, monitoring etc. 
> Please see attached for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1754) KOYA - Kafka on YARN

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200455#comment-14200455
 ] 

Thomas Weise commented on KAFKA-1754:
-

[~acmurthy] Thanks for the support. Disk reservation will benefit Kafka 
greatly. Will reach out with a few other questions. 

{quote}
Inspired by this jira, I've opened YARN-2817... I'd like to throw out the idea 
that Kafka could start by reserving entire drives on nodes, exclusively when 
running on YARN. This would ensure that Kafka would not get interference from 
other applications like HDFS, MR etc.
{quote}

> KOYA - Kafka on YARN
> 
>
> Key: KAFKA-1754
> URL: https://issues.apache.org/jira/browse/KAFKA-1754
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Thomas Weise
> Attachments: DT-KOYA-Proposal- JIRA.pdf
>
>
> YARN (Hadoop 2.x) has enabled clusters to be used for a variety of workloads, 
> emerging as distributed operating system for big data applications. 
> Initiatives are on the way to bring long running services under the YARN 
> umbrella, leveraging it for centralized resource management and operations 
> ([YARN-896] and examples such as HBase, Accumulo or Memcached through 
> Slider). This JIRA is to propose KOYA (Kafka On Yarn), a YARN application 
> master to launch and manage Kafka clusters running on YARN. Brokers will use 
> resources allocated through YARN with support for recovery, monitoring etc. 
> Please see attached for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1754) KOYA - Kafka on YARN

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200457#comment-14200457
 ] 

Thomas Weise commented on KAFKA-1754:
-

[~acmurthy] Thanks for the support. Disk reservation will benefit Kafka
greatly. Will reach out with a few other questions.



> KOYA - Kafka on YARN
> 
>
> Key: KAFKA-1754
> URL: https://issues.apache.org/jira/browse/KAFKA-1754
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Thomas Weise
> Attachments: DT-KOYA-Proposal- JIRA.pdf
>
>
> YARN (Hadoop 2.x) has enabled clusters to be used for a variety of workloads, 
> emerging as distributed operating system for big data applications. 
> Initiatives are on the way to bring long running services under the YARN 
> umbrella, leveraging it for centralized resource management and operations 
> ([YARN-896] and examples such as HBase, Accumulo or Memcached through 
> Slider). This JIRA is to propose KOYA (Kafka On Yarn), a YARN application 
> master to launch and manage Kafka clusters running on YARN. Brokers will use 
> resources allocated through YARN with support for recovery, monitoring etc. 
> Please see attached for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 27684: Patch for KAFKA-1743

[Manikumar Reddy O]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27684/
---

Review request for kafka.


Bugs: KAFKA-1743
https://issues.apache.org/jira/browse/KAFKA-1743


Repository: kafka


Description
---

def commitOffsets method added to make ConsumerConnector backward  compatible


Diffs
-

  core/src/main/scala/kafka/consumer/ConsumerConnector.scala 
07677c1c26768ef9c9032626180d0015f12cb0e0 
  core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
fbc680fde21b02f11285a4f4b442987356abd17b 

Diff: https://reviews.apache.org/r/27684/diff/


Testing
---


Thanks,

Manikumar Reddy O

[jira] [Commented] (KAFKA-1743) ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200478#comment-14200478
 ] 

Manikumar Reddy commented on KAFKA-1743:


Created reviewboard https://reviews.apache.org/r/27684/diff/
 against branch origin/0.8.2

> ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible
> ---
>
> Key: KAFKA-1743
> URL: https://issues.apache.org/jira/browse/KAFKA-1743
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1743.patch
>
>
> In 0.8.1.x, ConsumerConnector has the following api:
>   def commitOffsets
> This is changed to the following in 0.8.2 and breaks compatibility
>   def commitOffsets(retryOnFailure: Boolean = true)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1743) ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-1743:
---
Attachment: KAFKA-1743.patch

> ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible
> ---
>
> Key: KAFKA-1743
> URL: https://issues.apache.org/jira/browse/KAFKA-1743
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1743.patch
>
>
> In 0.8.1.x, ConsumerConnector has the following api:
>   def commitOffsets
> This is changed to the following in 0.8.2 and breaks compatibility
>   def commitOffsets(retryOnFailure: Boolean = true)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1743) ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-1743:
---
Assignee: Manikumar Reddy
  Status: Patch Available  (was: Open)

> ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible
> ---
>
> Key: KAFKA-1743
> URL: https://issues.apache.org/jira/browse/KAFKA-1743
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Manikumar Reddy
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1743.patch
>
>
> In 0.8.1.x, ConsumerConnector has the following api:
>   def commitOffsets
> This is changed to the following in 0.8.2 and breaks compatibility
>   def commitOffsets(retryOnFailure: Boolean = true)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200510#comment-14200510
 ] 

Jun Rao commented on KAFKA-1738:


You actually found a real bug, thanks! We exposed an existing problem after 
adding the ability to kill idle connections in KAFKA-1282. The default max idle 
time happens to be 10 minutes. That's why you only see the issue if the topics 
are created more than 10 mins apart. I will attach a patch soon.

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1, 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInf

Review Request 27690: Patch for kafka-1738

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27690/
---

Review request for kafka.


Bugs: kafka-1738
https://issues.apache.org/jira/browse/kafka-1738


Repository: kafka


Description
---

try/catch should include channel.receive()


Diffs
-

  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 

Diff: https://reviews.apache.org/r/27690/diff/


Testing
---


Thanks,

Jun Rao

[jira] [Updated] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1738:
---
Attachment: kafka-1738.patch

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1, 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt, 
> kafka-1738.patch
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0)
>  to broker id:0,host:DMIPVM,port:9092. Reconnecting to broker. 
> (kafka.controller.RequestSendThread)
> java.nio.channels.ClosedChannelException
>   at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)

[jira] [Updated] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1738:
---
Assignee: Jun Rao
  Status: Patch Available  (was: Open)

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1, 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
>Assignee: Jun Rao
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt, 
> kafka-1738.patch
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0)
>  to broker id:0,host:DMIPVM,port:9092. Reconnecting to broker. 
> (kafka.controller.RequestSendThread)
> java.nio.channels.ClosedChannelException
>   a

[jira] [Commented] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200512#comment-14200512
 ] 

Jun Rao commented on KAFKA-1738:


Created reviewboard https://reviews.apache.org/r/27690/diff/
 against branch origin/trunk

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1, 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt, 
> kafka-1738.patch
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0)
>  to broker id:0,host:DMIPVM,port:9092. Reconnecting to broker. 
> (kafka.controller.RequestSendThread)

[jira] [Updated] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1738:
---
 Priority: Blocker  (was: Major)
Affects Version/s: (was: 0.8.1.1)
Fix Version/s: 0.8.2

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
>Assignee: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt, 
> kafka-1738.patch
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0)
>  to broker id:0,host:DMIPVM,port:9092. Reconnecting

[jira] [Commented] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200522#comment-14200522
 ] 

Jun Rao commented on KAFKA-1738:


Sri, Pradeep,

Do you think you can try the patch and see if this fixes your issue?

Also marking this as an 0.8.2 blocker.

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
>Assignee: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt, 
> kafka-1738.patch
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),Al

Re: No longer supporting Java 6, if? when?

[Steven Schlansker]

Java 6 has been End of Life since Feb 2013.
Java 7 (and 8, but unfortunately that's too new still) has very compelling
features which can make development a lot easier.

The sooner more projects drop Java 6 the better, in my opinion :)

On Nov 5, 2014, at 7:45 PM, Worthy LaFollette  wrote:

> Mostly converted now to 1.7, this would be welcomed to get any new
> features.
> 
> On Wed Nov 05 2014 at 7:32:55 PM Joe Stein  wrote:
> 
>> This has been coming up in a lot of projects and for other reasons too I
>> wanted to kick off the discussion about if/when we end support for Java 6.
>> Besides any API we may want to use in >= 7 we also compile our binaries for
>> 6 for release currently.
>> 
>> /***
>> Joe Stein
>> Founder, Principal Consultant
>> Big Data Open Source Security LLC
>> http://www.stealth.ly
>> Twitter: @allthingshadoop 
>> /
>> 

[jira] [Commented] (KAFKA-1723) make the metrics name in new producer more standard

[Otis Gospodnetic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200536#comment-14200536
 ] 

Otis Gospodnetic commented on KAFKA-1723:
-

I don't follow this 100%, [~junrao].

The "old producer" should be all set once KAFKA-1481 is committed - all (not 
just producer) mbean names will be consistent and easily parseable.

I think there are really 2 parts here:
# mbean naming consistency
# existence of mbean names

For 1) I think you are saying that when this "new producer" is released in 
0.8.3 whoever works on it will make sure the mbean names are consistent with 
the naming we got going via KAFKA-1481.
But what about 2)?  Are you saying this "new producer" will have some *new 
mbeans*?  Some *additional mbeans*?  Or are some mbeans that exist in 0.8.2 
going to be *removed*?

> make the metrics name in new producer more standard
> ---
>
> Key: KAFKA-1723
> URL: https://issues.apache.org/jira/browse/KAFKA-1723
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Fix For: 0.8.3
>
>
> The jmx name in the new producer looks like the following:
> kafka.producer.myclientid:type=mytopic
> However, this can be ambiguous since we allow "." in client id and topic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: No longer supporting Java 6, if? when?

[Gwen Shapira]

+1 for dropping Java 6

On Thu, Nov 6, 2014 at 9:31 AM, Steven Schlansker  wrote:

> Java 6 has been End of Life since Feb 2013.
> Java 7 (and 8, but unfortunately that's too new still) has very compelling
> features which can make development a lot easier.
>
> The sooner more projects drop Java 6 the better, in my opinion :)
>
> On Nov 5, 2014, at 7:45 PM, Worthy LaFollette  wrote:
>
> > Mostly converted now to 1.7, this would be welcomed to get any new
> > features.
> >
> > On Wed Nov 05 2014 at 7:32:55 PM Joe Stein  wrote:
> >
> >> This has been coming up in a lot of projects and for other reasons too I
> >> wanted to kick off the discussion about if/when we end support for Java
> 6.
> >> Besides any API we may want to use in >= 7 we also compile our binaries
> for
> >> 6 for release currently.
> >>
> >> /***
> >> Joe Stein
> >> Founder, Principal Consultant
> >> Big Data Open Source Security LLC
> >> http://www.stealth.ly
> >> Twitter: @allthingshadoop 
> >> /
> >>
>
>

Review Request 27691: Patch for KAFKA-1476

[Balaji Seshadri]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27691/
---

Review request for kafka.


Bugs: KAFKA-1476
https://issues.apache.org/jira/browse/KAFKA-1476


Repository: kafka


Description
---

KAFKA-328 Write unit test for kafka server startup and shutdown API


Diffs
-

  core/src/main/scala/kafka/tools/ConsumerCommand.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
56e3e88e0cc6d917b0ffd1254e173295c1c4aabd 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
1bfb501b2f29c50f3fc5f930fdaad02e03b91e4f 
  core/src/test/scala/unit/kafka/server/ServerStartupTest.scala 
a0ed4855f2550a0eb2e363dd2fccd8377a9ac172 

Diff: https://reviews.apache.org/r/27691/diff/


Testing
---


Thanks,

Balaji Seshadri

[jira] [Updated] (KAFKA-1476) Get a list of consumer groups

[BalajiSeshadri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BalajiSeshadri updated KAFKA-1476:
--
Attachment: KAFKA-1476.patch

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

[BalajiSeshadri (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200550#comment-14200550
 ] 

BalajiSeshadri commented on KAFKA-1476:
---

Created reviewboard https://reviews.apache.org/r/27691/diff/
 against branch origin/trunk

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1476) Get a list of consumer groups

[BalajiSeshadri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BalajiSeshadri updated KAFKA-1476:
--
Status: Patch Available  (was: In Progress)

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

[BalajiSeshadri (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200556#comment-14200556
 ] 

BalajiSeshadri commented on KAFKA-1476:
---

[~nehanarkhede] or [~junrao] or [~jkreps] Can anyone of you please review this 
?.I created review board for both of my patches KAFKA-328 and KAFKA-1476.

https://reviews.apache.org/r/27691/diff/

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 27693: Patch for KAFKA-1476

[Balaji Seshadri]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27693/
---

Review request for kafka.


Bugs: KAFKA-1476
https://issues.apache.org/jira/browse/KAFKA-1476


Repository: kafka


Description
---

KAFKA-1476 Get list of consumer groups


Diffs
-

  core/src/main/scala/kafka/tools/ConsumerCommand.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
56e3e88e0cc6d917b0ffd1254e173295c1c4aabd 

Diff: https://reviews.apache.org/r/27693/diff/


Testing
---


Thanks,

Balaji Seshadri

[jira] [Updated] (KAFKA-1476) Get a list of consumer groups

[BalajiSeshadri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BalajiSeshadri updated KAFKA-1476:
--
Attachment: KAFKA-1476.patch

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch, KAFKA-1476.patch
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

[BalajiSeshadri (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200570#comment-14200570
 ] 

BalajiSeshadri commented on KAFKA-1476:
---

Created reviewboard https://reviews.apache.org/r/27693/diff/
 against branch origin/trunk

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch, KAFKA-1476.patch
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

[BalajiSeshadri (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200575#comment-14200575
 ] 

BalajiSeshadri commented on KAFKA-1476:
---

Created review board for just this JIRA.

[~nehanarkhede] or [~junrao] or [~jkreps] Please review.

https://reviews.apache.org/r/27693/diff/

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch, KAFKA-1476.patch
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Announcing Confluent

[Jay Kreps]

Hey all,

I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a
company around Kafka called Confluent. We are planning on productizing the
kind of Kafka-based real-time data platform we built out at LinkedIn. We
are doing this because we think this is a really powerful idea and we felt
there was a lot to do to make this idea really take root. We wanted to make
that our full time mission and focus.

There is a blog post that goes into a little more depth here:
http://blog.confluent.io/

LinkedIn will remain a heavy Kafka user and contributor. Combined with our
additional resources from the funding of the company this should be a
really good thing for the Kafka development effort. Especially when
combined with the increasing contributions from the rest of the development
community. This is great news, as there is a lot of work to do. We'll need
to really focus on scaling this distributed development in a healthy way.

One thing I do want to emphasize is that the addition of a company in the
Kafka ecosystem won’t mean meddling with open source. Kafka will remain
100% open source and community focused, as of course is true of any Apache
project. I have been doing open source for a long time and strongly believe
it is the right model for infrastructure software development.

Confluent is just getting off the ground now. We left LinkedIn, raised some
money, and we have an office (but no furniture yet!). None the less, f you
are interested in finding out more about the company and either getting
help with your Kafka usage or joining us to help build all this, by all
means reach out to us, we’d love to talk.

Wish us luck!

-Jay

[jira] [Commented] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[schandr (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200603#comment-14200603
 ] 

schandr commented on KAFKA-1738:


GreatAnd thank you for the Patch. How should we apply this patch? Are there 
any instructions on how to apply the patch.

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
>Assignee: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt, 
> kafka-1738.patch
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),A

Re: Announcing Confluent

[Niek Sanders]

Congrats!

On Thu, Nov 6, 2014 at 10:28 AM, Jay Kreps  wrote:
> Hey all,
>
> I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a
> company around Kafka called Confluent. We are planning on productizing the
> kind of Kafka-based real-time data platform we built out at LinkedIn. We
> are doing this because we think this is a really powerful idea and we felt
> there was a lot to do to make this idea really take root. We wanted to make
> that our full time mission and focus.
>
> There is a blog post that goes into a little more depth here:
> http://blog.confluent.io/
>
> LinkedIn will remain a heavy Kafka user and contributor. Combined with our
> additional resources from the funding of the company this should be a
> really good thing for the Kafka development effort. Especially when
> combined with the increasing contributions from the rest of the development
> community. This is great news, as there is a lot of work to do. We'll need
> to really focus on scaling this distributed development in a healthy way.
>
> One thing I do want to emphasize is that the addition of a company in the
> Kafka ecosystem won’t mean meddling with open source. Kafka will remain
> 100% open source and community focused, as of course is true of any Apache
> project. I have been doing open source for a long time and strongly believe
> it is the right model for infrastructure software development.
>
> Confluent is just getting off the ground now. We left LinkedIn, raised some
> money, and we have an office (but no furniture yet!). None the less, f you
> are interested in finding out more about the company and either getting
> help with your Kafka usage or joining us to help build all this, by all
> means reach out to us, we’d love to talk.
>
> Wish us luck!
>
> -Jay

Re: Announcing Confluent

[Rajasekar Elango]

Congrats. Wish you all the very best and success.

Thanks,
Raja.

On Thu, Nov 6, 2014 at 1:36 PM, Niek Sanders  wrote:

> Congrats!
>
> On Thu, Nov 6, 2014 at 10:28 AM, Jay Kreps  wrote:
> > Hey all,
> >
> > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a
> > company around Kafka called Confluent. We are planning on productizing
> the
> > kind of Kafka-based real-time data platform we built out at LinkedIn. We
> > are doing this because we think this is a really powerful idea and we
> felt
> > there was a lot to do to make this idea really take root. We wanted to
> make
> > that our full time mission and focus.
> >
> > There is a blog post that goes into a little more depth here:
> > http://blog.confluent.io/
> >
> > LinkedIn will remain a heavy Kafka user and contributor. Combined with
> our
> > additional resources from the funding of the company this should be a
> > really good thing for the Kafka development effort. Especially when
> > combined with the increasing contributions from the rest of the
> development
> > community. This is great news, as there is a lot of work to do. We'll
> need
> > to really focus on scaling this distributed development in a healthy way.
> >
> > One thing I do want to emphasize is that the addition of a company in the
> > Kafka ecosystem won’t mean meddling with open source. Kafka will remain
> > 100% open source and community focused, as of course is true of any
> Apache
> > project. I have been doing open source for a long time and strongly
> believe
> > it is the right model for infrastructure software development.
> >
> > Confluent is just getting off the ground now. We left LinkedIn, raised
> some
> > money, and we have an office (but no furniture yet!). None the less, f
> you
> > are interested in finding out more about the company and either getting
> > help with your Kafka usage or joining us to help build all this, by all
> > means reach out to us, we’d love to talk.
> >
> > Wish us luck!
> >
> > -Jay
>



-- 
Thanks,
Raja.

Re: No longer supporting Java 6, if? when?

[Koert Kuipers]

when is java 6 dropped by the hadoop distros?

i am still aware of many clusters that are java 6 only at the moment.



On Thu, Nov 6, 2014 at 12:44 PM, Gwen Shapira  wrote:

> +1 for dropping Java 6
>
> On Thu, Nov 6, 2014 at 9:31 AM, Steven Schlansker <
> sschlans...@opentable.com
> > wrote:
>
> > Java 6 has been End of Life since Feb 2013.
> > Java 7 (and 8, but unfortunately that's too new still) has very
> compelling
> > features which can make development a lot easier.
> >
> > The sooner more projects drop Java 6 the better, in my opinion :)
> >
> > On Nov 5, 2014, at 7:45 PM, Worthy LaFollette  wrote:
> >
> > > Mostly converted now to 1.7, this would be welcomed to get any new
> > > features.
> > >
> > > On Wed Nov 05 2014 at 7:32:55 PM Joe Stein 
> wrote:
> > >
> > >> This has been coming up in a lot of projects and for other reasons
> too I
> > >> wanted to kick off the discussion about if/when we end support for
> Java
> > 6.
> > >> Besides any API we may want to use in >= 7 we also compile our
> binaries
> > for
> > >> 6 for release currently.
> > >>
> > >> /***
> > >> Joe Stein
> > >> Founder, Principal Consultant
> > >> Big Data Open Source Security LLC
> > >> http://www.stealth.ly
> > >> Twitter: @allthingshadoop 
> > >> /
> > >>
> >
> >
>

[jira] [Commented] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200650#comment-14200650
 ] 

Jun Rao commented on KAFKA-1738:


You can follow the instruction at 
https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review#Patchsubmissionandreview-Reviewerworkflow:
 to apply the patch.

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
>Assignee: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt, 
> kafka-1738.patch
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInfo:(Leader:0,ISR:0,Lea

Re: No longer supporting Java 6, if? when?

[Gwen Shapira]

Java6 is supported on CDH4 but not CDH5.

On Thu, Nov 6, 2014 at 9:54 AM, Koert Kuipers  wrote:

> when is java 6 dropped by the hadoop distros?
>
> i am still aware of many clusters that are java 6 only at the moment.
>
>
>
> On Thu, Nov 6, 2014 at 12:44 PM, Gwen Shapira 
> wrote:
>
> > +1 for dropping Java 6
> >
> > On Thu, Nov 6, 2014 at 9:31 AM, Steven Schlansker <
> > sschlans...@opentable.com
> > > wrote:
> >
> > > Java 6 has been End of Life since Feb 2013.
> > > Java 7 (and 8, but unfortunately that's too new still) has very
> > compelling
> > > features which can make development a lot easier.
> > >
> > > The sooner more projects drop Java 6 the better, in my opinion :)
> > >
> > > On Nov 5, 2014, at 7:45 PM, Worthy LaFollette 
> wrote:
> > >
> > > > Mostly converted now to 1.7, this would be welcomed to get any new
> > > > features.
> > > >
> > > > On Wed Nov 05 2014 at 7:32:55 PM Joe Stein 
> > wrote:
> > > >
> > > >> This has been coming up in a lot of projects and for other reasons
> > too I
> > > >> wanted to kick off the discussion about if/when we end support for
> > Java
> > > 6.
> > > >> Besides any API we may want to use in >= 7 we also compile our
> > binaries
> > > for
> > > >> 6 for release currently.
> > > >>
> > > >> /***
> > > >> Joe Stein
> > > >> Founder, Principal Consultant
> > > >> Big Data Open Source Security LLC
> > > >> http://www.stealth.ly
> > > >> Twitter: @allthingshadoop 
> > > >> /
> > > >>
> > >
> > >
> >
>

Re: No longer supporting Java 6, if? when?

[Jay Kreps]

Yeah it is a little bit silly that people are still using Java 6.

I guess this is a tradeoff--being more conservative in our java support
means more people can use our software, whereas upgrading gives us
developers a better experience since we aren't stuck with ancient stuff.

Nonetheless I would argue for being a bit conservative here. Sadly a
shocking number of people are still using Java 6. The Kafka clients get
embedded in applications all over the place, and likely having even one
application not yet upgraded would block adopting the new Kafka version
that dropped java 6 support. So unless there is something in Java 7 we
really really want I think it might be good to hold out a bit.

As an example we dropped java 6 support in Samza and immediately had people
blocked by that, and unlike the Kafka clients, Samza use is pretty
centralized.

-Jay

On Wed, Nov 5, 2014 at 5:32 PM, Joe Stein  wrote:

> This has been coming up in a lot of projects and for other reasons too I
> wanted to kick off the discussion about if/when we end support for Java 6.
> Besides any API we may want to use in >= 7 we also compile our binaries for
> 6 for release currently.
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>

Re: Announcing Confluent

[Bhavesh Mistry]

HI Guys,

Thanks for your awesome support.  I wish you good luck !!   Thanks for open
sources Kafka !!

Thanks,

Bhavesh

On Thu, Nov 6, 2014 at 10:52 AM, Rajasekar Elango 
wrote:

> Congrats. Wish you all the very best and success.
>
> Thanks,
> Raja.
>
> On Thu, Nov 6, 2014 at 1:36 PM, Niek Sanders 
> wrote:
>
> > Congrats!
> >
> > On Thu, Nov 6, 2014 at 10:28 AM, Jay Kreps  wrote:
> > > Hey all,
> > >
> > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a
> > > company around Kafka called Confluent. We are planning on productizing
> > the
> > > kind of Kafka-based real-time data platform we built out at LinkedIn.
> We
> > > are doing this because we think this is a really powerful idea and we
> > felt
> > > there was a lot to do to make this idea really take root. We wanted to
> > make
> > > that our full time mission and focus.
> > >
> > > There is a blog post that goes into a little more depth here:
> > > http://blog.confluent.io/
> > >
> > > LinkedIn will remain a heavy Kafka user and contributor. Combined with
> > our
> > > additional resources from the funding of the company this should be a
> > > really good thing for the Kafka development effort. Especially when
> > > combined with the increasing contributions from the rest of the
> > development
> > > community. This is great news, as there is a lot of work to do. We'll
> > need
> > > to really focus on scaling this distributed development in a healthy
> way.
> > >
> > > One thing I do want to emphasize is that the addition of a company in
> the
> > > Kafka ecosystem won’t mean meddling with open source. Kafka will remain
> > > 100% open source and community focused, as of course is true of any
> > Apache
> > > project. I have been doing open source for a long time and strongly
> > believe
> > > it is the right model for infrastructure software development.
> > >
> > > Confluent is just getting off the ground now. We left LinkedIn, raised
> > some
> > > money, and we have an office (but no furniture yet!). None the less, f
> > you
> > > are interested in finding out more about the company and either getting
> > > help with your Kafka usage or joining us to help build all this, by all
> > > means reach out to us, we’d love to talk.
> > >
> > > Wish us luck!
> > >
> > > -Jay
> >
>
>
>
> --
> Thanks,
> Raja.
>

Re: Review Request 27690: Patch for kafka-1738

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27690/#review60215
---

Ship it!


Ship It!

- Neha Narkhede


On Nov. 6, 2014, 5:34 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27690/
> ---
> 
> (Updated Nov. 6, 2014, 5:34 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: kafka-1738
> https://issues.apache.org/jira/browse/kafka-1738
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> try/catch should include channel.receive()
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
> 
> Diff: https://reviews.apache.org/r/27690/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

Re: Review Request 27690: Patch for kafka-1738

[Neha Narkhede]

> On Nov. 6, 2014, 7:20 p.m., Neha Narkhede wrote:
> > Ship It!

Minor nit: Could you change fails to failed?


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27690/#review60215
---


On Nov. 6, 2014, 5:34 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27690/
> ---
> 
> (Updated Nov. 6, 2014, 5:34 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: kafka-1738
> https://issues.apache.org/jira/browse/kafka-1738
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> try/catch should include channel.receive()
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
> 
> Diff: https://reviews.apache.org/r/27690/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

[jira] [Updated] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1738:
-
Reviewer: Neha Narkhede

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
>Assignee: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt, 
> kafka-1738.patch
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0)
>  to broker id:0,host:DMIPVM,port:9092. Reconnecting to broker. 
> (kafka.controller.RequestSendThread)
> java.nio.channels.Clos

[jira] [Commented] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200708#comment-14200708
 ] 

Neha Narkhede commented on KAFKA-1738:
--

Good catch. Thanks for following up on this, Jun. Reviewed the patch, looks 
good.

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
>Assignee: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt, 
> kafka-1738.patch
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0)
>  to broker id:

[jira] [Commented] (KAFKA-1738) Partitions for topic not created after restart from forced shutdown

[schandr (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200748#comment-14200748
 ] 

schandr commented on KAFKA-1738:


Will apply the patch against the 0.8.2-beta and post the update.

> Partitions for topic not created after restart from forced shutdown
> ---
>
> Key: KAFKA-1738
> URL: https://issues.apache.org/jira/browse/KAFKA-1738
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
> Environment: Linux, 2GB RAM, 2 Core CPU
>Reporter: Pradeep
>Assignee: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: 1738.zip, ServerLogForFailedTopicCreation.txt, 
> ServerLogForFailedTopicCreation.txt, ServerLogForSuccessfulTopicCreation.txt, 
> kafka-1738.patch
>
>
> We are using Kafka Topic APIs to create the topic. But in some cases, the 
> topic gets created but we don't see the partition specific files and when 
> producer/consumer tries to get the topic metadata and it fails with 
> exception. Same happens if one tries to create using the command line.
> k.p.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for 
> topic tloader1 -> No partition metadata for topic tloader1 due to 
> kafka.common.UnknownTopicOrPartitionException}] for topic [tloader1]: class 
> kafka.common.UnknownTopicOrPartitionException
> Steps to reproduce - 
> 1.  Stop kafka using kill  -9 
> 2.  Start Kafka
> 3.  Create Topic with partition and replication factor of 1.
> 4.  Check the response “Created topic ”
> 5.  Run the list command to verify if its created.
> 6.  Now check the data directory of kakfa. There would not be any for the 
> newly created topic.
> We see issues when we are creating new topics. This happens randomly and we 
> dont know the exact reasons. We see the below logs in controller during the 
> time of creation of topics which doesnt have the partition files.
> 2014-11-03 13:12:50,625] INFO [Controller 0]: New topic creation callback for 
> [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Controller 0]: New partition creation 
> callback for [JobJTopic,0] (kafka.controller.KafkaController)
> [2014-11-03 13:12:50,626] INFO [Partition state machine on Controller 0]: 
> Invoking state change to NewPartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,653] INFO [Replica state machine on controller 0]: 
> Invoking state change to NewReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,654] INFO [Partition state machine on Controller 0]: 
> Invoking state change to OnlinePartition for partitions [JobJTopic,0] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Live assigned replicas for partition [JobJTopic,0] are: [List(0)] 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,654] DEBUG [Partition state machine on Controller 0]: 
> Initializing leader and isr for partition [JobJTopic,0] to 
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2) 
> (kafka.controller.PartitionStateMachine)
> [2014-11-03 13:12:50,667] INFO [Replica state machine on controller 0]: 
> Invoking state change to OnlineReplica for replicas 
> [Topic=JobJTopic,Partition=0,Replica=0] (kafka.controller.ReplicaStateMachine)
> [2014-11-03 13:12:50,794] WARN [Controller-0-to-broker-0-send-thread], 
> Controller 0 fails to send a request to broker id:0,host:DMIPVM,port:9092 
> (kafka.controller.RequestSendThread)
> java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed.
>   at kafka.utils.Utils$.read(Utils.scala:381)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
>   at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2014-11-03 13:12:50,965] ERROR [Controller-0-to-broker-0-send-thread], 
> Controller 0 epoch 2 failed to send request 
> Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:2;CorrelationId:43;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:DMIPVM,port:9092;PartitionState:[JobJTopic,0]
>  -> 
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0)
>  to broker id:0,host:DMIPVM,port:9092. Recon

[jira] [Issue Comment Deleted] (KAFKA-1754) KOYA - Kafka on YARN

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated KAFKA-1754:

Comment: was deleted

(was: [~acmurthy] Thanks for the support. Disk reservation will benefit Kafka
greatly. Will reach out with a few other questions.

)

> KOYA - Kafka on YARN
> 
>
> Key: KAFKA-1754
> URL: https://issues.apache.org/jira/browse/KAFKA-1754
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Thomas Weise
> Attachments: DT-KOYA-Proposal- JIRA.pdf
>
>
> YARN (Hadoop 2.x) has enabled clusters to be used for a variety of workloads, 
> emerging as distributed operating system for big data applications. 
> Initiatives are on the way to bring long running services under the YARN 
> umbrella, leveraging it for centralized resource management and operations 
> ([YARN-896] and examples such as HBase, Accumulo or Memcached through 
> Slider). This JIRA is to propose KOYA (Kafka On Yarn), a YARN application 
> master to launch and manage Kafka clusters running on YARN. Brokers will use 
> resources allocated through YARN with support for recovery, monitoring etc. 
> Please see attached for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Announcing Confluent

[Steve Morin]

Jay, Neha and Jun congratz!!

On Thu, Nov 6, 2014 at 11:09 AM, Bhavesh Mistry 
wrote:

> HI Guys,
>
> Thanks for your awesome support.  I wish you good luck !!   Thanks for open
> sources Kafka !!
>
> Thanks,
>
> Bhavesh
>
> On Thu, Nov 6, 2014 at 10:52 AM, Rajasekar Elango 
> wrote:
>
> > Congrats. Wish you all the very best and success.
> >
> > Thanks,
> > Raja.
> >
> > On Thu, Nov 6, 2014 at 1:36 PM, Niek Sanders 
> > wrote:
> >
> > > Congrats!
> > >
> > > On Thu, Nov 6, 2014 at 10:28 AM, Jay Kreps 
> wrote:
> > > > Hey all,
> > > >
> > > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating
> a
> > > > company around Kafka called Confluent. We are planning on
> productizing
> > > the
> > > > kind of Kafka-based real-time data platform we built out at LinkedIn.
> > We
> > > > are doing this because we think this is a really powerful idea and we
> > > felt
> > > > there was a lot to do to make this idea really take root. We wanted
> to
> > > make
> > > > that our full time mission and focus.
> > > >
> > > > There is a blog post that goes into a little more depth here:
> > > > http://blog.confluent.io/
> > > >
> > > > LinkedIn will remain a heavy Kafka user and contributor. Combined
> with
> > > our
> > > > additional resources from the funding of the company this should be a
> > > > really good thing for the Kafka development effort. Especially when
> > > > combined with the increasing contributions from the rest of the
> > > development
> > > > community. This is great news, as there is a lot of work to do. We'll
> > > need
> > > > to really focus on scaling this distributed development in a healthy
> > way.
> > > >
> > > > One thing I do want to emphasize is that the addition of a company in
> > the
> > > > Kafka ecosystem won’t mean meddling with open source. Kafka will
> remain
> > > > 100% open source and community focused, as of course is true of any
> > > Apache
> > > > project. I have been doing open source for a long time and strongly
> > > believe
> > > > it is the right model for infrastructure software development.
> > > >
> > > > Confluent is just getting off the ground now. We left LinkedIn,
> raised
> > > some
> > > > money, and we have an office (but no furniture yet!). None the less,
> f
> > > you
> > > > are interested in finding out more about the company and either
> getting
> > > > help with your Kafka usage or joining us to help build all this, by
> all
> > > > means reach out to us, we’d love to talk.
> > > >
> > > > Wish us luck!
> > > >
> > > > -Jay
> > >
> >
> >
> >
> > --
> > Thanks,
> > Raja.
> >
>

Re: Announcing Confluent

[Joe Brown]

Best of luck!!!

J

On 6 Nov 2014, at 18:28, Jay Kreps  wrote:

> Hey all,
> 
> I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a
> company around Kafka called Confluent. We are planning on productizing the
> kind of Kafka-based real-time data platform we built out at LinkedIn. We
> are doing this because we think this is a really powerful idea and we felt
> there was a lot to do to make this idea really take root. We wanted to make
> that our full time mission and focus.
> 
> There is a blog post that goes into a little more depth here:
> http://blog.confluent.io/
> 
> LinkedIn will remain a heavy Kafka user and contributor. Combined with our
> additional resources from the funding of the company this should be a
> really good thing for the Kafka development effort. Especially when
> combined with the increasing contributions from the rest of the development
> community. This is great news, as there is a lot of work to do. We'll need
> to really focus on scaling this distributed development in a healthy way.
> 
> One thing I do want to emphasize is that the addition of a company in the
> Kafka ecosystem won’t mean meddling with open source. Kafka will remain
> 100% open source and community focused, as of course is true of any Apache
> project. I have been doing open source for a long time and strongly believe
> it is the right model for infrastructure software development.
> 
> Confluent is just getting off the ground now. We left LinkedIn, raised some
> money, and we have an office (but no furniture yet!). None the less, f you
> are interested in finding out more about the company and either getting
> help with your Kafka usage or joining us to help build all this, by all
> means reach out to us, we’d love to talk.
> 
> Wish us luck!
> 
> -Jay

[jira] [Updated] (KAFKA-1757) Can not delete Topic index on Windows

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1757:
-
Reviewer: Jay Kreps
Assignee: (was: Jay Kreps)

> Can not delete Topic index on Windows
> -
>
> Key: KAFKA-1757
> URL: https://issues.apache.org/jira/browse/KAFKA-1757
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.2
>Reporter: Lukáš Vyhlídka
>Priority: Minor
> Fix For: 0.8.2
>
> Attachments: lucky-v.patch
>
>
> When running the Kafka 0.8.2-Beta (Scala 2.10) on Windows, an attempt to 
> delete the Topic throwed an error:
> ERROR [KafkaApi-1] error when handling request Name: StopReplicaRequest; 
> Version: 0; CorrelationId: 38; ClientId: ; DeletePartitions: true; 
> ControllerId: 0; ControllerEpoch: 3; Partitions: [test,0] 
> (kafka.server.KafkaApis)
> kafka.common.KafkaStorageException: Delete of index 
> .index failed.
> at kafka.log.LogSegment.delete(LogSegment.scala:283)
> at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
> at kafka.log.Log$$anonfun$delete$1.apply(Log.scala:608)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.log.Log.delete(Log.scala:608)
> at kafka.log.LogManager.deleteLog(LogManager.scala:375)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply$mcV$sp(Partition.scala:144)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
> at 
> kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:139)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.utils.Utils$.inWriteLock(Utils.scala:543)
> at kafka.cluster.Partition.delete(Partition.scala:139)
> at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:158)
> at 
> kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:191)
> at 
> kafka.server.ReplicaManager$$anonfun$stopReplicas$3.apply(ReplicaManager.scala:190)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
> at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:190)
> at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:96)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
> at java.lang.Thread.run(Thread.java:744)
> When I have investigated the issue I figured out that the index file (in my 
> environment it was 
> C:\tmp\kafka-logs\----0014-0\.index)
>  was locked by the kafka process and the OS did not allow to delete that file.
> I tried to fix the problem in source codes and when I added close() method 
> call into LogSegment.delete(), the Topic deletion started to work.
> I will add here (not sure how to upload the file during issue creation) a 
> diff with the changes I have made so You can take a look on that whether it 
> is reasonable or not. It would be perfect if it could make it into the 
> product...
> In the end I would like to say that on Linux the deletion works just fine...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1654) Provide a way to override server configuration from command line

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201096#comment-14201096
 ] 

Jarek Jarcec Cecho commented on KAFKA-1654:
---

Looking at feedback proposal more closely [~nehanarkhede], I wanted to explain 
a bit of my thinking. My intention is to add ability to specify command line 
arguments while starting Kafka broker similarly as we do have for the command 
line tools. In this patch I'm adding one argument only ({{--set}} that will be 
renamed to {{--override}} as you've suggested), that happens to have the 
ability to be specified multiple times. That explains why I'm expecting that 
you have to specify it for every property that is being overriden and also why 
I do have the [kafka options] section without mentioning that you have to use 
{{--set}} (you might want to add different parameter in the future, like 
completely bogus {{--disable-jmx}} or whatever :)). Perhaps I should have also 
submitted a second documentation patch to cover the usage. Does the reasoning 
resonate with you or would you prefer to simply add ability to override the 
config properties without opening the code to introduce additional command line 
arguments in the future? I'll definitely address the fact that I'm not properly 
failing on error cases, that is a huge problem in my mind - thank you for 
uncovering it!

> Provide a way to override server configuration from command line
> 
>
> Key: KAFKA-1654
> URL: https://issues.apache.org/jira/browse/KAFKA-1654
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 0.8.3
>
> Attachments: KAFKA-1654.patch
>
>
> I've been recently playing with Kafka and I found the current way of server 
> configuration quite inflexible. All the configuration options have to be 
> inside a properties file and there is no way how they can be overridden for 
> execution.  In order to temporarily change one property I had to copy the 
> config file and change the property there. Hence, I'm wondering if people 
> would be open to provide a way how to specify and override the configs from 
> the command line when starting Kafka?
> Something like:
> {code}
> ./bin/kafka-server-start.sh -Dmy.cool.property=X kafka.properties
> {code}
> or 
> {code}
> ./bin/kafka-server-start.sh --set my.cool.property=X kafka.properties
> {code}
> I'm more than happy to take a stab at it, but I would like to see if there is 
> an interest for such capability?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1654) Provide a way to override server configuration from command line

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201096#comment-14201096
 ] 

Jarek Jarcec Cecho edited comment on KAFKA-1654 at 11/6/14 10:42 PM:
-

Looking at feedback proposal more closely [~nehanarkhede], I wanted to explain 
a bit of my thinking. My intention is to add ability to specify command line 
arguments while starting Kafka broker similarly as we do have for the command 
line tools. In this patch I'm adding one argument only ({{\-\-set}} that will 
be renamed to {{\-\-override}} as you've suggested), that happens to have the 
ability to be specified multiple times. That explains why I'm expecting that 
you have to specify it for every property that is being overriden and also why 
I do have the [kafka options] section without mentioning that you have to use 
{{\-\-set}} (you might want to add different parameter in the future, like 
completely bogus {{\-\-disable\-jmx}} or whatever :)). Perhaps I should have 
also submitted a second documentation patch to cover the usage. Does the 
reasoning resonate with you or would you prefer to simply add ability to 
override the config properties without opening the code to introduce additional 
command line arguments in the future? I'll definitely address the fact that I'm 
not properly failing on error cases, that is a huge problem in my mind - thank 
you for uncovering it!


was (Author: jarcec):
Looking at feedback proposal more closely [~nehanarkhede], I wanted to explain 
a bit of my thinking. My intention is to add ability to specify command line 
arguments while starting Kafka broker similarly as we do have for the command 
line tools. In this patch I'm adding one argument only ({{--set}} that will be 
renamed to {{--override}} as you've suggested), that happens to have the 
ability to be specified multiple times. That explains why I'm expecting that 
you have to specify it for every property that is being overriden and also why 
I do have the [kafka options] section without mentioning that you have to use 
{{--set}} (you might want to add different parameter in the future, like 
completely bogus {{--disable-jmx}} or whatever :)). Perhaps I should have also 
submitted a second documentation patch to cover the usage. Does the reasoning 
resonate with you or would you prefer to simply add ability to override the 
config properties without opening the code to introduce additional command line 
arguments in the future? I'll definitely address the fact that I'm not properly 
failing on error cases, that is a huge problem in my mind - thank you for 
uncovering it!

> Provide a way to override server configuration from command line
> 
>
> Key: KAFKA-1654
> URL: https://issues.apache.org/jira/browse/KAFKA-1654
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 0.8.3
>
> Attachments: KAFKA-1654.patch
>
>
> I've been recently playing with Kafka and I found the current way of server 
> configuration quite inflexible. All the configuration options have to be 
> inside a properties file and there is no way how they can be overridden for 
> execution.  In order to temporarily change one property I had to copy the 
> config file and change the property there. Hence, I'm wondering if people 
> would be open to provide a way how to specify and override the configs from 
> the command line when starting Kafka?
> Something like:
> {code}
> ./bin/kafka-server-start.sh -Dmy.cool.property=X kafka.properties
> {code}
> or 
> {code}
> ./bin/kafka-server-start.sh --set my.cool.property=X kafka.properties
> {code}
> I'm more than happy to take a stab at it, but I would like to see if there is 
> an interest for such capability?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1723) make the metrics name in new producer more standard

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201138#comment-14201138
 ] 

Jun Rao commented on KAFKA-1723:


[~otis], let me explain that a bit more. Historically, we have been using 
metrics-core to expose mbeans in both the client and the server. When we added 
the new java producer, we decided to use a metrics package of our own 
(org.apache.kafka.common.metrics) to simplify the jar dependency. The new java 
producer exposes a bunch of new mbeans (completely different from the old 
producer) using our own metrics. The naming of those mbean is in the 
description of the jira and is not completely consistent with those created by 
metrics-core (after kafka-1481 is fixed). This jira is intended to make mbean 
names created by our own metrics consistent with what's in metrics-core.

Longer term, our thinking is to eventually use our own metrics for the server 
side mbean too.

> make the metrics name in new producer more standard
> ---
>
> Key: KAFKA-1723
> URL: https://issues.apache.org/jira/browse/KAFKA-1723
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Fix For: 0.8.3
>
>
> The jmx name in the new producer looks like the following:
> kafka.producer.myclientid:type=mytopic
> However, this can be ambiguous since we allow "." in client id and topic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1753) add --decommission-broker option

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201162#comment-14201162
 ] 

Neha Narkhede commented on KAFKA-1753:
--

+1 on decommission-broker

> add --decommission-broker option
> 
>
> Key: KAFKA-1753
> URL: https://issues.apache.org/jira/browse/KAFKA-1753
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1753) add --decommission-broker option

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1753:
-
Reviewer: Neha Narkhede

> add --decommission-broker option
> 
>
> Key: KAFKA-1753
> URL: https://issues.apache.org/jira/browse/KAFKA-1753
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27391: Fix KAFKA-1634: Add the global retentionTime (in milis) and disable the per-offset timestamp

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27391/
---

(Updated Nov. 6, 2014, 11:35 p.m.)


Review request for kafka.


Bugs: KAFKA-1634
https://issues.apache.org/jira/browse/KAFKA-1634


Repository: kafka


Description
---

The timestamp field of OffsetAndMetadata is preserved since we need to be 
backward compatible with older versions


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
7517b879866fc5dad5f8d8ad30636da8bbe7784a 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java 
3ee5cbad55ce836fd04bb954dcf6ef2f9bc3288f 
  
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java 
df37fc6d8f0db0b8192a948426af603be3444da4 
  core/src/main/scala/kafka/api/OffsetCommitRequest.scala 
050615c72efe7dbaa4634f53943bd73273d20ffb 
  core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 
4cabffeacea09a49913505db19a96a55d58c0909 
  core/src/main/scala/kafka/server/KafkaApis.scala 
968b0c4f809ea9f9c3f322aef9db105f9d2e0e23 
  core/src/main/scala/kafka/server/KafkaServer.scala 
4de812374e8fb1fed834d2be3f9655f55b511a74 
  core/src/main/scala/kafka/server/OffsetManager.scala 
2957bc435102bc4004d8f100dbcdd56287c8ffae 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
cd16ced5465d098be7a60498326b2a98c248f343 
  core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
8c5364fa97da1be09973c176d1baeb339455d319 

Diff: https://reviews.apache.org/r/27391/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1634) Improve semantics of timestamp in OffsetCommitRequests and update documentation

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201211#comment-14201211
 ] 

Guozhang Wang commented on KAFKA-1634:
--

Updated reviewboard https://reviews.apache.org/r/27391/diff/
 against branch origin/trunk

> Improve semantics of timestamp in OffsetCommitRequests and update 
> documentation
> ---
>
> Key: KAFKA-1634
> URL: https://issues.apache.org/jira/browse/KAFKA-1634
> Project: Kafka
>  Issue Type: Bug
>Reporter: Neha Narkhede
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 0.8.3
>
> Attachments: KAFKA-1634.patch, KAFKA-1634_2014-11-06_15:35:46.patch
>
>
> From the mailing list -
> following up on this -- I think the online API docs for OffsetCommitRequest
> still incorrectly refer to client-side timestamps:
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommitRequest
> Wasn't that removed and now always handled server-side now?  Would one of
> the devs mind updating the API spec wiki?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1634) Improve semantics of timestamp in OffsetCommitRequests and update documentation

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1634:
-
Attachment: KAFKA-1634_2014-11-06_15:35:46.patch

> Improve semantics of timestamp in OffsetCommitRequests and update 
> documentation
> ---
>
> Key: KAFKA-1634
> URL: https://issues.apache.org/jira/browse/KAFKA-1634
> Project: Kafka
>  Issue Type: Bug
>Reporter: Neha Narkhede
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 0.8.3
>
> Attachments: KAFKA-1634.patch, KAFKA-1634_2014-11-06_15:35:46.patch
>
>
> From the mailing list -
> following up on this -- I think the online API docs for OffsetCommitRequest
> still incorrectly refer to client-side timestamps:
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommitRequest
> Wasn't that removed and now always handled server-side now?  Would one of
> the devs mind updating the API spec wiki?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1751) handle "broker not exists" scenario

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201243#comment-14201243
 ] 

Neha Narkhede commented on KAFKA-1751:
--

[~junrao] This is something I found while using the tool. See this 
https://issues.apache.org/jira/browse/KAFKA-1678?focusedCommentId=14178620&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14178620

> handle "broker not exists" scenario
> ---
>
> Key: KAFKA-1751
> URL: https://issues.apache.org/jira/browse/KAFKA-1751
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
> Attachments: kafka-1751.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1752) add --replace-broker option

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201245#comment-14201245
 ] 

Neha Narkhede commented on KAFKA-1752:
--

+1 on [~gwenshap]'s suggestion.

> add --replace-broker option
> ---
>
> Key: KAFKA-1752
> URL: https://issues.apache.org/jira/browse/KAFKA-1752
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-1481:
--
Attachment: alternateLayout2.png
alternateLayout1.png
diff-for-alternate-layout1.patch
diff-for-alternate-layout2.patch
originalLayout.png

Can you rebase? Sorry I know you have rebased a couple times already. 
Hopefully this should be the last time as these are minor comments.

KafkaMetricsGroup: 64: foreach

KafkaMetricsGroup: toMbeanName: 150/153: can you use filter { case(tagKey, 
tagValue) =>
...}


For aggregate topic metrics, since allTopics=true appears at the end it is a 
bit weird when browsing mbeans in jvisualvm/other tools. i.e., the mbean is
listed as "true". I understand why - it is just a bit weird. I'm referring 
to (for example)
kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,allTopics=true
See the attached originalLayout.png

Personally I prefer aggregate=true to allTopics=true. A further improvement 
with aggregate=true is the following: in KafkaMetricsGroup.metricName you
can check in the tags map if aggregate=true. If so, then modify the typeName 
by pre-pending Aggregate to it and then strip off the aggregate=true tag. So
you will end up with:
kafka.server:type=BrokerTopicMetrics,name=AggregateBytesOutPerSec 

See alternateLayout1.png 

Another alternative is to modify the name (not the typeName). See 
alternateLayout2.png
 
The aggregate=true approach seems generic enough to apply to any other
all-topic, all-request, or all-broker level mbeans. What do you think?


> Stop using dashes AND underscores as separators in MBean names
> --
>
> Key: KAFKA-1481
> URL: https://issues.apache.org/jira/browse/KAFKA-1481
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Otis Gospodnetic
>Priority: Critical
>  Labels: patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
> KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
> KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
> KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
> KAFKA-1481_2014-10-31_14-35-43.patch, 
> KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-03_17-02-23.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
> alternateLayout2.png, diff-for-alternate-layout1.patch, 
> diff-for-alternate-layout2.patch, originalLayout.png
>
>
> MBeans should not use dashes or underscores as separators because these 
> characters are allowed in hostnames, topics, group and consumer IDs, etc., 
> and these are embedded in MBeans names making it impossible to parse out 
> individual bits from MBeans.
> Perhaps a pipe character should be used to avoid the conflict. 
> This looks like a major blocker because it means nobody can write Kafka 0.8.x 
> monitoring tools unless they are doing it for themselves AND do not use 
> dashes AND do not use underscores.
> See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27534: Patch for KAFKA-1746

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27534/#review60276
---

Ship it!


Ship It!

- Neha Narkhede


On Nov. 3, 2014, 7:46 p.m., Ewen Cheslack-Postava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27534/
> ---
> 
> (Updated Nov. 3, 2014, 7:46 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1746
> https://issues.apache.org/jira/browse/KAFKA-1746
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1746 Make system tests return a useful exit code.
> 
> 
> KAFKA-1746 Check the exit code when running DumpLogSegments to verify data.
> 
> 
> Diffs
> -
> 
>   system_test/mirror_maker_testsuite/mirror_maker_test.py 
> c0117c64cbb7687ca8fbcec6b5c188eb880300ef 
>   system_test/offset_management_testsuite/offset_management_test.py 
> 12b5cd25140e1eb407dd57eef63d9783257688b2 
>   system_test/replication_testsuite/replica_basic_test.py 
> 660006cc253bbae3e7cd9f02601f1c1937dd1714 
>   system_test/system_test_runner.py ee7aa252333553e8eb0bc046edf968ec99dddb70 
>   system_test/utils/kafka_system_test_utils.py 
> 1093b660ebd0cb5ab6d3731d26f151e1bf717f8a 
> 
> Diff: https://reviews.apache.org/r/27534/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ewen Cheslack-Postava
> 
>

[jira] [Updated] (KAFKA-1755) Log cleaner thread should not exit on errors

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1755:
-
Labels: newbie++  (was: )

> Log cleaner thread should not exit on errors
> 
>
> Key: KAFKA-1755
> URL: https://issues.apache.org/jira/browse/KAFKA-1755
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>  Labels: newbie++
> Fix For: 0.8.3
>
>
> The log cleaner is a critical process when using compacted topics.
> However, if there is any error in any topic (notably if a key is missing) 
> then the cleaner exits and all other compacted topics will also be adversely 
> affected - i.e., compaction stops across the board.
> This can be improved by just aborting compaction for a topic on any error and 
> keep the thread from exiting.
> Another improvement would be to reject messages without keys that are sent to 
> compacted topics although this is not enough by itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27535: Patch for KAFKA-1747

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27535/#review60279
---

Ship it!


Ship It!

- Neha Narkhede


On Nov. 3, 2014, 7:46 p.m., Ewen Cheslack-Postava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27535/
> ---
> 
> (Updated Nov. 3, 2014, 7:46 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1747
> https://issues.apache.org/jira/browse/KAFKA-1747
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAKFA-1747 Fix TestcaseEnv so state isn't shared between instances.
> 
> 
> Diffs
> -
> 
>   system_test/utils/testcase_env.py b3c29105c04348f036efbbdc430e14e099ca8c70 
> 
> Diff: https://reviews.apache.org/r/27535/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ewen Cheslack-Postava
> 
>

[jira] [Commented] (KAFKA-1744) Fetch Response contains messages prior to the requested offset

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201365#comment-14201365
 ] 

Neha Narkhede commented on KAFKA-1744:
--

[~eapache] I'm assuming that you are referring to a non-java consumer right? We 
would always want to do this in the java consumer, so it's worth fixing the 
docs. Can you point me to the spec where you found this?

> Fetch Response contains messages prior to the requested offset
> --
>
> Key: KAFKA-1744
> URL: https://issues.apache.org/jira/browse/KAFKA-1744
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Evan Huus
>
> As reported in https://github.com/Shopify/sarama/issues/166 there are cases 
> where a FetchRequest for a particular offset returns some messages prior to 
> that offset.
> The spec does not seem to indicate that this is possible; it does state that 
> "As an optimization the server is allowed to return a partial message at the 
> end of the message set." but otherwise implies that a request for offset X 
> will only return complete messages starting at X. 
> The scala consumer does seem to handle this case gracefully though, if I am 
> reading it correctly (my scala is not the best): 
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsumerIterator.scala#L96-L99
> So is this a bug or just a case that needs to be added to the spec? Something 
> like "As an optimization the server is allowed to return some messages in the 
> message set prior to the requested offset. Clients should handle this case."? 
> Although I can't imagine why sending extra data would be faster than only 
> sending the necessary messages...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1744) Fetch Response contains messages prior to the requested offset

[Evan Huus (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201374#comment-14201374
 ] 

Evan Huus commented on KAFKA-1744:
--

[~nehanarkhede] this was discovered in the golang consumer I maintain - the 
scala consumer (as I linked) seems to handle this case already. I have not 
checked the java consumer.

The [spec for the fetch 
API|https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchAPI]
 implies (though it does not explicitly state) that if I perform a fetch 
request for offset X, the fetch response will contain messages whose offset is 
strictly >= X. If this is not true (in practice I have seen messages with 
offsets < X) I would suggest explicitly noting this in the spec to avoid 
confusion.

Alternatively it may be a real bug in the broker, in which case the spec is 
fine and the broker should be fixed. I don't have enough information to say for 
sure.

> Fetch Response contains messages prior to the requested offset
> --
>
> Key: KAFKA-1744
> URL: https://issues.apache.org/jira/browse/KAFKA-1744
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Evan Huus
>
> As reported in https://github.com/Shopify/sarama/issues/166 there are cases 
> where a FetchRequest for a particular offset returns some messages prior to 
> that offset.
> The spec does not seem to indicate that this is possible; it does state that 
> "As an optimization the server is allowed to return a partial message at the 
> end of the message set." but otherwise implies that a request for offset X 
> will only return complete messages starting at X. 
> The scala consumer does seem to handle this case gracefully though, if I am 
> reading it correctly (my scala is not the best): 
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsumerIterator.scala#L96-L99
> So is this a bug or just a case that needs to be added to the spec? Something 
> like "As an optimization the server is allowed to return some messages in the 
> message set prior to the requested offset. Clients should handle this case."? 
> Although I can't imagine why sending extra data would be faster than only 
> sending the necessary messages...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1741) consumer get always old messages

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede resolved KAFKA-1741.
--
Resolution: Won't Fix

> consumer get always old messages
> 
>
> Key: KAFKA-1741
> URL: https://issues.apache.org/jira/browse/KAFKA-1741
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.1.1, 0.8.2
>Reporter: hamza ezzi
>Assignee: Neha Narkhede
>
> every time when a consumer get a message, i have this error, and when i 
> restart consumer i get old message knowing i specified in my consumer config 
> to do not get old message 
> my nodejs consumer code :
> var kafka = require('kafka-node');
> var HighLevelConsumer = kafka.HighLevelConsumer;
> var Offset = kafka.Offset;
> var Client = kafka.Client;
> var argv = require('optimist').argv;
> var topic = argv.topic || 'sLNzXYHLJA';
> var client = new Client('XXX.XXX.XXX:2181','consumer'+process.pid);
> var payloads = [{topic:topic}];
> var options = {
> groupId: 'kafka-node-group',
> // Auto commit config
> autoCommit: true,
> autoCommitMsgCount: 100,
> autoCommitIntervalMs: 5000,
> // Fetch message config
> fetchMaxWaitMs: 100,
> fetchMinBytes: 1,
> fetchMaxBytes: 1024 * 10,
> fromOffset: false,
> fromBeginning: false
> };
> var consumer = new HighLevelConsumer(client, payloads, options);
> var offset = new Offset(client);
> consumer.on('message', function (message) {
> console.log(this.id, message);
> });
> consumer.on('error', function (err) {
> console.log('error', err);
> });
> consumer.on('offsetOutOfRange', function (topic) {
> console.log("- offsetOutOfRange ");
> topic.maxNum = 2;
> offset.fetch([topic], function (err, offsets) {
> var min = Math.min.apply(null, offsets[topic.topic][topic.partition]);
> consumer.setOffset(topic.topic, topic.partition, min);
> });
> });
> error kafka log :
> [2014-10-31 17:13:32,173] ERROR Closing socket for /212.XXX.XXX.XXX because 
> of error (kafka.network.Processor)
> java.nio.BufferUnderflowException
>   at java.nio.Buffer.nextGetIndex(Buffer.java:498)
>   at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:406)
>   at 
> kafka.api.OffsetCommitRequest$$anonfun$1$$anonfun$apply$1.apply(OffsetCommitRequest.scala:62)
>   at 
> kafka.api.OffsetCommitRequest$$anonfun$1$$anonfun$apply$1.apply(OffsetCommitRequest.scala:58)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at 
> kafka.api.OffsetCommitRequest$$anonfun$1.apply(OffsetCommitRequest.scala:58)
>   at 
> kafka.api.OffsetCommitRequest$$anonfun$1.apply(OffsetCommitRequest.scala:55)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at 
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
>   at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
>   at kafka.api.OffsetCommitRequest$.readFrom(OffsetCommitRequest.scala:55)
>   at kafka.api.RequestKeys$$anonfun$9.apply(RequestKeys.scala:47)
>   at kafka.api.RequestKeys$$anonfun$9.apply(RequestKeys.scala:47)
>   at kafka.network.RequestChannel$Request.(RequestChannel.scala:50)
>   at kafka.network.Processor.read(SocketServer.scala:450)
>   at kafka.network.Processor.run(SocketServer.scala:340)
>   at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1741) consumer get always old messages

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201391#comment-14201391
 ] 

Neha Narkhede commented on KAFKA-1741:
--

It seems like the format of the OffsetCommitRequest you are using is different 
from what the server expects. See the format explained here - 
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommitRequest.
 Best if we take such questions to the mailing list though.

> consumer get always old messages
> 
>
> Key: KAFKA-1741
> URL: https://issues.apache.org/jira/browse/KAFKA-1741
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.1.1, 0.8.2
>Reporter: hamza ezzi
>Assignee: Neha Narkhede
>
> every time when a consumer get a message, i have this error, and when i 
> restart consumer i get old message knowing i specified in my consumer config 
> to do not get old message 
> my nodejs consumer code :
> var kafka = require('kafka-node');
> var HighLevelConsumer = kafka.HighLevelConsumer;
> var Offset = kafka.Offset;
> var Client = kafka.Client;
> var argv = require('optimist').argv;
> var topic = argv.topic || 'sLNzXYHLJA';
> var client = new Client('XXX.XXX.XXX:2181','consumer'+process.pid);
> var payloads = [{topic:topic}];
> var options = {
> groupId: 'kafka-node-group',
> // Auto commit config
> autoCommit: true,
> autoCommitMsgCount: 100,
> autoCommitIntervalMs: 5000,
> // Fetch message config
> fetchMaxWaitMs: 100,
> fetchMinBytes: 1,
> fetchMaxBytes: 1024 * 10,
> fromOffset: false,
> fromBeginning: false
> };
> var consumer = new HighLevelConsumer(client, payloads, options);
> var offset = new Offset(client);
> consumer.on('message', function (message) {
> console.log(this.id, message);
> });
> consumer.on('error', function (err) {
> console.log('error', err);
> });
> consumer.on('offsetOutOfRange', function (topic) {
> console.log("- offsetOutOfRange ");
> topic.maxNum = 2;
> offset.fetch([topic], function (err, offsets) {
> var min = Math.min.apply(null, offsets[topic.topic][topic.partition]);
> consumer.setOffset(topic.topic, topic.partition, min);
> });
> });
> error kafka log :
> [2014-10-31 17:13:32,173] ERROR Closing socket for /212.XXX.XXX.XXX because 
> of error (kafka.network.Processor)
> java.nio.BufferUnderflowException
>   at java.nio.Buffer.nextGetIndex(Buffer.java:498)
>   at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:406)
>   at 
> kafka.api.OffsetCommitRequest$$anonfun$1$$anonfun$apply$1.apply(OffsetCommitRequest.scala:62)
>   at 
> kafka.api.OffsetCommitRequest$$anonfun$1$$anonfun$apply$1.apply(OffsetCommitRequest.scala:58)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at 
> kafka.api.OffsetCommitRequest$$anonfun$1.apply(OffsetCommitRequest.scala:58)
>   at 
> kafka.api.OffsetCommitRequest$$anonfun$1.apply(OffsetCommitRequest.scala:55)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at 
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
>   at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
>   at kafka.api.OffsetCommitRequest$.readFrom(OffsetCommitRequest.scala:55)
>   at kafka.api.RequestKeys$$anonfun$9.apply(RequestKeys.scala:47)
>   at kafka.api.RequestKeys$$anonfun$9.apply(RequestKeys.scala:47)
>   at kafka.network.RequestChannel$Request.(RequestChannel.scala:50)
>   at kafka.network.Processor.read(SocketServer.scala:450)
>   at kafka.network.Processor.run(SocketServer.scala:340)
>   at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1744) Fetch Response contains messages prior to the requested offset

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201394#comment-14201394
 ] 

Neha Narkhede commented on KAFKA-1744:
--

The broker sends data to the consumer using zero-copy, so it cannot filter the 
extra messages out. The spec already says Clients should handle this case. 
Should we close this JIRA?

> Fetch Response contains messages prior to the requested offset
> --
>
> Key: KAFKA-1744
> URL: https://issues.apache.org/jira/browse/KAFKA-1744
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Evan Huus
>
> As reported in https://github.com/Shopify/sarama/issues/166 there are cases 
> where a FetchRequest for a particular offset returns some messages prior to 
> that offset.
> The spec does not seem to indicate that this is possible; it does state that 
> "As an optimization the server is allowed to return a partial message at the 
> end of the message set." but otherwise implies that a request for offset X 
> will only return complete messages starting at X. 
> The scala consumer does seem to handle this case gracefully though, if I am 
> reading it correctly (my scala is not the best): 
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsumerIterator.scala#L96-L99
> So is this a bug or just a case that needs to be added to the spec? Something 
> like "As an optimization the server is allowed to return some messages in the 
> message set prior to the requested offset. Clients should handle this case."? 
> Although I can't imagine why sending extra data would be faster than only 
> sending the necessary messages...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1695) Authenticate connection to Zookeeper

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201393#comment-14201393
 ] 

Gwen Shapira commented on KAFKA-1695:
-

The pull request is in!

I think it makes sense to open separate JIRA for upgrading zkclient?

> Authenticate connection to Zookeeper
> 
>
> Key: KAFKA-1695
> URL: https://issues.apache.org/jira/browse/KAFKA-1695
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jay Kreps
>Assignee: Gwen Shapira
>
> We need to make it possible to secure the Zookeeper cluster Kafka is using. 
> This would make use of the normal authentication ZooKeeper provides. 
> ZooKeeper supports a variety of authentication mechanisms so we will need to 
> figure out what has to be passed in to the zookeeper client.
> The intention is that when the current round of client work is done it should 
> be possible to run without clients needing access to Zookeeper so all we need 
> here is to make it so that only the Kafka cluster is able to read and write 
> to the Kafka znodes  (we shouldn't need to set any kind of acl on a per-znode 
> basis).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1756) never allow the replica fetch size to be less than the max message size

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201404#comment-14201404
 ] 

Jun Rao commented on KAFKA-1756:


Joe,

The fix for 1) is a bit hard to do. kafka-topics just creates some paths in ZK 
and it doesn't have access to the default broker side configs. Once the topic 
and its config are created in ZK, it's hard to back in out in the broker. For 
2), we are already doing this check.

Gwen,

What you mentioned is the benefit for any topic level config. My point is that 
we probably shouldn't make max.message.size a topic level config given the 
implication to downstream consumers such as mirrormaker and the replica 
fetchers.

> never allow the replica fetch size to be less than the max message size
> ---
>
> Key: KAFKA-1756
> URL: https://issues.apache.org/jira/browse/KAFKA-1756
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1, 0.8.2
>Reporter: Joe Stein
>Priority: Blocker
> Fix For: 0.8.2
>
>
> There exists a very hazardous scenario where if the max.message.bytes is 
> greather than the replica.fetch.max.bytes the message will never replicate. 
> This will bring the ISR down to 1 (eventually/quickly once 
> replica.lag.max.messages is reached). If during this window the leader itself 
> goes out of the ISR then the new leader will commit the last offset it 
> replicated. This is also bad for sync producers with -1 ack because they will 
> all block (heard affect caused upstream) in this scenario too.
> The fix here is two fold
> 1) when setting max.message.bytes using kafka-topics we must check first each 
> and every broker (which will need some thought about how todo this because of 
> the topiccommand zk notification) that max.message.bytes <= 
> replica.fetch.max.bytes and if it is NOT then DO NOT create the topic
> 2) if you change this in server.properties then the broker should not start 
> if max.message.bytes > replica.fetch.max.bytes
> This does beg the question/issue some about centralizing certain/some/all 
> configurations so that inconsistencies do not occur (where broker 1 has 
> max.message.bytes > replica.fetch.max.bytes but broker 2 max.message.bytes <= 
> replica.fetch.max.bytes because of error in properties). I do not want to 
> conflate this ticket but I think it is worth mentioning/bringing up here as 
> it is a good example where it could make sense. 
> I set this as BLOCKER for 0.8.2-beta because we did so much work to enable 
> consistency vs availability and 0 data loss this corner case should be part 
> of 0.8.2-final
> Also, I could go one step further (though I would not consider this part as a 
> blocker for 0.8.2 but interested to what other folks think) about a consumer 
> replica fetch size so that if the message max is increased messages will no 
> longer be consumed (since the consumer fetch max would be <  max.message.bytes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1695) Authenticate connection to Zookeeper

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201419#comment-14201419
 ] 

Neha Narkhede commented on KAFKA-1695:
--

bq. I think it makes sense to open separate JIRA for upgrading zkclient?

Yup. That'll be great

> Authenticate connection to Zookeeper
> 
>
> Key: KAFKA-1695
> URL: https://issues.apache.org/jira/browse/KAFKA-1695
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jay Kreps
>Assignee: Gwen Shapira
>
> We need to make it possible to secure the Zookeeper cluster Kafka is using. 
> This would make use of the normal authentication ZooKeeper provides. 
> ZooKeeper supports a variety of authentication mechanisms so we will need to 
> figure out what has to be passed in to the zookeeper client.
> The intention is that when the current round of client work is done it should 
> be possible to run without clients needing access to Zookeeper so all we need 
> here is to make it so that only the Kafka cluster is able to read and write 
> to the Kafka znodes  (we shouldn't need to set any kind of acl on a per-znode 
> basis).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1744) Fetch Response contains messages prior to the requested offset

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201424#comment-14201424
 ] 

Jun Rao commented on KAFKA-1744:


Evan,

I added the following explanation to the wiki.

"In general, the return messages will have offsets larger than or equal to the 
starting offset. However, with compressed messages, it's possible for the 
returned messages to have offsets smaller than the starting offset. The number 
of such messages is typically small and the caller is responsible for filter 
out those messages."

> Fetch Response contains messages prior to the requested offset
> --
>
> Key: KAFKA-1744
> URL: https://issues.apache.org/jira/browse/KAFKA-1744
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Evan Huus
>
> As reported in https://github.com/Shopify/sarama/issues/166 there are cases 
> where a FetchRequest for a particular offset returns some messages prior to 
> that offset.
> The spec does not seem to indicate that this is possible; it does state that 
> "As an optimization the server is allowed to return a partial message at the 
> end of the message set." but otherwise implies that a request for offset X 
> will only return complete messages starting at X. 
> The scala consumer does seem to handle this case gracefully though, if I am 
> reading it correctly (my scala is not the best): 
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsumerIterator.scala#L96-L99
> So is this a bug or just a case that needs to be added to the spec? Something 
> like "As an optimization the server is allowed to return some messages in the 
> message set prior to the requested offset. Clients should handle this case."? 
> Although I can't imagine why sending extra data would be faster than only 
> sending the necessary messages...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1744) Fetch Response contains messages prior to the requested offset

[Evan Huus (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201425#comment-14201425
 ] 

Evan Huus commented on KAFKA-1744:
--

[~junrao] very nice stealth edit of the spec, thank you :)

That clarification is what I was looking for, this ticket can be closed.

> Fetch Response contains messages prior to the requested offset
> --
>
> Key: KAFKA-1744
> URL: https://issues.apache.org/jira/browse/KAFKA-1744
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Evan Huus
>
> As reported in https://github.com/Shopify/sarama/issues/166 there are cases 
> where a FetchRequest for a particular offset returns some messages prior to 
> that offset.
> The spec does not seem to indicate that this is possible; it does state that 
> "As an optimization the server is allowed to return a partial message at the 
> end of the message set." but otherwise implies that a request for offset X 
> will only return complete messages starting at X. 
> The scala consumer does seem to handle this case gracefully though, if I am 
> reading it correctly (my scala is not the best): 
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsumerIterator.scala#L96-L99
> So is this a bug or just a case that needs to be added to the spec? Something 
> like "As an optimization the server is allowed to return some messages in the 
> message set prior to the requested offset. Clients should handle this case."? 
> Although I can't imagine why sending extra data would be faster than only 
> sending the necessary messages...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1744) Fetch Response contains messages prior to the requested offset

[Evan Huus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Evan Huus resolved KAFKA-1744.
--
Resolution: Fixed

> Fetch Response contains messages prior to the requested offset
> --
>
> Key: KAFKA-1744
> URL: https://issues.apache.org/jira/browse/KAFKA-1744
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Evan Huus
>
> As reported in https://github.com/Shopify/sarama/issues/166 there are cases 
> where a FetchRequest for a particular offset returns some messages prior to 
> that offset.
> The spec does not seem to indicate that this is possible; it does state that 
> "As an optimization the server is allowed to return a partial message at the 
> end of the message set." but otherwise implies that a request for offset X 
> will only return complete messages starting at X. 
> The scala consumer does seem to handle this case gracefully though, if I am 
> reading it correctly (my scala is not the best): 
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsumerIterator.scala#L96-L99
> So is this a bug or just a case that needs to be added to the spec? Something 
> like "As an optimization the server is allowed to return some messages in the 
> message set prior to the requested offset. Clients should handle this case."? 
> Although I can't imagine why sending extra data would be faster than only 
> sending the necessary messages...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1744) Fetch Response contains messages prior to the requested offset

[Evan Huus (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201425#comment-14201425
 ] 

Evan Huus edited comment on KAFKA-1744 at 11/7/14 2:00 AM:
---

[~junrao] thanks for the clarification, that's what I was looking for, this 
ticket can be closed.


was (Author: eapache):
[~junrao] very nice stealth edit of the spec, thank you :)

That clarification is what I was looking for, this ticket can be closed.

> Fetch Response contains messages prior to the requested offset
> --
>
> Key: KAFKA-1744
> URL: https://issues.apache.org/jira/browse/KAFKA-1744
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Evan Huus
>
> As reported in https://github.com/Shopify/sarama/issues/166 there are cases 
> where a FetchRequest for a particular offset returns some messages prior to 
> that offset.
> The spec does not seem to indicate that this is possible; it does state that 
> "As an optimization the server is allowed to return a partial message at the 
> end of the message set." but otherwise implies that a request for offset X 
> will only return complete messages starting at X. 
> The scala consumer does seem to handle this case gracefully though, if I am 
> reading it correctly (my scala is not the best): 
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsumerIterator.scala#L96-L99
> So is this a bug or just a case that needs to be added to the spec? Something 
> like "As an optimization the server is allowed to return some messages in the 
> message set prior to the requested offset. Clients should handle this case."? 
> Although I can't imagine why sending extra data would be faster than only 
> sending the necessary messages...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1747) TestcaseEnv improperly shares state between instances

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1747:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks for the patch, Ewen! Pushed to trunk

> TestcaseEnv improperly shares state between instances
> -
>
> Key: KAFKA-1747
> URL: https://issues.apache.org/jira/browse/KAFKA-1747
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Attachments: KAFKA-1747.patch
>
>
> TestcaseEnv in system tests uses class variables instead of instance 
> variables for a bunch of state. This causes the data to persist between 
> tests. In some cases this can cause tests to break (e.g. there will be state 
> from a service running in a previous test that doesn't exist in the current 
> test; trying to look up state about that service raises an exception or 
> produces invalid data).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1746) System tests don't handle errors well

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1746:
-
Component/s: system tests

> System tests don't handle errors well
> -
>
> Key: KAFKA-1746
> URL: https://issues.apache.org/jira/browse/KAFKA-1746
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 0.8.1.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-1746.patch
>
>
> The system test scripts don't handle errors well. A couple of key issues:
> * Unexpected exceptions during tests are just ignored and the tests appear to 
> be successful in the reports.
> * The scripts exit code is always 0, even if tests fail.
> * Almost no subprocess calls are checked. In a lot of cases this is ok, and 
> sometimes it's not possible (e.g. after starting a long-running remote 
> process), but in some cases such as calls to DumpLogSegments, the tests can 
> miss that the tools is exiting with an exception and the test appears to be 
> successful even though no data was verified.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1746) System tests don't handle errors well

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1746:
-
Fix Version/s: 0.8.3

> System tests don't handle errors well
> -
>
> Key: KAFKA-1746
> URL: https://issues.apache.org/jira/browse/KAFKA-1746
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 0.8.1.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-1746.patch
>
>
> The system test scripts don't handle errors well. A couple of key issues:
> * Unexpected exceptions during tests are just ignored and the tests appear to 
> be successful in the reports.
> * The scripts exit code is always 0, even if tests fail.
> * Almost no subprocess calls are checked. In a lot of cases this is ok, and 
> sometimes it's not possible (e.g. after starting a long-running remote 
> process), but in some cases such as calls to DumpLogSegments, the tests can 
> miss that the tools is exiting with an exception and the test appears to be 
> successful even though no data was verified.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1747) TestcaseEnv improperly shares state between instances

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1747:
-
Fix Version/s: 0.8.3

> TestcaseEnv improperly shares state between instances
> -
>
> Key: KAFKA-1747
> URL: https://issues.apache.org/jira/browse/KAFKA-1747
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 0.8.1.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-1747.patch
>
>
> TestcaseEnv in system tests uses class variables instead of instance 
> variables for a bunch of state. This causes the data to persist between 
> tests. In some cases this can cause tests to break (e.g. there will be state 
> from a service running in a previous test that doesn't exist in the current 
> test; trying to look up state about that service raises an exception or 
> produces invalid data).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1747) TestcaseEnv improperly shares state between instances

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1747:
-
Component/s: system tests

> TestcaseEnv improperly shares state between instances
> -
>
> Key: KAFKA-1747
> URL: https://issues.apache.org/jira/browse/KAFKA-1747
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 0.8.1.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-1747.patch
>
>
> TestcaseEnv in system tests uses class variables instead of instance 
> variables for a bunch of state. This causes the data to persist between 
> tests. In some cases this can cause tests to break (e.g. there will be state 
> from a service running in a previous test that doesn't exist in the current 
> test; trying to look up state about that service raises an exception or 
> produces invalid data).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1746) System tests don't handle errors well

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1746:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks for the patch! Pushed to trunk

> System tests don't handle errors well
> -
>
> Key: KAFKA-1746
> URL: https://issues.apache.org/jira/browse/KAFKA-1746
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 0.8.1.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-1746.patch
>
>
> The system test scripts don't handle errors well. A couple of key issues:
> * Unexpected exceptions during tests are just ignored and the tests appear to 
> be successful in the reports.
> * The scripts exit code is always 0, even if tests fail.
> * Almost no subprocess calls are checked. In a lot of cases this is ok, and 
> sometimes it's not possible (e.g. after starting a long-running remote 
> process), but in some cases such as calls to DumpLogSegments, the tests can 
> miss that the tools is exiting with an exception and the test appears to be 
> successful even though no data was verified.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1756) never allow the replica fetch size to be less than the max message size

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201449#comment-14201449
 ] 

Gwen Shapira commented on KAFKA-1756:
-

I wonder if it makes sense for the brokers to persist their configs in their 
respective ZK nodes, so we can validate topic configs against broker configs?

> never allow the replica fetch size to be less than the max message size
> ---
>
> Key: KAFKA-1756
> URL: https://issues.apache.org/jira/browse/KAFKA-1756
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1, 0.8.2
>Reporter: Joe Stein
>Priority: Blocker
> Fix For: 0.8.2
>
>
> There exists a very hazardous scenario where if the max.message.bytes is 
> greather than the replica.fetch.max.bytes the message will never replicate. 
> This will bring the ISR down to 1 (eventually/quickly once 
> replica.lag.max.messages is reached). If during this window the leader itself 
> goes out of the ISR then the new leader will commit the last offset it 
> replicated. This is also bad for sync producers with -1 ack because they will 
> all block (heard affect caused upstream) in this scenario too.
> The fix here is two fold
> 1) when setting max.message.bytes using kafka-topics we must check first each 
> and every broker (which will need some thought about how todo this because of 
> the topiccommand zk notification) that max.message.bytes <= 
> replica.fetch.max.bytes and if it is NOT then DO NOT create the topic
> 2) if you change this in server.properties then the broker should not start 
> if max.message.bytes > replica.fetch.max.bytes
> This does beg the question/issue some about centralizing certain/some/all 
> configurations so that inconsistencies do not occur (where broker 1 has 
> max.message.bytes > replica.fetch.max.bytes but broker 2 max.message.bytes <= 
> replica.fetch.max.bytes because of error in properties). I do not want to 
> conflate this ticket but I think it is worth mentioning/bringing up here as 
> it is a good example where it could make sense. 
> I set this as BLOCKER for 0.8.2-beta because we did so much work to enable 
> consistency vs availability and 0 data loss this corner case should be part 
> of 0.8.2-final
> Also, I could go one step further (though I would not consider this part as a 
> blocker for 0.8.2 but interested to what other folks think) about a consumer 
> replica fetch size so that if the message max is increased messages will no 
> longer be consumed (since the consumer fetch max would be <  max.message.bytes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1282) Disconnect idle socket connection in Selector

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201475#comment-14201475
 ] 

Jun Rao commented on KAFKA-1282:


Nicu,

I was doing some manual testing of this feature. What I observed is that 
sometimes, the idle connections are not closed. The following was what I did.

1. Configure a small connections.max.idle.ms = 1.
2. start ZK and Kafka broker
3. start a console consumer
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic topic1 
--from-beginning
4. start a console producer and type in sth every 15 secs or so. 
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic1 
--request-required-acks 1

What I observed was that initially, the producer connections kept getting 
killed by the broker correctly after being idle for 10 secs. The next producer 
send would hit an IOException and trigger a resend. However, after typing in 10 
or so messages, at some point, no idle connections were killed by the broker 
any more and the producer send always succeeded.

> Disconnect idle socket connection in Selector
> -
>
> Key: KAFKA-1282
> URL: https://issues.apache.org/jira/browse/KAFKA-1282
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: nicu marasoiu
>  Labels: newbie++
> Fix For: 0.9.0
>
> Attachments: 1282_brush.patch, 1282_brushed_up.patch, 
> KAFKA-1282_Disconnect_idle_socket_connection_in_Selector.patch
>
>
> To reduce # socket connections, it would be useful for the new producer to 
> close socket connections that are idle. We can introduce a new producer 
> config for the idle time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27391: Fix KAFKA-1634: Add the global retentionTime (in milis) and disable the per-offset timestamp

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27391/#review60285
---



clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java


@Deprecated



clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java


This confused me a bit, and I think it is because initCommonFields is 
intended to initialize fields common to all versions of the request. It is a 
useful helper method but it becomes somewhat clunky when removing fields. The 
partition-level timestamp is no longer a common field.

If this is v2 then we should _never_ set anything in the timestamp field of 
the struct; and if it is < v2 then we should _always_ set the timestamp field 
(even if it is the default). However, since the timestamp field in the Field 
declaration for OFFSET_COMMIT_REQUEST_PARTITION_V0 does not have a default 
explicitly specified, I think this will break with a SchemaException("missing 
value...") for offset commit request v0, v1 if we choose to write to a 
bytebuffer under those versions with this code.

One option is to explicitly pass in the constructor version (0, 1, 2) to 
initCommonFields and use that to decide whether to include/exclude this field, 
but that is weird. Another alternative is a separate helper method for v0v1. 
That is even weirder.



clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java


Would help to add a comment "This field only exists in v0 and v1"



core/src/main/scala/kafka/common/OffsetMetadataAndError.scala


Can we mark this @Deprecated as well?

We should probably make the primary constructor without timestamp and add a 
secondary constructor with timestamp and mark deprecated there.

Also, can we use case class.copy if timestamp needs to be modified? 
However, per comment further down I don't think it needs to be touched.



core/src/main/scala/kafka/server/OffsetManager.scala


This was already there, but it would be clearer to use:

filter { case (..., ...) => 
}



core/src/main/scala/kafka/server/OffsetManager.scala


Found %d expired offsets.



core/src/main/scala/kafka/server/OffsetManager.scala


Actually, since we always set the retention period (for v0, v1) in 
KafkaApis do we need to even touch this timestamp? i.e., we should basically 
ignore it right? So we only need to do:
value.set(VALUE_TIMESTAMP_FIELD, expirationTimestamp).


- Joel Koshy


On Nov. 6, 2014, 11:35 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27391/
> ---
> 
> (Updated Nov. 6, 2014, 11:35 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1634
> https://issues.apache.org/jira/browse/KAFKA-1634
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> The timestamp field of OffsetAndMetadata is preserved since we need to be 
> backward compatible with older versions
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
> 7517b879866fc5dad5f8d8ad30636da8bbe7784a 
>   
> clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java
>  3ee5cbad55ce836fd04bb954dcf6ef2f9bc3288f 
>   
> clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java
>  df37fc6d8f0db0b8192a948426af603be3444da4 
>   core/src/main/scala/kafka/api/OffsetCommitRequest.scala 
> 050615c72efe7dbaa4634f53943bd73273d20ffb 
>   core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 
> 4cabffeacea09a49913505db19a96a55d58c0909 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 968b0c4f809ea9f9c3f322aef9db105f9d2e0e23 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 4de812374e8fb1fed834d2be3f9655f55b511a74 
>   core/src/main/scala/kafka/server/OffsetManager.scala 
> 2957bc435102bc4004d8f100dbcdd56287c8ffae 
>   core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
> cd16ced5465d098be7a60498326b2a98c248f343 
>   core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
> 8c5364fa97da1be09973c176d1baeb339455d319 
> 
> Diff: https://reviews.apache.org/r/27391/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Jenkins build is back to normal : Kafka-trunk #324

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-1758) corrupt recovery file prevents startup

[Jason Rosenberg (JIRA)]

Jason Rosenberg created KAFKA-1758:
--

 Summary: corrupt recovery file prevents startup
 Key: KAFKA-1758
 URL: https://issues.apache.org/jira/browse/KAFKA-1758
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Rosenberg


Hi,

We recently had a kafka node go down suddenly. When it came back up, it 
apparently had a corrupt recovery file, and refused to startup:

{code}
2014-11-06 08:17:19,299  WARN [main] server.KafkaServer - Error starting up 
KafkaServer
java.lang.NumberFormatException: For input string: 
"^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@"
at 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:481)
at java.lang.Integer.parseInt(Integer.java:527)
at 
scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
at kafka.server.OffsetCheckpoint.read(OffsetCheckpoint.scala:76)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:106)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
{code}

And the app is under a monitor (so it was repeatedly restarting and failing 
with this error for several minutes before we got to it)…

We moved the ‘recovery-point-offset-checkpoint’ file out of the way, and it 
then restarted cleanly (but of course re-synced all it’s data from replicas, so 
we had no data loss).

Anyway, I’m wondering if that’s the expected behavior? Or should it not declare 
it corrupt and then proceed automatically to an unclean restart?

Should this NumberFormatException be handled a bit more gracefully?

We saved the corrupt file if it’s worth inspecting (although I doubt it will be 
useful!)….

The corrupt files appeared to be all zeroes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27634: Patch for KAFKA-1667

[Gwen Shapira]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27634/#review60188
---


Awesome. Its building successfully now.


core/src/main/scala/kafka/utils/Utils.scala


I think I'm not seeing why we need this. 
Shouldn't Scala's JavaConversion class handle this exact case?



core/src/test/scala/kafka/log/LogConfigTest.scala


Can you add a test that shows that we indeed fail validation when creating 
LogConfig with invalid values?



core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala


String.valueOf(false) should evaluate to "false", right? Why do we need 
"nottrue"? 

If String.valueOf(false) no longer works as expected, it looks like a bug 
waiting to happen...



core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala


why are we removing a test?


- Gwen Shapira


On Nov. 6, 2014, 4:12 p.m., Dmytro Kostiuchenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27634/
> ---
> 
> (Updated Nov. 6, 2014, 4:12 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1667
> https://issues.apache.org/jira/browse/KAFKA-1667
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1667 Fixed bugs in LogConfig. Added test and documentation
> 
> 
> KAFKA-1667 Updated tests to reflect new boolean property parsing logic
> 
> 
> KAFKA-1667 renamed methods to match naming convention
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java 
> c4cea2cc072f4db4ce014b63d226431d3766bef1 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 0b2735e7fc42ef9894bef1997b1f06a8ebee5439 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> e48922a97727dd0b98f3ae630ebb0af3bef2373d 
>   core/src/main/scala/kafka/utils/Utils.scala 
> 23aefb4715b177feae1d2f83e8b910653ea10c5f 
>   core/src/test/scala/kafka/log/LogConfigTest.scala PRE-CREATION 
>   core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala 
> f44568cb25edf25db857415119018fd4c9922f61 
> 
> Diff: https://reviews.apache.org/r/27634/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dmytro Kostiuchenko
> 
>

Re: Announcing Confluent

[pankaj ojha]

Best of Luck..keep rocking...

On Fri, Nov 7, 2014 at 1:30 AM, Joe Brown  wrote:

> Best of luck!!!
>
> J
>
> On 6 Nov 2014, at 18:28, Jay Kreps  wrote:
>
> > Hey all,
> >
> > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a
> > company around Kafka called Confluent. We are planning on productizing
> the
> > kind of Kafka-based real-time data platform we built out at LinkedIn. We
> > are doing this because we think this is a really powerful idea and we
> felt
> > there was a lot to do to make this idea really take root. We wanted to
> make
> > that our full time mission and focus.
> >
> > There is a blog post that goes into a little more depth here:
> > http://blog.confluent.io/
> >
> > LinkedIn will remain a heavy Kafka user and contributor. Combined with
> our
> > additional resources from the funding of the company this should be a
> > really good thing for the Kafka development effort. Especially when
> > combined with the increasing contributions from the rest of the
> development
> > community. This is great news, as there is a lot of work to do. We'll
> need
> > to really focus on scaling this distributed development in a healthy way.
> >
> > One thing I do want to emphasize is that the addition of a company in the
> > Kafka ecosystem won’t mean meddling with open source. Kafka will remain
> > 100% open source and community focused, as of course is true of any
> Apache
> > project. I have been doing open source for a long time and strongly
> believe
> > it is the right model for infrastructure software development.
> >
> > Confluent is just getting off the ground now. We left LinkedIn, raised
> some
> > money, and we have an office (but no furniture yet!). None the less, f
> you
> > are interested in finding out more about the company and either getting
> > help with your Kafka usage or joining us to help build all this, by all
> > means reach out to us, we’d love to talk.
> >
> > Wish us luck!
> >
> > -Jay
>
>


-- 
Thanks,
Pankaj Ojha

[jira] [Updated] (KAFKA-1742) ControllerContext removeTopic does not correctly update state

[Onur Karaman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Onur Karaman updated KAFKA-1742:

Attachment: KAFKA-1742.patch

> ControllerContext removeTopic does not correctly update state
> -
>
> Key: KAFKA-1742
> URL: https://issues.apache.org/jira/browse/KAFKA-1742
> Project: Kafka
>  Issue Type: Bug
>Reporter: Onur Karaman
>Assignee: Onur Karaman
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1742.patch
>
>
> removeTopic does not correctly update the state of ControllerContext.
> This is because it removes the topic from some underlying maps through 
> dropWhile.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)