[jira] [Commented] (KAFKA-1196) java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872210#comment-13872210
 ] 

Jun Rao commented on KAFKA-1196:


Yes, I wasn't thinking of fixing the issue since I can't think of an easy way. 
I was just thinking of giving the client the right error message so that the 
client is not confused  btw this case and an actual log corruption.

> java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
> ---
>
> Key: KAFKA-1196
> URL: https://issues.apache.org/jira/browse/KAFKA-1196
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.0
> Environment: running java 1.7, linux and kafka compiled against scala 
> 2.9.2
>Reporter: Gerrit Jansen van Vuuren
>Assignee: Neha Narkhede
>Priority: Blocker
>  Labels: newbie
> Fix For: 0.9.0
>
>
> I have 6 topics each with 8 partitions spread over 4 kafka servers.
> the servers are 24 core 72 gig ram.
> While consuming from the topics I get an IlegalArgumentException and all 
> consumption stops, the error keeps on throwing.
> I've tracked it down to FectchResponse.scala line 33
> The error happens when the FetchResponsePartitionData object's readFrom 
> method calls:
> messageSetBuffer.limit(messageSetSize)
> I put in some debug code the the messageSetSize is 671758648, while the 
> buffer.capacity() gives 155733313, for some reason the buffer is smaller than 
> the required message size.
> I don't know the consumer code enough to debug this. It doesn't matter if 
> compression is used or not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (KAFKA-1196) java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33

[Gerrit Jansen van Vuuren (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872215#comment-13872215
 ] 

Gerrit Jansen van Vuuren commented on KAFKA-1196:
-

:) , it doesn't seem like there is an easy fix, 
 I'm trying to write my own consumer that would try and work around this error, 
i.e. ignore the initial message length, and still read the message sets, even 
though the actual topic/partition/messageset sequences might go over the 
initial 4 byte int message size. will comment on how this goes.

> java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
> ---
>
> Key: KAFKA-1196
> URL: https://issues.apache.org/jira/browse/KAFKA-1196
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.0
> Environment: running java 1.7, linux and kafka compiled against scala 
> 2.9.2
>Reporter: Gerrit Jansen van Vuuren
>Assignee: Neha Narkhede
>Priority: Blocker
>  Labels: newbie
> Fix For: 0.9.0
>
>
> I have 6 topics each with 8 partitions spread over 4 kafka servers.
> the servers are 24 core 72 gig ram.
> While consuming from the topics I get an IlegalArgumentException and all 
> consumption stops, the error keeps on throwing.
> I've tracked it down to FectchResponse.scala line 33
> The error happens when the FetchResponsePartitionData object's readFrom 
> method calls:
> messageSetBuffer.limit(messageSetSize)
> I put in some debug code the the messageSetSize is 671758648, while the 
> buffer.capacity() gives 155733313, for some reason the buffer is smaller than 
> the required message size.
> I don't know the consumer code enough to debug this. It doesn't matter if 
> compression is used or not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (KAFKA-1202) optimize ZK access in KafkaController

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1202.


   Resolution: Fixed
Fix Version/s: 0.8.1

Thanks for the review. Committed to trunk. Also, incorporated changes in 
kafka-1020.

> optimize ZK access in KafkaController
> -
>
> Key: KAFKA-1202
> URL: https://issues.apache.org/jira/browse/KAFKA-1202
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1
>Reporter: Jun Rao
>Assignee: Jun Rao
> Fix For: 0.8.1
>
> Attachments: KAFKA-1202.patch
>
>
> In KafkaController, we access ZK in the following places. Those accesses are 
> not necessary since we can read from the cache in the controller.
> In onBrokerFailure(deadBrokers: Seq[Int]),
> replicaStateMachine.handleStateChanges(getAllReplicasOnBroker(zkClient, 
> controllerContext.allTopics.toSeq, deadBrokers), OfflineReplica)
>   }
> In onBrokerStartup(newBrokers: Seq[Int])
> replicaStateMachine.handleStateChanges(getAllReplicasOnBroker(zkClient, 
> controllerContext.allTopics.toSeq, newBrokers), OnlineReplica)
>   }
> In  shutdownBroker(),
> getPartitionsAssignedToBroker(zkClient, controllerContext.allTopics.toSeq, 
> id).map {
>   }



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1020) Remove getAllReplicasOnBroker from KafkaController

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1020:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Fixed as part of kafka-1202.

> Remove getAllReplicasOnBroker from KafkaController
> --
>
> Key: KAFKA-1020
> URL: https://issues.apache.org/jira/browse/KAFKA-1020
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.8.1
>
> Attachments: KAFKA-1020.v1.patch
>
>
> Today KafkaController call getAllReplicasOnBroker on broker failure and new 
> broker start up to get all the replicas that broker is holding (or suppose to 
> hold). This function actually issue a read on each topic's partition znodes. 
> With large number of topic/partitions this could seriously increase the 
> latency of handling broker failure and new broker startup.
> On the other hand, ControllerContext maintains a partitionReplicaAssignment 
> cache, which is designed to keep the most updated partition replica 
> assignment according to ZK. So instead of reading from ZK, we could just read 
> from the local cache, given that partitionReplicaAssignment is guaranteed to 
> be up-to-date.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (KAFKA-1196) java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33

[Gerrit Jansen van Vuuren (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872281#comment-13872281
 ] 

Gerrit Jansen van Vuuren commented on KAFKA-1196:
-

I've written a consumer https://github.com/gerritjvv/kafka-fast, its running on 
the same dataset implemented what I said in the previous comment, I do not see 
any errors yet.


> java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
> ---
>
> Key: KAFKA-1196
> URL: https://issues.apache.org/jira/browse/KAFKA-1196
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.0
> Environment: running java 1.7, linux and kafka compiled against scala 
> 2.9.2
>Reporter: Gerrit Jansen van Vuuren
>Assignee: Neha Narkhede
>Priority: Blocker
>  Labels: newbie
> Fix For: 0.9.0
>
>
> I have 6 topics each with 8 partitions spread over 4 kafka servers.
> the servers are 24 core 72 gig ram.
> While consuming from the topics I get an IlegalArgumentException and all 
> consumption stops, the error keeps on throwing.
> I've tracked it down to FectchResponse.scala line 33
> The error happens when the FetchResponsePartitionData object's readFrom 
> method calls:
> messageSetBuffer.limit(messageSetSize)
> I put in some debug code the the messageSetSize is 671758648, while the 
> buffer.capacity() gives 155733313, for some reason the buffer is smaller than 
> the required message size.
> I don't know the consumer code enough to debug this. It doesn't matter if 
> compression is used or not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Kafka server start from resource negotiator(s)

[Joe Stein]

Hi, I am working at adding some code to the kafka.Kafka.scala file.

What I was thinking for the Utils.loadProps is to create some
ResourceNegotiatedConfiguration class or something to sit with it so
various frameworks can be supported.

For Mesos I am thinking of storing the properties in zookeeper. For a new
node joining => registering a default config (originally set by a tool) and
saving it unique for that broker in a znode.  This znode would be the input
to the serverConfig for the KafkaServerStartable.

I am going to create a kafka.tools.ResourceNegotiator.ApacheMesos tool too.
 that tool will take in a server.properties file to set the original
default. Once loaded in zookeeper another command to take a property and
flag it with a function to run (like the "use new broker id value") and
such.

I would rather have some implementation in Kafka.scala and have object
KafkaMesos be in Kafka.scala too but wasn't sure what other thoughts folks
might have?

I will be creating JIRA soon for it.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/

[jira] [Created] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

Joe Stein created KAFKA-1206:


 Summary: allow Kafka to start from a resource negotiator system
 Key: KAFKA-1206
 URL: https://issues.apache.org/jira/browse/KAFKA-1206
 Project: Kafka
  Issue Type: Bug
Reporter: Joe Stein
 Fix For: 0.8.1






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (KAFKA-1207) Launch Kafka from within Apache Mesos

[Joe Stein (JIRA)]

Joe Stein created KAFKA-1207:


 Summary: Launch Kafka from within Apache Mesos
 Key: KAFKA-1207
 URL: https://issues.apache.org/jira/browse/KAFKA-1207
 Project: Kafka
  Issue Type: Bug
Reporter: Joe Stein
 Fix For: 0.8.1


What I was thinking for the Utils.loadProps is to create some 
ResourceNegotiatedConfiguration class or something to sit with it so various 
frameworks can be supported.

For Mesos I am thinking of storing the properties in zookeeper. For a new node 
joining => registering a default config (originally set by a tool) and saving 
it unique for that broker in a znode.  This znode would be the input to the 
serverConfig for the KafkaServerStartable.  

I am going to create a kafka.tools.ResourceNegotiator.ApacheMesos tool too.  
that tool will take in a server.properties file to set the original default. 
Once loaded in zookeeper another command to take a property and flag it with a 
function to run (like the "use new broker id value") and such.

I would rather have some implementation in Kafka.scala and have object 
KafkaMesos be in Kafka.scala too but wasn't sure what other thoughts folks 
might have?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1207) Launch Kafka from within Apache Mesos

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1207:
-

Labels: mesos  (was: )

> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
>
> What I was thinking for the Utils.loadProps is to create some 
> ResourceNegotiatedConfiguration class or something to sit with it so various 
> frameworks can be supported.
> For Mesos I am thinking of storing the properties in zookeeper. For a new 
> node joining => registering a default config (originally set by a tool) and 
> saving it unique for that broker in a znode.  This znode would be the input 
> to the serverConfig for the KafkaServerStartable.  
> I am going to create a kafka.tools.ResourceNegotiator.ApacheMesos tool too.  
> that tool will take in a server.properties file to set the original default. 
> Once loaded in zookeeper another command to take a property and flag it with 
> a function to run (like the "use new broker id value") and such.
> I would rather have some implementation in Kafka.scala and have object 
> KafkaMesos be in Kafka.scala too but wasn't sure what other thoughts folks 
> might have?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1206:
-

Labels: mesos  (was: )

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1206:
-

Description: 
We need a generic implementation to hold the property information for brokers, 
producers and consumers.  We want the resource negotiator to store this 
information however it wants and give it respond with a java.util.Properties.  
This can get used then in the Kafka.scala as serverConfigs for the 
KafkaServerStartable.


> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Review Request 16923: Patch for KAFKA-1206

[Joe Stein]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16923/
---

Review request for kafka.


Bugs: KAFKA-1206
https://issues.apache.org/jira/browse/KAFKA-1206


Repository: kafka


Description
---

KAFKA-1206 allow Kafka to start from a resource negotiator system


Diffs
-

  core/src/main/scala/kafka/Kafka.scala 
988014a51c332ba04609ee7fbb06f0b09546f267 

Diff: https://reviews.apache.org/r/16923/diff/


Testing
---


Thanks,

Joe Stein

[jira] [Updated] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1206:
-

Attachment: KAFKA-1206.patch

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872749#comment-13872749
 ] 

Joe Stein commented on KAFKA-1206:
--

Created reviewboard https://reviews.apache.org/r/16923/
 against branch origin/trunk

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (KAFKA-1208) Update system test still to use kafka-topics instead of kafka-create-topics shell

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-1208:


 Summary: Update system test still to use kafka-topics instead of 
kafka-create-topics shell
 Key: KAFKA-1208
 URL: https://issues.apache.org/jira/browse/KAFKA-1208
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
Assignee: Guozhang Wang
 Fix For: 0.8.1






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (KAFKA-1209) System test transient failure due to ProducerPerformance's connection to servers kept broken

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-1209:


 Summary: System test transient failure due to 
ProducerPerformance's connection to servers kept broken
 Key: KAFKA-1209
 URL: https://issues.apache.org/jira/browse/KAFKA-1209
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
 Fix For: 0.8.1


The 400x testsuit has transient test failures, which is related to producer 
performance cannot connect to any of the brokers. Example log on the producer 
end:

---
[2014-01-10 13:25:04,653] WARN Failed to send producer request with correlation 
id 128 to broker 1 with data for partitions [test_1,0] 
(kafka.producer.async.DefaultEventHandler)
java.net.SocketTimeoutException
at 
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
at 
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:221)
at kafka.utils.Utils$.read(Utils.scala:375)


[2014-01-11 13:23:42,220] WARN Failed to send producer request with correlation 
id 126 to broker 2 with data for partitions [test_1,1] 
(kafka.producer.async.DefaultEventHandler)
java.io.EOFException: Received -1 when reading from channel, socket has likely 
been closed.
at kafka.utils.Utils$.read(Utils.scala:376)
at 
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at 
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)





--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Review Request 16360: Fix the offset bug while adding fetchers

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16360/
---

(Updated Jan. 16, 2014, 12:43 a.m.)


Review request for kafka.


Summary (updated)
-

Fix the offset bug while adding fetchers


Bugs: KAFKA-1188
https://issues.apache.org/jira/browse/KAFKA-1188


Repository: kafka


Description (updated)
---

KAFKA-1188.v5


Dummy


KAFKA-1188.v4


KAFKA-1188.v3


KAFKA-1118.v2


KAFKA-1188.v1


Diffs (updated)
-

  core/src/main/scala/kafka/cluster/Partition.scala 
1087a2e91c86e36a2494a95913a3ec2daf238287 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
f9d10d385cee49a1e3be8c82e3ffa22ef87a8fd6 

Diff: https://reviews.apache.org/r/16360/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1188) Stale LeaderAndIsr request could be handled by the broker on Controller failover

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1188:
-

Attachment: KAFKA-1188_2014-01-15_16:43:30.patch

> Stale LeaderAndIsr request could be handled by the broker on Controller 
> failover
> 
>
> Key: KAFKA-1188
> URL: https://issues.apache.org/jira/browse/KAFKA-1188
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
> Attachments: KAFKA-1188.patch, KAFKA-1188_2013-12-24_11:07:12.patch, 
> KAFKA-1188_2014-01-02_10:43:54.patch, KAFKA-1188_2014-01-15_16:43:30.patch
>
>
> ... which will cause the broker to truncate log as responding in 
> makeFollower, and hence lose data. 
> One procedure to produce this issue:
> 3 brokers, 3 partitions, replication factor = 3.
> 1. Broker 1 is the original controller.
> 2. Broker 3 lost registration, 1 send LeaderAndIsr to 1 for isr shrinking  
> (leader of partition 1), but not to 2 since it is not the leader.
> 3. Broker 1 dies, new controller 2 sends all LeaderAndIsr to 2 and 3 
> (resumed), and 2 handles the makeFollower, truncates data.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (KAFKA-1188) Stale LeaderAndIsr request could be handled by the broker on Controller failover

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872882#comment-13872882
 ] 

Guozhang Wang commented on KAFKA-1188:
--

Updated reviewboard https://reviews.apache.org/r/16360/
 against branch origin/trunk

> Stale LeaderAndIsr request could be handled by the broker on Controller 
> failover
> 
>
> Key: KAFKA-1188
> URL: https://issues.apache.org/jira/browse/KAFKA-1188
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
> Attachments: KAFKA-1188.patch, KAFKA-1188_2013-12-24_11:07:12.patch, 
> KAFKA-1188_2014-01-02_10:43:54.patch, KAFKA-1188_2014-01-15_16:43:30.patch
>
>
> ... which will cause the broker to truncate log as responding in 
> makeFollower, and hence lose data. 
> One procedure to produce this issue:
> 3 brokers, 3 partitions, replication factor = 3.
> 1. Broker 1 is the original controller.
> 2. Broker 3 lost registration, 1 send LeaderAndIsr to 1 for isr shrinking  
> (leader of partition 1), but not to 2 since it is not the leader.
> 3. Broker 1 dies, new controller 2 sends all LeaderAndIsr to 2 and 3 
> (resumed), and 2 handles the makeFollower, truncates data.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Review Request 16360: Fix the offset bug while adding fetchers

[Guozhang Wang]

> On Jan. 7, 2014, 5:12 p.m., Neha Narkhede wrote:
> > In general, does the alternate approach of sending all commands to every 
> > broker and let the broker ignore a become follower if it is for the same 
> > leader as the current one? The reason why this would work is because all 
> > controller-broker communication is synchronous, so even if the same broker 
> > became leader in quick succession (1->2->1), the follower would see the 
> > 1->2 transition before seeing the 2->1, so it wouldn't incorrectly ignore a 
> > become follower request from the same leader.

Sending LeaderAndIsr request to followers on non-leader change has the 
following risk: since the follower will only increase its HW on the next fetch 
request after the leader has increased its HW, during this time if a 
LeaderAndIsr request reached the follower it may truncate the last fetched data 
since they are not within HW yet. Then if there is a leader change immediately 
the data will be lost without even notified by the controller.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16360/#review31299
---


On Jan. 16, 2014, 12:43 a.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16360/
> ---
> 
> (Updated Jan. 16, 2014, 12:43 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1188
> https://issues.apache.org/jira/browse/KAFKA-1188
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1188.v5
> 
> 
> Dummy
> 
> 
> KAFKA-1188.v4
> 
> 
> KAFKA-1188.v3
> 
> 
> KAFKA-1118.v2
> 
> 
> KAFKA-1188.v1
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 1087a2e91c86e36a2494a95913a3ec2daf238287 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> f9d10d385cee49a1e3be8c82e3ffa22ef87a8fd6 
> 
> Diff: https://reviews.apache.org/r/16360/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

development branches in git

[Joe Stein]

What are folks thoughts about development branches with the branch being
the ticket number?  I think it should continue to follow the bylaws but how
do we vote in the initial commit?

Can a committer just +1 whether the JIRA ticket gets a branch or can we
have it so committer (even if committer is working on the code) can branch
for a ticket so the work can get collaborated on?

I want to create a branch for
https://issues.apache.org/jira/browse/KAFKA-1207 which will likely have 3-4
sub tickets it looks like now (which would be nice to include KAFKA-1206
too if folks have a chance to look at the change so if there are issue they
can get in).

Would it make sense to just create a 0.9 branch and do it under that?

I was thinking of doing a few security ones also.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/

Re: development branches in git

[Guozhang Wang]

For mos of the JIRAs one person is mainly in charge of the patch. For large
JIRAs that would need multiple sub-tickets we can just assign different
people on sub-tasks.

That said, I think it makes sense to create branch when the development
period would take long (> 1 month, say), and hence consecutive pulling and
rebasing would become a burden..

Guozhang


On Wed, Jan 15, 2014 at 5:48 PM, Joe Stein  wrote:

> What are folks thoughts about development branches with the branch being
> the ticket number?  I think it should continue to follow the bylaws but how
> do we vote in the initial commit?
>
> Can a committer just +1 whether the JIRA ticket gets a branch or can we
> have it so committer (even if committer is working on the code) can branch
> for a ticket so the work can get collaborated on?
>
> I want to create a branch for
> https://issues.apache.org/jira/browse/KAFKA-1207 which will likely have
> 3-4
> sub tickets it looks like now (which would be nice to include KAFKA-1206
> too if folks have a chance to look at the change so if there are issue they
> can get in).
>
> Would it make sense to just create a 0.9 branch and do it under that?
>
> I was thinking of doing a few security ones also.
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>



-- 
-- Guozhang

Review Request 16944: Use kafka-topics for system-test

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16944/
---

Review request for kafka.


Bugs: KAFKA-1208
https://issues.apache.org/jira/browse/KAFKA-1208


Repository: kafka


Description
---

KAFKA-1208.v1


Diffs
-

  system_test/utils/kafka_system_test_utils.py 
705c9179688f2418e31080ed973dedd1566f877a 

Diff: https://reviews.apache.org/r/16944/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1208) Update system test still to use kafka-topics instead of kafka-create-topics shell

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873013#comment-13873013
 ] 

Guozhang Wang commented on KAFKA-1208:
--

Created reviewboard https://reviews.apache.org/r/16944/
 against branch origin/trunk

> Update system test still to use kafka-topics instead of kafka-create-topics 
> shell
> -
>
> Key: KAFKA-1208
> URL: https://issues.apache.org/jira/browse/KAFKA-1208
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.8.1
>
> Attachments: KAFKA-1208.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1208) Update system test still to use kafka-topics instead of kafka-create-topics shell

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1208:
-

Attachment: KAFKA-1208.patch

> Update system test still to use kafka-topics instead of kafka-create-topics 
> shell
> -
>
> Key: KAFKA-1208
> URL: https://issues.apache.org/jira/browse/KAFKA-1208
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.8.1
>
> Attachments: KAFKA-1208.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (KAFKA-1208) Update system test still to use kafka-topics instead of kafka-create-topics shell

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1208.


Resolution: Fixed

Thanks for the patch. +1 and committed to trunk.

> Update system test still to use kafka-topics instead of kafka-create-topics 
> shell
> -
>
> Key: KAFKA-1208
> URL: https://issues.apache.org/jira/browse/KAFKA-1208
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.8.1
>
> Attachments: KAFKA-1208.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Review Request 16360: Fix the offset bug while adding fetchers

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16360/#review32002
---


Looks good. Just a couple of minor comments.


core/src/main/scala/kafka/server/ReplicaManager.scala


Should controllerId be logged using %d?



core/src/main/scala/kafka/server/ReplicaManager.scala


Since this foreach is not a liner, it's probably better to use {, instead 
of (?


- Jun Rao


On Jan. 16, 2014, 12:43 a.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16360/
> ---
> 
> (Updated Jan. 16, 2014, 12:43 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1188
> https://issues.apache.org/jira/browse/KAFKA-1188
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1188.v5
> 
> 
> Dummy
> 
> 
> KAFKA-1188.v4
> 
> 
> KAFKA-1188.v3
> 
> 
> KAFKA-1118.v2
> 
> 
> KAFKA-1188.v1
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 1087a2e91c86e36a2494a95913a3ec2daf238287 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> f9d10d385cee49a1e3be8c82e3ffa22ef87a8fd6 
> 
> Diff: https://reviews.apache.org/r/16360/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Re: Kafka server start from resource negotiator(s)

[Jun Rao]

Hi, Joe,

Thanks for looking into this. Do you plan to add a dependency to Mesos in
kafka jar?

Jun


On Wed, Jan 15, 2014 at 10:50 AM, Joe Stein  wrote:

> Hi, I am working at adding some code to the kafka.Kafka.scala file.
>
> What I was thinking for the Utils.loadProps is to create some
> ResourceNegotiatedConfiguration class or something to sit with it so
> various frameworks can be supported.
>
> For Mesos I am thinking of storing the properties in zookeeper. For a new
> node joining => registering a default config (originally set by a tool) and
> saving it unique for that broker in a znode.  This znode would be the input
> to the serverConfig for the KafkaServerStartable.
>
> I am going to create a kafka.tools.ResourceNegotiator.ApacheMesos tool too.
>  that tool will take in a server.properties file to set the original
> default. Once loaded in zookeeper another command to take a property and
> flag it with a function to run (like the "use new broker id value") and
> such.
>
> I would rather have some implementation in Kafka.scala and have object
> KafkaMesos be in Kafka.scala too but wasn't sure what other thoughts folks
> might have?
>
> I will be creating JIRA soon for it.
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>

[jira] [Updated] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1206:
-

Attachment: KAFKA-1206_2014-01-16_00:36:52.patch

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206.patch, KAFKA-1206_2014-01-16_00:36:52.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Review Request 16923: Patch for KAFKA-1206

[Joe Stein]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16923/
---

(Updated Jan. 16, 2014, 5:37 a.m.)


Review request for kafka.


Bugs: KAFKA-1206
https://issues.apache.org/jira/browse/KAFKA-1206


Repository: kafka


Description (updated)
---

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
KAFKA-1206


KAFKA-1206 allow Kafka to start from a resource negotiator system


Diffs (updated)
-

  core/src/main/scala/kafka/Kafka.scala 
988014a51c332ba04609ee7fbb06f0b09546f267 

Diff: https://reviews.apache.org/r/16923/diff/


Testing
---


Thanks,

Joe Stein

[jira] [Commented] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873063#comment-13873063
 ] 

Joe Stein commented on KAFKA-1206:
--

Updated reviewboard https://reviews.apache.org/r/16923/
 against branch origin/trunk

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206.patch, KAFKA-1206_2014-01-16_00:36:52.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1206:
-

Attachment: (was: KAFKA-1206.patch)

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206_2014-01-16_00:36:52.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Review Request 16923: Patch for KAFKA-1206

[Joe Stein]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16923/
---

(Updated Jan. 16, 2014, 5:39 a.m.)


Review request for kafka.


Bugs: KAFKA-1206
https://issues.apache.org/jira/browse/KAFKA-1206


Repository: kafka


Description (updated)
---

KAFKA-1206 we don't need the mesos jar for this change


Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
KAFKA-1206


KAFKA-1206 allow Kafka to start from a resource negotiator system


Diffs (updated)
-

  core/build.sbt b5bcb44f783ae6fe2cfdc091e53e867a56a1e592 
  core/src/main/scala/kafka/Kafka.scala 
988014a51c332ba04609ee7fbb06f0b09546f267 

Diff: https://reviews.apache.org/r/16923/diff/


Testing
---


Thanks,

Joe Stein

[jira] [Updated] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1206:
-

Attachment: (was: KAFKA-1206_2014-01-16_00:36:52.patch)

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Review Request 16923: Patch for KAFKA-1206

[Joe Stein]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16923/
---

(Updated Jan. 16, 2014, 5:40 a.m.)


Review request for kafka.


Bugs: KAFKA-1206
https://issues.apache.org/jira/browse/KAFKA-1206


Repository: kafka


Description (updated)
---

KAFKA-1206 we don't need the mesos jar for this change


KAFKA-1206 we don't need the mesos jar for this change


Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
KAFKA-1206


KAFKA-1206 allow Kafka to start from a resource negotiator system


Diffs (updated)
-

  core/src/main/scala/kafka/Kafka.scala 
988014a51c332ba04609ee7fbb06f0b09546f267 

Diff: https://reviews.apache.org/r/16923/diff/


Testing
---


Thanks,

Joe Stein

[jira] [Updated] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1206:
-

Attachment: KAFKA-1206_2014-01-16_00:40:30.patch

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206_2014-01-16_00:40:30.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1206:
-

Attachment: (was: KAFKA-1206_2014-01-16_00:39:12.patch)

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206_2014-01-16_00:40:30.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873065#comment-13873065
 ] 

Joe Stein commented on KAFKA-1206:
--

Updated reviewboard https://reviews.apache.org/r/16923/
 against branch origin/trunk

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206_2014-01-16_00:40:30.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1206:
-

Attachment: KAFKA-1206_2014-01-16_00:39:12.patch

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206_2014-01-16_00:40:30.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873064#comment-13873064
 ] 

Joe Stein commented on KAFKA-1206:
--

Updated reviewboard https://reviews.apache.org/r/16923/
 against branch origin/trunk

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206_2014-01-16_00:40:30.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Kafka server start from resource negotiator(s)

[Joe Stein]

Right now I am testing it that way. Once it is working and if that becomes
a problem then moving it into a mesosFramework project is no problem.   if
we do that it would be beneficial if they were organized under some
resource manager folder or something.

The way I wrote https://issues.apache.org/jira/browse/KAFKA-1206 was with
that in mind so folks have a light interface for doing this.  The specific
systems resource negotiator implementation could go in their project and
just importing the ResourceNegotiator trait in the project's reference of
the jar.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Thu, Jan 16, 2014 at 12:23 AM, Jun Rao  wrote:

> Hi, Joe,
>
> Thanks for looking into this. Do you plan to add a dependency to Mesos in
> kafka jar?
>
> Jun
>
>
> On Wed, Jan 15, 2014 at 10:50 AM, Joe Stein  wrote:
>
> > Hi, I am working at adding some code to the kafka.Kafka.scala file.
> >
> > What I was thinking for the Utils.loadProps is to create some
> > ResourceNegotiatedConfiguration class or something to sit with it so
> > various frameworks can be supported.
> >
> > For Mesos I am thinking of storing the properties in zookeeper. For a new
> > node joining => registering a default config (originally set by a tool)
> and
> > saving it unique for that broker in a znode.  This znode would be the
> input
> > to the serverConfig for the KafkaServerStartable.
> >
> > I am going to create a kafka.tools.ResourceNegotiator.ApacheMesos tool
> too.
> >  that tool will take in a server.properties file to set the original
> > default. Once loaded in zookeeper another command to take a property and
> > flag it with a function to run (like the "use new broker id value") and
> > such.
> >
> > I would rather have some implementation in Kafka.scala and have object
> > KafkaMesos be in Kafka.scala too but wasn't sure what other thoughts
> folks
> > might have?
> >
> > I will be creating JIRA soon for it.
> >
> > /***
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop 
> > /
> >
>

[jira] [Updated] (KAFKA-1207) Launch Kafka from within Apache Mesos

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1207:
-

Status: Patch Available  (was: Open)

> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
>
> What I was thinking for the Utils.loadProps is to create some 
> ResourceNegotiatedConfiguration class or something to sit with it so various 
> frameworks can be supported.
> For Mesos I am thinking of storing the properties in zookeeper. For a new 
> node joining => registering a default config (originally set by a tool) and 
> saving it unique for that broker in a znode.  This znode would be the input 
> to the serverConfig for the KafkaServerStartable.  
> I am going to create a kafka.tools.ResourceNegotiator.ApacheMesos tool too.  
> that tool will take in a server.properties file to set the original default. 
> Once loaded in zookeeper another command to take a property and flag it with 
> a function to run (like the "use new broker id value") and such.
> I would rather have some implementation in Kafka.scala and have object 
> KafkaMesos be in Kafka.scala too but wasn't sure what other thoughts folks 
> might have?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1207) Launch Kafka from within Apache Mesos

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1207:
-

Status: Open  (was: Patch Available)

> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
>
> What I was thinking for the Utils.loadProps is to create some 
> ResourceNegotiatedConfiguration class or something to sit with it so various 
> frameworks can be supported.
> For Mesos I am thinking of storing the properties in zookeeper. For a new 
> node joining => registering a default config (originally set by a tool) and 
> saving it unique for that broker in a znode.  This znode would be the input 
> to the serverConfig for the KafkaServerStartable.  
> I am going to create a kafka.tools.ResourceNegotiator.ApacheMesos tool too.  
> that tool will take in a server.properties file to set the original default. 
> Once loaded in zookeeper another command to take a property and flag it with 
> a function to run (like the "use new broker id value") and such.
> I would rather have some implementation in Kafka.scala and have object 
> KafkaMesos be in Kafka.scala too but wasn't sure what other thoughts folks 
> might have?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1206) allow Kafka to start from a resource negotiator system

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1206:
-

Status: Patch Available  (was: Open)

> allow Kafka to start from a resource negotiator system
> --
>
> Key: KAFKA-1206
> URL: https://issues.apache.org/jira/browse/KAFKA-1206
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Fix For: 0.8.1
>
> Attachments: KAFKA-1206_2014-01-16_00:40:30.patch
>
>
> We need a generic implementation to hold the property information for 
> brokers, producers and consumers.  We want the resource negotiator to store 
> this information however it wants and give it respond with a 
> java.util.Properties.  This can get used then in the Kafka.scala as 
> serverConfigs for the KafkaServerStartable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)