[jira] [Commented] (KAFKA-843) Re-add the release-zip sbt target
[ https://issues.apache.org/jira/browse/KAFKA-843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632656#comment-13632656 ] Cosmin Lehene commented on KAFKA-843: - [~junrao] Is the last patch ok? > Re-add the release-zip sbt target > - > > Key: KAFKA-843 > URL: https://issues.apache.org/jira/browse/KAFKA-843 > Project: Kafka > Issue Type: Sub-task > Components: packaging >Affects Versions: 0.8 >Reporter: Cosmin Lehene >Assignee: Cosmin Lehene > Fix For: 0.8 > > Attachments: > 0001-KAFKA-843-Addendum-comply-with-Semantic-Versioning-2.patch, > KAFKA-843.patch, KAFKA-843.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > This target has been removed a while ago. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-823) merge 0.8 (51421fcc0111031bb77f779a6f6c00520d526a34) to trunk
[ https://issues.apache.org/jira/browse/KAFKA-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632925#comment-13632925 ] Jay Kreps commented on KAFKA-823: - Comments: - I think moving Log.append to return a tuple of longs is a worse api as it won't support extension in the future. This is the spot where we do our iteration of the message set so it is likely that we will have other stats we want to compute. Let's keep the append info object. - If there are wrong values in the append info object they need to be fixed or renamed (shallowCount?) we can't just leave them there wrong - You moved back to having the if(assignOffsets) block return the last offset. How come? - In general I think the logic of using (lastOffset - firstOffset + 1) as the number of appended messages isn't correct. I think using the shallow message count would be better or passing back the actual number of messages from the deep iteration which would require some refactoring. The problem with using this is that when bootstrapping a follower it would flush on every append for any topic with sparse offsets. > merge 0.8 (51421fcc0111031bb77f779a6f6c00520d526a34) to trunk > - > > Key: KAFKA-823 > URL: https://issues.apache.org/jira/browse/KAFKA-823 > Project: Kafka > Issue Type: Task > Components: core >Reporter: Jun Rao >Assignee: Jun Rao > Attachments: kafka-823.patch, kafka-823_v2.patch, kafka-823_v3.patch > > > merge 0.8 up to the following commit to trunk > commit 51421fcc0111031bb77f779a6f6c00520d526a34 > Author: Neha Narkhede > Date: Fri Mar 22 09:32:27 2013 -0700 > KAFKA-816 Reduce noise in Kafka server logs due to > NotLeaderForPartitionException; reviewed by Jun Rao -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-843) Re-add the release-zip sbt target
[ https://issues.apache.org/jira/browse/KAFKA-843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633003#comment-13633003 ] Jun Rao commented on KAFKA-843: --- Thanks for the addendum. It looks fine. I found another minor issue (exists without the new patch). When doing a zip release, I saw the zip file in two places: ./target/kafka_2.8.0-0.8-SNAPSHOT.zip ./target/RELEASE/kafka_2.8.0-0.8-SNAPSHOT.zip The tar release only exists in the RELEASE directory. Could you fix that an provide a new patch? > Re-add the release-zip sbt target > - > > Key: KAFKA-843 > URL: https://issues.apache.org/jira/browse/KAFKA-843 > Project: Kafka > Issue Type: Sub-task > Components: packaging >Affects Versions: 0.8 >Reporter: Cosmin Lehene >Assignee: Cosmin Lehene > Fix For: 0.8 > > Attachments: > 0001-KAFKA-843-Addendum-comply-with-Semantic-Versioning-2.patch, > KAFKA-843.patch, KAFKA-843.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > This target has been removed a while ago. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: interesting paper on log replication
More notable differences from Kafka as far as log replication protocol is concerned - - Raft considers log entries as committed as soon as it is acknowledged by a majority of the servers in a cluster. Compare this to Kafka where we have the notion of "in-sync followers" that are required to ack every batch of log entries in order for the leader to commit those. - Raft uses the election voting mechanism to select a new leader whose log is as “up-to-date” as possible. Compare this to Kafka where we can pick ANY of the "in-sync followers" as the next leader, we typically pick the first one in the list. We do not try to pick the "in-sync follower" with the largest log for simplicity and fewer RPCs. - In Raft, when the follower's log diverts from the leader's (in the presence of multiple failures), the leader-follower RPC truncates the follower's log up to the diversion point and then replicate the rest of the leader's log. This ensures that follower's log is identical to that of the leader's in such situations. Compare this to Kafka, where we allow the logs to divert and don't reconcile perfectly. Thanks, Neha On Sun, Apr 14, 2013 at 9:42 PM, Jun Rao wrote: > Thanks for the link. This paper provides an alternative, but similar > implementation to that in Zookeeper. The key difference seems to be that > the former supports membership reconfiguration. > > Kafka replication is simpler because it separates the leader election part > from log replication. Such separation has a few benefits: (1) the leader > election part is easier to implement by leveraging a consensus system (e.g. > Zookeeper); (2) the log format is simpler since the log itself is not used > for leader election; (3) the replication factor for the log is decoupled > from the number of parties required for leader election (e.g., in Kafka we > can choose a replication factor of 2 for the log while using an ensemble of > 5 for a Zookeeper cluster). > > Both Rafe and Zookeeper are solving a harder problem than Kafka replication > because they have no consensus service to rely upon for their own leader > election since they are implementing a consensus service. > > Thanks, > > Jun > > > On Tue, Apr 9, 2013 at 10:34 PM, Jay Kreps wrote: > >> Very similar in design to kafka replication >> https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf >> >> -Jay >>
[jira] [Created] (KAFKA-864) bin/kafka-gen-reassignment.sh
Scott Clasen created KAFKA-864: -- Summary: bin/kafka-gen-reassignment.sh Key: KAFKA-864 URL: https://issues.apache.org/jira/browse/KAFKA-864 Project: Kafka Issue Type: Improvement Affects Versions: 0.8 Reporter: Scott Clasen Better tooling for replacing failed brokers and reassigning partitions. The output of ./bin/kafka-list-topic.sh makes doing this in bulk painful. Tool should output json format acceptable to ./bin/kafka-reassign-partitions.sh Spec: bin/kafka-gen-reassignment.sh Option Description -- --- --topic REQUIRED: The topic to be reassigned. Defaults to all existing topics. (default: ) --partition REQUIRED: The partition to be reassigned. Defaults to all partitions. (default: ) --from REQUIRED: The broker to reassign the partition from --to REQUIRED: The broker to reassign the partition to. --zookeeper REQUIRED: The connection string for the zookeeper connection in the form host:port. Multiple URLS can be given to allow fail-over. Workflow: Replacing a broker ./bin/kafka-gen-reassignment.sh --zookeeper --from --to > reassign.json ./bin/kafka-reassign-partitions.sh --zookeeper --path-to-json-file reassign.json -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-823) merge 0.8 (51421fcc0111031bb77f779a6f6c00520d526a34) to trunk
[ https://issues.apache.org/jira/browse/KAFKA-823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-823: -- Attachment: kafka-823_v4.patch Thanks for the review. The suggestions make sense. Attach patch v4 that addresses those issues. The implication is that now the flush interval is based on # of shallow messages, instead of deep. We can revisit that separately. > merge 0.8 (51421fcc0111031bb77f779a6f6c00520d526a34) to trunk > - > > Key: KAFKA-823 > URL: https://issues.apache.org/jira/browse/KAFKA-823 > Project: Kafka > Issue Type: Task > Components: core >Reporter: Jun Rao >Assignee: Jun Rao > Attachments: kafka-823.patch, kafka-823_v2.patch, kafka-823_v3.patch, > kafka-823_v4.patch > > > merge 0.8 up to the following commit to trunk > commit 51421fcc0111031bb77f779a6f6c00520d526a34 > Author: Neha Narkhede > Date: Fri Mar 22 09:32:27 2013 -0700 > KAFKA-816 Reduce noise in Kafka server logs due to > NotLeaderForPartitionException; reviewed by Jun Rao -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: interesting paper on log replication
On the last point, in general, Kafka logs are identical among replicas. The only case that they may not be identical is when an unclean leader election happens, i.e., a leader has to be elected from a replica not in in-sync replica set). Unclean leader election should be rare since this requires multiple broker failures around the same time. Thanks, Jun On Tue, Apr 16, 2013 at 11:14 AM, Neha Narkhede wrote: > More notable differences from Kafka as far as log replication protocol > is concerned - > > - Raft considers log entries as committed as soon as it is > acknowledged by a majority of the servers in a cluster. Compare this > to Kafka where we have the notion of "in-sync followers" that are > required to ack every batch of log entries in order for the leader to > commit those. > > - Raft uses the election voting mechanism to select a new leader > whose log is as “up-to-date” as possible. Compare this to Kafka where > we can pick ANY of the "in-sync followers" as the next leader, we > typically pick the first one in the list. We do not try to pick the > "in-sync follower" with the largest log for simplicity and fewer RPCs. > > - In Raft, when the follower's log diverts from the leader's (in the > presence of multiple failures), the leader-follower RPC truncates the > follower's log up to the diversion point and then replicate the rest > of the leader's log. This ensures that follower's log is identical to > that of the leader's in such situations. Compare this to Kafka, where > we allow the logs to divert and don't reconcile perfectly. > > Thanks, > Neha > > On Sun, Apr 14, 2013 at 9:42 PM, Jun Rao wrote: > > Thanks for the link. This paper provides an alternative, but similar > > implementation to that in Zookeeper. The key difference seems to be that > > the former supports membership reconfiguration. > > > > Kafka replication is simpler because it separates the leader election > part > > from log replication. Such separation has a few benefits: (1) the leader > > election part is easier to implement by leveraging a consensus system > (e.g. > > Zookeeper); (2) the log format is simpler since the log itself is not > used > > for leader election; (3) the replication factor for the log is decoupled > > from the number of parties required for leader election (e.g., in Kafka > we > > can choose a replication factor of 2 for the log while using an ensemble > of > > 5 for a Zookeeper cluster). > > > > Both Rafe and Zookeeper are solving a harder problem than Kafka > replication > > because they have no consensus service to rely upon for their own leader > > election since they are implementing a consensus service. > > > > Thanks, > > > > Jun > > > > > > On Tue, Apr 9, 2013 at 10:34 PM, Jay Kreps wrote: > > > >> Very similar in design to kafka replication > >> > https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf > >> > >> -Jay > >> >
