I would like to initiate the voting process for the "KIP-4 Create Topics Schema changes". This is not a vote for all of KIP-4, but specifically for the create topics changes. I have included the exact changes below for clarity: > > Create Topics Request (KAFKA-2945 > <https://issues.apache.org/jira/browse/KAFKA-2945>) > > CreateTopics Request (Version: 0) => [create_topic_requests] timeout > create_topic_requests => topic num_partitions replication_factor > [replica_assignment] [configs] > topic => STRING > num_partitions => INT32 > replication_factor => INT16 > replica_assignment => partition_id [replicas] > partition_id => INT32 > replicas => INT32 > configs => config_key config_value > config_key => STRING > config_value => STRING > timeout => INT32 > > CreateTopicsRequest is a batch request to initiate topic creation with > either predefined or automatic replica assignment and optionally topic > configuration. > > Request semantics: > > 1. Must be sent to the controller broker > 2. If there are multiple instructions for the same topic in one > request an InvalidRequestException will be logged on the broker and the > client will be disconnected. > - This is because the list of topics is modeled server side as a > map with TopicName as the key > 3. The principal must be authorized to the "Create" Operation on the > "Cluster" resource to create topics. > - Unauthorized requests will receive a ClusterAuthorizationException > 4. > > Only one from ReplicaAssignment or (num_partitions + replication_factor > ), can be defined in one instruction. > - If both parameters are specified an InvalidRequestException will be > logged on the broker and the client will be disconnected. > - In the case ReplicaAssignment is defined number of partitions and > replicas will be calculated from the supplied replica_assignment. > - In the case of defined (num_partitions + replication_factor) > replica assignment will be automatically generated by the server. > - One or the other must be defined. The existing broker side auto > create defaults will not be used > (default.replication.factor, num.partitions). The client implementation > can > have defaults for these options when generating the messages. > - The first replica in [replicas] is assumed to be the preferred > leader. This matches current behavior elsewhere. > 5. Setting a timeout > 0 will allow the request to block until the > topic metadata is "complete" on the controller node. > - Complete means the local topic metadata cache been completely > populated and all partitions have leaders > - The topic metadata is updated when the controller sends out > update metadata requests to the brokers > - If a timeout error occurs, the topic could still be created > successfully at a later time. Its up to the client to query for the > state > at that point. > 6. Setting a timeout <= 0 will validate arguments and trigger the > create topics and return immediately. > - This is essentially the fully asynchronous mode we have in the > Zookeeper tools today. > - The error code in the response will either contain an argument > validation exception or a timeout exception. If you receive a timeout > exception, because you asked for 0 timeout, you can assume the message > was > valid and the topic creation was triggered. > 7. The request is not transactional. > 1. If an error occurs on one topic, the others could still be > created. > 2. Errors are reported independently. > > QA: > > - Why is CreateTopicsRequest a batch request? > - Scenarios where tools or admins want to create many topics should > be able to with fewer requests > - Example: MirrorMaker may want to create the topics downstream > - What happens if some topics error immediately? Will it > return immediately? > - The request will block until all topics have either been created, > errors, or the timeout has been hit > - There is no "short circuiting" where 1 error stops the other > topics from being created > - Why implement "partial blocking" instead of fully async or fully > consistent? > - See Cluster Consistent Blocking > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking> > below > - Why require the request to go to the controller? > - The controller is responsible for the cluster metadata and its > propagation > - See Request Forwarding > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request> > below > > Create Topics Response > > > > CreateTopics Response (Version: 0) => [topic_error_codes] > topic_error_codes => topic error_code > topic => STRING > error_code => INT16 > > CreateTopicsResponse contains a map between topic and topic creation > result error code (see New Protocol Errors > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors> > ). >
The KIP is available here for reference (linked to the Create Topics schema section): *https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945) <https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)>* A pull request is available implementing the proposed changes here: https://github.com/apache/kafka/pull/1489 Here is a link to the past discussion on the mailing list: *http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema <http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema>* Thank you, Grant -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
