Hi Harsha/Satish, Thanks for the discussion today. Here is a link to the KIP-405 development milestones google doc we discussed in the meeting today: https://docs.google.com/document/d/1B5_jaZvWWb2DUpgbgImq0k_IPZ4DWrR8Ru7YpuJrXdc/edit . I have shared it with you. Please have a look and share your feedback/improvements. As we discussed, things are clear until milestone 1. Beyond that, we can discuss it again (perhaps in next sync or later), once you have thought through the implementation plan/milestones and release into preview in 3.0.
Cheers, Kowshik On Tue, Nov 10, 2020 at 6:56 AM Satish Duggana <satish.dugg...@gmail.com> wrote: > Hi Jun, > Thanks for your comments. Please find the inline replies below. > > 605.2 "Build the local leader epoch cache by cutting the leader epoch > sequence received from remote storage to [LSO, ELO]." I mentioned an issue > earlier. Suppose the leader's local start offset is 100. The follower finds > a remote segment covering offset range [80, 120). The producerState with > this remote segment is up to offset 120. To trim the producerState to > offset 100 requires more work since one needs to download the previous > producerState up to offset 80 and then replay the messages from 80 to 100. > It seems that it's simpler in this case for the follower just to take the > remote segment as it is and start fetching from offset 120. > > We chose that approach to avoid any edge cases here. It may be > possible that the remote log segment that is received may not have the > same leader epoch sequence from 100-120 as it contains on the > leader(this can happen due to unclean leader). It is safe to start > from what the leader returns here.Another way is to find the remote > log segment > > 5016. Just to echo what Kowshik was saying. It seems that > RLMM.onPartitionLeadershipChanges() is only called on the replicas for a > partition, not on the replicas for the __remote_log_segment_metadata > partition. It's not clear how the leader of __remote_log_segment_metadata > obtains the metadata for remote segments for deletion. > > RLMM will always receive the callback for the remote log metadata > topic partitions hosted on the local broker and these will be > subscribed. I will make this clear in the KIP. > > 5100. KIP-516 has been accepted and is being implemented now. Could you > update the KIP based on topicID? > > We mentioned KIP-516 and how it helps. We will update this KIP with > all the changes it brings with KIP-516. > > 5101. RLMM: It would be useful to clarify how the following two APIs are > used. According to the wiki, the former is used for topic deletion and the > latter is used for retention. It seems that retention should use the former > since remote segments without a matching epoch in the leader (potentially > due to unclean leader election) also need to be garbage collected. The > latter seems to be used for the new leader to determine the last tiered > segment. > default Iterator<RemoteLogSegmentMetadata> > listRemoteLogSegments(TopicPartition topicPartition) > Iterator<RemoteLogSegmentMetadata> listRemoteLogSegments(TopicPartition > topicPartition, long leaderEpoch); > > Right,.that is what we are currently doing. We will update the > javadocs and wiki with that. Earlier, we did not want to remove the > segments which are not matched with leader epochs from the ladder > partition as they may be used later by a replica which can become a > leader (unclean leader election) and refer those segments. But that > may leak these segments in remote storage until the topic lifetime. We > decided to cleanup the segments with the oldest incase of size based > retention also. > > 5102. RSM: > 5102.1 For methods like fetchLogSegmentData(), it seems that they can > use RemoteLogSegmentId instead of RemoteLogSegmentMetadata. > > It will be useful to have metadata for RSM to fetch log segment. It > may create location/path using id with other metadata too. > > 5102.2 In fetchLogSegmentData(), should we use long instead of Long? > > Wanted to keep endPosition as optional to read till the end of the > segment and avoid sentinels. > > 5102.3 Why only some of the methods have default implementation and others > Don't? > > Actually, RSM will not have any default implementations. Those 3 > methods were made default earlier for tests etc. Updated the wiki. > > 5102.4. Could we define RemoteLogSegmentMetadataUpdate > and DeletePartitionUpdate? > > Sure, they will be added. > > > 5102.5 LogSegmentData: It seems that it's easier to pass > in leaderEpochIndex as a ByteBuffer or byte array than a file since it will > be generated in memory. > > Right, this is in plan. > > 5102.6 RemoteLogSegmentMetadata: It seems that it needs both baseOffset and > startOffset. For example, deleteRecords() could move the startOffset to the > middle of a segment. If we copy the full segment to remote storage, the > baseOffset and the startOffset will be different. > > Good point. startOffset is baseOffset by default, if not set explicitly. > > 5102.7 Could we define all the public methods for RemoteLogSegmentMetadata > and LogSegmentData? > > Sure, updated the wiki. > > 5102.8 Could we document whether endOffset in RemoteLogSegmentMetadata is > inclusive/exclusive? > > It is inclusive, will update. > > 5103. configs: > 5103.1 Could we define the default value of non-required configs (e.g the > size of new thread pools)? > > Sure, that makes sense. > > 5103.2 It seems that local.log.retention.ms should default to retention.ms > , > instead of remote.log.retention.minutes. Similarly, it seems > that local.log.retention.bytes should default to segment.bytes. > > Right, we do not have remote.log.retention as we discussed earlier. > Thanks for catching the typo. > > 5103.3 remote.log.manager.thread.pool.size: The description says "used in > scheduling tasks to copy segments, fetch remote log indexes and clean up > remote log segments". However, there is a separate > config remote.log.reader.threads for fetching remote data. It's weird to > fetch remote index and log in different thread pools since both are used > for serving fetch requests. > > Right, remote.log.manager.thread.pool is mainly used for copy/cleanup > activities. Fetch path always goes through remote.log.reader.threads. > > 5103.4 remote.log.manager.task.interval.ms: Is that the amount of time to > back off when there is no work to do? If so, perhaps it can be renamed as > backoff.ms. > > This is the delay interval for each iteration. It may be renamed to > remote.log.manager.task.delay.ms > > 5103.5 Are rlm_process_interval_ms and rlm_retry_interval_ms configs? If > so, they need to be listed in this section. > > remote.log.manager.task.interval.ms is the process internal, retry > interval is missing in the configs, which will be updated in the KIP. > > 5104. "RLM maintains a bounded cache(possibly LRU) of the index files of > remote log segments to avoid multiple index fetches from the remote > storage." Is the RLM in memory or on disk? If on disk, where is it stored? > Do we need a configuration to bound the size? > > It is stored on disk. They are stored in a directory > `remote-log-index-cache` under log dir. We plan to have a config for > that instead of default. We will have a configuration for that. > > 5105. The KIP uses local-log-start-offset and Earliest Local Offset in > different places. It would be useful to standardize the terminology. > > Sure. > > 5106. The section on "In BuildingRemoteLogAux state". It listed two options > without saying which option is chosen. > We already mentioned in the KIP that we chose option-2. > > 5107. Follower to leader transition: It has step 2, but not step 1. > Step-1 is there but it is not explicitly highlighted. It is previous > table to step-2. > > 5108. If a consumer fetches from the remote data and the remote storage is > not available, what error code is used in the fetch response? > > Good point. We have not yet defined the error for this case. We need > to define an error message and send the same in fetch response. > > 5109. "ListOffsets: For timestamps >= 0, it returns the first message > offset whose timestamp is >= to the given timestamp in the request. That > means it checks in remote log time indexes first, after which local log > time indexes are checked." Could you document which method in RLMM is used > for this? > > Okay. > > 5110. Stopreplica: "it sets all the remote log segment metadata of that > partition with a delete marker and publishes them to RLMM." This seems > outdated given the new topic deletion logic. > > Will update with KIP-516 related points. > > 5111. "RLM follower fetches the earliest offset for the earliest leader > epoch by calling RLMM.earliestLogOffset(TopicPartition topicPartition, int > leaderEpoch) and updates that as the log start offset." Do we need that > since replication propagates logStartOffset already? > > Good point. Right, existing replication protocol takes care of > updating the followers’s log start offset received from the leader. > > 5112. Is the default maxWaitMs of 500ms enough for fetching from remote > storage? > > Remote reads may fail within the current default wait time, but > subsequent fetches would be able to serve as that data is stored in > the local cache. This cache is currently implemented in RSMs. But we > plan to pull this into the remote log messaging layer in future. > > 5113. "Committed offsets can be stored in a local file to avoid reading the > messages again when a broker is restarted." Could you describe the format > and the location of the file? Also, could the same message be processed by > RLMM again after broker restart? If so, how do we handle that? > > Sure, we will update in the KIP. > > 5114. Message format > 5114.1 There are two records named RemoteLogSegmentMetadataRecord with > apiKey 0 and 1. > > Nice catch, that was a typo. Fixed in the wiki. > > 5114.2 RemoteLogSegmentMetadataRecord: Could we document whether endOffset > is inclusive/exclusive? > It is inclusive, will update. > > 5114.3 RemoteLogSegmentMetadataRecord: Could you explain LeaderEpoch a bit > more? Is that the epoch of the leader when it copies the segment to remote > storage? Also, how will this field be used? > > Right, this is the leader epoch of the broker which copied this > segment. This is helpful in reason about which broker copied the > segment to remote storage. > > 5114.4 EventTimestamp: Could you explain this a bit more? Each record in > Kafka already has a timestamp field. Could we just use that? > > This is the timestamp at which the respective event occurred. Added > this to RemoteLogSegmentMetadata as RLMM can be any other > implementation. We thought about that but it looked cleaner to use at > the message structure level instead of getting that from the consumer > record and using that to build the respective event. > > > 5114.5 SegmentSizeInBytes: Could this just be int32? > > Right, it looks like config allows only int value >= 14. > > 5115. RemoteLogCleaner(RLC): This could be confused with the log cleaner > for compaction. Perhaps it can be renamed to sth like > RemotePartitionRemover. > > I am fine with RemotePartitionRemover or RemoteLogDeletionManager(we > have other manager classes like RLM, RLMM). > > 5116. "RLC receives the delete_partition_marked and processes it if it is > not yet processed earlier." How does it know whether > delete_partition_marked has been processed earlier? > > This is to handle duplicate delete_partition_marked events. RLC > internally maintains a state for the delete_partition events and if it > already has an existing event then it ignores if it is already being > processed. > > 5117. Should we add a new MessageFormatter to read the tier metadata topic? > > Right, this is in plan but did not mention it in the KIP. This will be > useful for debugging purposes too. > > 5118. "Maximum remote log reader thread pool task queue size. If the task > queue is full, broker will stop reading remote log segments." What do we > return to the fetch request in this case? > > We return an error response for that partition. > > 5119. It would be useful to list all things not supported in the first > version in a Future work or Limitations section. For example, compacted > topic, JBOD, changing remote.log.storage.enable from true to false, etc. > > We already have a non-goals section which is filled with some of these > details. Do we need another limitations section? > > Thanks, > Satish. > > On Wed, Nov 4, 2020 at 11:27 PM Jun Rao <j...@confluent.io> wrote: > > > > Hi, Satish, > > > > Thanks for the updated KIP. A few more comments below. > > > > 605.2 "Build the local leader epoch cache by cutting the leader epoch > > sequence received from remote storage to [LSO, ELO]." I mentioned an > issue > > earlier. Suppose the leader's local start offset is 100. The follower > finds > > a remote segment covering offset range [80, 120). The producerState with > > this remote segment is up to offset 120. To trim the producerState to > > offset 100 requires more work since one needs to download the previous > > producerState up to offset 80 and then replay the messages from 80 to > 100. > > It seems that it's simpler in this case for the follower just to take the > > remote segment as it is and start fetching from offset 120. > > > > 5016. Just to echo what Kowshik was saying. It seems that > > RLMM.onPartitionLeadershipChanges() is only called on the replicas for a > > partition, not on the replicas for the __remote_log_segment_metadata > > partition. It's not clear how the leader of __remote_log_segment_metadata > > obtains the metadata for remote segments for deletion. > > > > 5100. KIP-516 has been accepted and is being implemented now. Could you > > update the KIP based on topicID? > > > > 5101. RLMM: It would be useful to clarify how the following two APIs are > > used. According to the wiki, the former is used for topic deletion and > the > > latter is used for retention. It seems that retention should use the > former > > since remote segments without a matching epoch in the leader (potentially > > due to unclean leader election) also need to be garbage collected. The > > latter seems to be used for the new leader to determine the last tiered > > segment. > > default Iterator<RemoteLogSegmentMetadata> > > listRemoteLogSegments(TopicPartition topicPartition) > > Iterator<RemoteLogSegmentMetadata> > listRemoteLogSegments(TopicPartition > > topicPartition, long leaderEpoch); > > > > 5102. RSM: > > 5102.1 For methods like fetchLogSegmentData(), it seems that they can > > use RemoteLogSegmentId instead of RemoteLogSegmentMetadata. > > 5102.2 In fetchLogSegmentData(), should we use long instead of Long? > > 5102.3 Why only some of the methods have default implementation and > others > > don't? > > 5102.4. Could we define RemoteLogSegmentMetadataUpdate > > and DeletePartitionUpdate? > > 5102.5 LogSegmentData: It seems that it's easier to pass > > in leaderEpochIndex as a ByteBuffer or byte array than a file since it > will > > be generated in memory. > > 5102.6 RemoteLogSegmentMetadata: It seems that it needs both baseOffset > and > > startOffset. For example, deleteRecords() could move the startOffset to > the > > middle of a segment. If we copy the full segment to remote storage, the > > baseOffset and the startOffset will be different. > > 5102.7 Could we define all the public methods for > RemoteLogSegmentMetadata > > and LogSegmentData? > > 5102.8 Could we document whether endOffset in RemoteLogSegmentMetadata is > > inclusive/exclusive? > > > > 5103. configs: > > 5103.1 Could we define the default value of non-required configs (e.g the > > size of new thread pools)? > > 5103.2 It seems that local.log.retention.ms should default to > retention.ms, > > instead of remote.log.retention.minutes. Similarly, it seems > > that local.log.retention.bytes should default to segment.bytes. > > 5103.3 remote.log.manager.thread.pool.size: The description says "used in > > scheduling tasks to copy segments, fetch remote log indexes and clean up > > remote log segments". However, there is a separate > > config remote.log.reader.threads for fetching remote data. It's weird to > > fetch remote index and log in different thread pools since both are used > > for serving fetch requests. > > 5103.4 remote.log.manager.task.interval.ms: Is that the amount of time > to > > back off when there is no work to do? If so, perhaps it can be renamed as > > backoff.ms. > > 5103.5 Are rlm_process_interval_ms and rlm_retry_interval_ms configs? If > > so, they need to be listed in this section. > > > > 5104. "RLM maintains a bounded cache(possibly LRU) of the index files of > > remote log segments to avoid multiple index fetches from the remote > > storage." Is the RLM in memory or on disk? If on disk, where is it > stored? > > Do we need a configuration to bound the size? > > > > 5105. The KIP uses local-log-start-offset and Earliest Local Offset in > > different places. It would be useful to standardize the terminology. > > > > 5106. The section on "In BuildingRemoteLogAux state". It listed two > options > > without saying which option is chosen. > > > > 5107. Follower to leader transition: It has step 2, but not step 1. > > > > 5108. If a consumer fetches from the remote data and the remote storage > is > > not available, what error code is used in the fetch response? > > > > 5109. "ListOffsets: For timestamps >= 0, it returns the first message > > offset whose timestamp is >= to the given timestamp in the request. That > > means it checks in remote log time indexes first, after which local log > > time indexes are checked." Could you document which method in RLMM is > used > > for this? > > > > 5110. Stopreplica: "it sets all the remote log segment metadata of that > > partition with a delete marker and publishes them to RLMM." This seems > > outdated given the new topic deletion logic. > > > > 5111. "RLM follower fetches the earliest offset for the earliest leader > > epoch by calling RLMM.earliestLogOffset(TopicPartition topicPartition, > int > > leaderEpoch) and updates that as the log start offset." Do we need that > > since replication propagates logStartOffset already? > > > > 5112. Is the default maxWaitMs of 500ms enough for fetching from remote > > storage? > > > > 5113. "Committed offsets can be stored in a local file to avoid reading > the > > messages again when a broker is restarted." Could you describe the format > > and the location of the file? Also, could the same message be processed > by > > RLMM again after broker restart? If so, how do we handle that? > > > > 5114. Message format > > 5114.1 There are two records named RemoteLogSegmentMetadataRecord with > > apiKey 0 and 1. > > 5114.2 RemoteLogSegmentMetadataRecord: Could we document whether > endOffset > > is inclusive/exclusive? > > 5114.3 RemoteLogSegmentMetadataRecord: Could you explain LeaderEpoch a > bit > > more? Is that the epoch of the leader when it copies the segment to > remote > > storage? Also, how will this field be used? > > 5114.4 EventTimestamp: Could you explain this a bit more? Each record in > > Kafka already has a timestamp field. Could we just use that? > > 5114.5 SegmentSizeInBytes: Could this just be int32? > > > > 5115. RemoteLogCleaner(RLC): This could be confused with the log cleaner > > for compaction. Perhaps it can be renamed to sth like > > RemotePartitionRemover. > > > > 5116. "RLC receives the delete_partition_marked and processes it if it is > > not yet processed earlier." How does it know whether > > delete_partition_marked has been processed earlier? > > > > 5117. Should we add a new MessageFormatter to read the tier metadata > topic? > > > > 5118. "Maximum remote log reader thread pool task queue size. If the task > > queue is full, broker will stop reading remote log segments." What do we > > return to the fetch request in this case? > > > > 5119. It would be useful to list all things not supported in the first > > version in a Future work or Limitations section. For example, compacted > > topic, JBOD, changing remote.log.storage.enable from true to false, etc. > > > > Thanks, > > > > Jun > > > > On Tue, Oct 27, 2020 at 5:57 PM Kowshik Prakasam <kpraka...@confluent.io > > > > wrote: > > > > > Hi Satish, > > > > > > Thanks for the updates to the KIP. Here are my first batch of > > > comments/suggestions on the latest version of the KIP. > > > > > > 5012. In the RemoteStorageManager interface, there is an API defined > for > > > each file type. For example, fetchOffsetIndex, fetchTimestampIndex > etc. To > > > avoid the duplication, I'd suggest we can instead have a FileType enum > and > > > a common get API based on the FileType. > > > > > > 5013. There are some references to the Google doc in the KIP. I wasn't > sure > > > if the Google doc is expected to be in sync with the contents of the > wiki. > > > Going forward, it seems easier if just the KIP is maintained as the > source > > > of truth. In this regard, could you please move all the references to > the > > > Google doc, maybe to a separate References section at the bottom of the > > > KIP? > > > > > > 5014. There are some TODO sections in the KIP. Would these be filled > up in > > > future iterations? > > > > > > 5015. Under "Topic deletion lifecycle", I'm trying to understand why > do we > > > need delete_partition_marked as well as the delete_partition_started > > > messages. I couldn't spot a drawback if supposing we simplified the > design > > > such that the controller would only write delete_partition_started > message, > > > and RemoteLogCleaner (RLC) instance picks it up for processing. What > am I > > > missing? > > > > > > 5016. Under "Topic deletion lifecycle", step (4) is mentioned as "RLC > gets > > > all the remote log segments for the partition and each of these remote > log > > > segments is deleted with the next steps.". Since the RLC instance runs > on > > > each tier topic partition leader, how does the RLC then get the list of > > > remote log segments to be deleted? It will be useful to add that > detail to > > > the KIP. > > > > > > 5017. Under "Public Interfaces -> Configs", there is a line mentioning > "We > > > will support flipping remote.log.storage.enable in next versions." It > will > > > be useful to mention this in the "Future Work" section of the KIP too. > > > > > > 5018. The KIP introduces a number of configuration parameters. It will > be > > > useful to mention in the KIP if the user should assume these as static > > > configuration in the server.properties file, or dynamic configuration > which > > > can be modified without restarting the broker. > > > > > > 5019. Maybe this is planned as a future update to the KIP, but I > thought > > > I'd mention it here. Could you please add details to the KIP on why > RocksDB > > > was chosen as the default cache implementation of RLMM, and how it is > going > > > to be used? Were alternatives compared/considered? For example, it > would be > > > useful to explain/evaluate the following: 1) debuggability of the > RocksDB > > > JNI interface, 2) performance, 3) portability across platforms and 4) > > > interface parity of RocksDB’s JNI api with it's underlying C/C++ api. > > > > > > 5020. Following up on (5019), for the RocksDB cache, it will be useful > to > > > explain the relationship/mapping between the following in the KIP: 1) > # of > > > tiered partitions, 2) # of partitions of metadata topic > > > __remote_log_metadata and 3) # of RocksDB instances. i.e. is the plan > to > > > have a RocksDB instance per tiered partition, or per metadata topic > > > partition, or just 1 for per broker? > > > > > > 5021. I was looking at the implementation prototype (PR link: > > > https://github.com/apache/kafka/pull/7561). It seems that a boolean > > > attribute is being introduced into the Log layer to check if remote log > > > capability is enabled. While the boolean footprint is small at the > moment, > > > this can easily grow in the future and become harder to > > > test/maintain, considering that the Log layer is already pretty > complex. We > > > should start thinking about how to manage such changes to the Log layer > > > (for the purpose of improved testability, better separation of > concerns and > > > readability). One proposal I have is to take a step back and define a > > > higher level Log interface. Then, the Broker code can be changed to use > > > this interface. It can be changed such that only a handle to the > interface > > > is exposed to other components (such as LogCleaner, ReplicaManager > etc.) > > > and not the underlying Log object. This approach keeps the user of the > Log > > > layer agnostic of the whereabouts of the data. Underneath the > interface, > > > the implementing classes can completely separate local log capabilities > > > from the remote log. For example, the Log class can be simplified to > only > > > manage logic surrounding local log segments and metadata. > Additionally, a > > > wrapper class can be provided (implementing the higher level Log > interface) > > > which will contain any/all logic surrounding tiered data. The wrapper > > > class will wrap around an instance of the Log class delegating the > local > > > log logic to it. Finally, a handle to the wrapper class can be exposed > to > > > the other components wherever they need a handle to the higher level > Log > > > interface. > > > > > > > > > Cheers, > > > Kowshik > > > > > > On Mon, Oct 26, 2020 at 9:52 PM Satish Duggana < > satish.dugg...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > KIP is updated with 1) topic deletion lifecycle and its related items > > > > 2) Protocol changes(mainly related to ListOffsets) and other minor > > > > changes. > > > > Please go through them and let us know your comments. > > > > > > > > Thanks, > > > > Satish. > > > > > > > > On Mon, Sep 28, 2020 at 9:10 PM Satish Duggana < > satish.dugg...@gmail.com > > > > > > > > wrote: > > > > > > > > > > Hi Dhruvil, > > > > > Thanks for looking into the KIP and sending your comments. Sorry > for > > > > > the late reply, missed it in the mail thread. > > > > > > > > > > 1. Could you describe how retention would work with this KIP and > which > > > > > threads are responsible for driving this work? I believe there are > 3 > > > > kinds > > > > > of retention processes we are looking at: > > > > > (a) Regular retention for data in tiered storage as per > configured ` > > > > > retention.ms` / `retention.bytes`. > > > > > (b) Local retention for data in local storage as per configured ` > > > > > local.log.retention.ms` / `local.log.retention.bytes` > > > > > (c) Possibly regular retention for data in local storage, if the > > > > tiering > > > > > task is lagging or for data that is below the log start offset. > > > > > > > > > > Local log retention is done by the existing log cleanup tasks. > These > > > > > are not done for segments that are not yet copied to remote > storage. > > > > > Remote log cleanup is done by the leader partition’s RLMTask. > > > > > > > > > > 2. When does a segment become eligible to be tiered? Is it as soon > as > > > the > > > > > segment is rolled and the end offset is less than the last stable > > > offset > > > > as > > > > > mentioned in the KIP? I wonder if we need to consider other > parameters > > > > too, > > > > > like the highwatermark so that we are guaranteed that what we are > > > tiering > > > > > has been committed to the log and accepted by the ISR. > > > > > > > > > > AFAIK, last stable offset is always <= highwatermark. This will > make > > > > > sure we are always tiering the message segments which have been > > > > > accepted by ISR and transactionally completed. > > > > > > > > > > > > > > > 3. The section on "Follower Fetch Scenarios" is useful but is a bit > > > > > difficult to parse at the moment. It would be useful to summarize > the > > > > > changes we need in the ReplicaFetcher. > > > > > > > > > > It may become difficult for users to read/follow if we add code > changes > > > > here. > > > > > > > > > > 4. Related to the above, it's a bit unclear how we are planning on > > > > > restoring the producer state for a new replica. Could you expand on > > > that? > > > > > > > > > > It is mentioned in the KIP BuildingRemoteLogAuxState is introduced > to > > > > > build the state like leader epoch sequence and producer snapshots > > > > > before it starts fetching the data from the leader. We will make it > > > > > clear in the KIP. > > > > > > > > > > > > > > > 5. Similarly, it would be worth summarizing the behavior on unclean > > > > leader > > > > > election. There are several scenarios to consider here: data loss > from > > > > > local log, data loss from remote log, data loss from metadata > topic, > > > etc. > > > > > It's worth describing these in detail. > > > > > > > > > > We mentioned the cases about unclean leader election in the > follower > > > > > fetch scenarios. > > > > > If there are errors while fetching data from remote store or > metadata > > > > > store, it will work the same way as it works with local log. It > > > > > returns the error back to the caller. Please let us know if I am > > > > > missing your point here. > > > > > > > > > > > > > > > 7. For a READ_COMMITTED FetchRequest, how do we retrieve and > return the > > > > > aborted transaction metadata? > > > > > > > > > > When a fetch for a remote log is accessed, we will fetch aborted > > > > > transactions along with the segment if it is not found in the local > > > > > index cache. This includes the case of transaction index not > existing > > > > > in the remote log segment. That means, the cache entry can be > empty or > > > > > have a list of aborted transactions. > > > > > > > > > > > > > > > 8. The `LogSegmentData` class assumes that we have a log segment, > > > offset > > > > > index, time index, transaction index, producer snapshot and leader > > > epoch > > > > > index. How do we deal with cases where we do not have one or more > of > > > > these? > > > > > For example, we may not have a transaction index or producer > snapshot > > > > for a > > > > > particular segment. The former is optional, and the latter is only > kept > > > > for > > > > > up to the 3 latest segments. > > > > > > > > > > This is a good point, we discussed this in the last meeting. > > > > > Transaction index is optional and we will copy them only if it > exists. > > > > > We want to keep all the producer snapshots at each log segment > rolling > > > > > and they can be removed if the log copying is successful and it > still > > > > > maintains the existing latest 3 segments, We only delete the > producer > > > > > snapshots which have been copied to remote log segments on leader. > > > > > Follower will keep the log segments beyond the segments which have > not > > > > > been copied to remote storage. We will update the KIP with these > > > > > details. > > > > > > > > > > Thanks, > > > > > Satish. > > > > > > > > > > On Thu, Sep 17, 2020 at 1:47 AM Dhruvil Shah <dhru...@confluent.io > > > > > > wrote: > > > > > > > > > > > > Hi Satish, Harsha, > > > > > > > > > > > > Thanks for the KIP. Few questions below: > > > > > > > > > > > > 1. Could you describe how retention would work with this KIP and > > > which > > > > > > threads are responsible for driving this work? I believe there > are 3 > > > > kinds > > > > > > of retention processes we are looking at: > > > > > > (a) Regular retention for data in tiered storage as per > configured > > > ` > > > > > > retention.ms` / `retention.bytes`. > > > > > > (b) Local retention for data in local storage as per > configured ` > > > > > > local.log.retention.ms` / `local.log.retention.bytes` > > > > > > (c) Possibly regular retention for data in local storage, if > the > > > > tiering > > > > > > task is lagging or for data that is below the log start offset. > > > > > > > > > > > > 2. When does a segment become eligible to be tiered? Is it as > soon as > > > > the > > > > > > segment is rolled and the end offset is less than the last stable > > > > offset as > > > > > > mentioned in the KIP? I wonder if we need to consider other > > > parameters > > > > too, > > > > > > like the highwatermark so that we are guaranteed that what we are > > > > tiering > > > > > > has been committed to the log and accepted by the ISR. > > > > > > > > > > > > 3. The section on "Follower Fetch Scenarios" is useful but is a > bit > > > > > > difficult to parse at the moment. It would be useful to > summarize the > > > > > > changes we need in the ReplicaFetcher. > > > > > > > > > > > > 4. Related to the above, it's a bit unclear how we are planning > on > > > > > > restoring the producer state for a new replica. Could you expand > on > > > > that? > > > > > > > > > > > > 5. Similarly, it would be worth summarizing the behavior on > unclean > > > > leader > > > > > > election. There are several scenarios to consider here: data loss > > > from > > > > > > local log, data loss from remote log, data loss from metadata > topic, > > > > etc. > > > > > > It's worth describing these in detail. > > > > > > > > > > > > 6. It would be useful to add details about how we plan on using > > > > RocksDB in > > > > > > the default implementation of `RemoteLogMetadataManager`. > > > > > > > > > > > > 7. For a READ_COMMITTED FetchRequest, how do we retrieve and > return > > > the > > > > > > aborted transaction metadata? > > > > > > > > > > > > 8. The `LogSegmentData` class assumes that we have a log segment, > > > > offset > > > > > > index, time index, transaction index, producer snapshot and > leader > > > > epoch > > > > > > index. How do we deal with cases where we do not have one or > more of > > > > these? > > > > > > For example, we may not have a transaction index or producer > snapshot > > > > for a > > > > > > particular segment. The former is optional, and the latter is > only > > > > kept for > > > > > > up to the 3 latest segments. > > > > > > > > > > > > Thanks, > > > > > > Dhruvil > > > > > > > > > > > > On Mon, Sep 7, 2020 at 6:54 PM Harsha Ch <harsha...@gmail.com> > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > We are all working through the last meeting feedback. I'll > cancel > > > the > > > > > > > tomorrow 's meeting and we can meanwhile continue our > discussion in > > > > mailing > > > > > > > list. We can start the regular meeting from next week onwards. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Harsha > > > > > > > > > > > > > > On Fri, Sep 04, 2020 at 8:41 AM, Satish Duggana < > > > > satish.dugg...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jun, > > > > > > > > Thanks for your thorough review and comments. Please find the > > > > inline > > > > > > > > replies below. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 600. The topic deletion logic needs more details. > > > > > > > > 600.1 The KIP mentions "The controller considers the topic > > > > partition is > > > > > > > > deleted only when it determines that there are no log > segments > > > for > > > > that > > > > > > > > topic partition by using RLMM". How is this done? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It uses RLMM#listSegments() returns all the segments for the > > > given > > > > topic > > > > > > > > partition. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 600.2 "If the delete option is enabled then the leader will > stop > > > > RLM task > > > > > > > > and stop processing and it sets all the remote log segment > > > > metadata of > > > > > > > > that partition with a delete marker and publishes them to > RLMM." > > > We > > > > > > > > discussed this earlier. When a topic is being deleted, there > may > > > > not be a > > > > > > > > leader for the deleted partition. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is a good point. As suggested in the meeting, we will > add a > > > > separate > > > > > > > > section for topic/partition deletion lifecycle and this > scenario > > > > will be > > > > > > > > addressed. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 601. Unclean leader election > > > > > > > > 601.1 Scenario 1: new empty follower > > > > > > > > After step 1, the follower restores up to offset 3. So why > does > > > it > > > > have > > > > > > > > LE-2 <https://issues.apache.org/jira/browse/LE-2> at offset > 5? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nice catch. It was showing the leader epoch fetched from the > > > remote > > > > > > > > storage. It should be shown with the truncated till offset 3. > > > > Updated the > > > > > > > > KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 601.2 senario 5: After Step 3, leader A has inconsistent data > > > > between its > > > > > > > > local and the tiered data. For example. offset 3 has msg 3 > LE-0 > > > > <https://issues.apache.org/jira/browse/LE-0> locally, > > > > > > > > but msg 5 LE-1 <https://issues.apache.org/jira/browse/LE-1> > in > > > > the remote store. While it's ok for the unclean leader > > > > > > > > to lose data, it should still return consistent data, whether > > > it's > > > > from > > > > > > > > the local or the remote store. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There is no inconsistency here as LE-0 > > > > <https://issues.apache.org/jira/browse/LE-0> offsets are [0, 4] and > LE-2 > > > > <https://issues.apache.org/jira/browse/LE-2>: > > > > > > > > [5, ]. It will always get the right records for the given > offset > > > > and > > > > > > > > leader epoch. In case of remote, RSM is invoked to get the > remote > > > > log > > > > > > > > segment that contains the given offset with the leader epoch. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 601.4 It seems that retention is based on > > > > > > > > listRemoteLogSegments(TopicPartition topicPartition, long > > > > leaderEpoch). > > > > > > > > When there is an unclean leader election, it's possible for > the > > > new > > > > > > > leader > > > > > > > > to not to include certain epochs in its epoch cache. How are > > > remote > > > > > > > > segments associated with those epochs being cleaned? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That is a good point. This leader will also cleanup the > epochs > > > > earlier to > > > > > > > > its start leader epoch and delete those segments. It gets the > > > > earliest > > > > > > > > epoch for a partition and starts deleting segments from that > > > leader > > > > > > > epoch. > > > > > > > > We need one more API in RLMM to get the earliest leader > epoch. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 601.5 The KIP discusses the handling of unclean leader > elections > > > > for user > > > > > > > > topics. What about unclean leader elections on > > > > > > > > __remote_log_segment_metadata? > > > > > > > > This is the same as other system topics like > consumer_offsets, > > > > > > > > __transaction_state topics. As discussed in the meeting, we > will > > > > add the > > > > > > > > behavior of __remote_log_segment_metadata topic’s unclean > leader > > > > > > > > truncation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 602. It would be useful to clarify the limitations in the > initial > > > > > > > release. > > > > > > > > The KIP mentions not supporting compacted topics. What about > JBOD > > > > and > > > > > > > > changing the configuration of a topic from delete to compact > > > after > > > > > > > remote. > > > > > > > > log. storage. enable ( http://remote.log.storage.enable/ ) > is > > > > enabled? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This was updated in the KIP earlier. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 603. RLM leader tasks: > > > > > > > > 603.1"It checks for rolled over LogSegments (which have the > last > > > > message > > > > > > > > offset less than last stable offset of that topic partition) > and > > > > copies > > > > > > > > them along with their offset/time/transaction indexes and > leader > > > > epoch > > > > > > > > cache to the remote tier." It needs to copy the producer > snapshot > > > > too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Right. It copies producer snapshots too as mentioned in > > > > LogSegmentData. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 603.2 "Local logs are not cleaned up till those segments are > > > copied > > > > > > > > successfully to remote even though their retention time/size > is > > > > reached" > > > > > > > > This seems weird. If the tiering stops because the remote > store > > > is > > > > not > > > > > > > > available, we don't want the local data to grow forever. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It was clarified in the discussion that the comment was more > > > about > > > > the > > > > > > > > local storage goes beyond the log.retention. The above > statement > > > > is about > > > > > > > > local.log.retention but not for the complete log.retention. > When > > > it > > > > > > > > reaches the log.retention then it will delete the local logs > even > > > > though > > > > > > > > those are not copied to remote storage. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 604. "RLM maintains a bounded cache(possibly LRU) of the > index > > > > files of > > > > > > > > remote log segments to avoid multiple index fetches from the > > > remote > > > > > > > > storage. These indexes can be used in the same way as local > > > segment > > > > > > > > indexes are used." Could you provide more details on this? > Are > > > the > > > > > > > indexes > > > > > > > > cached in memory or on disk? If on disk, where are they > stored? > > > > Are the > > > > > > > > cached indexes bound by a certain size? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > These are cached on disk and stored in log.dir with a name > > > > > > > > “__remote_log_index_cache”. They are bound by the total size. > > > This > > > > will > > > > > > > be > > > > > > > > exposed as a user configuration, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 605. BuildingRemoteLogAux > > > > > > > > 605.1 In this section, two options are listed. Which one is > > > chosen? > > > > > > > > Option-2, updated the KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 605.2 In option 2, it says "Build the local leader epoch > cache by > > > > cutting > > > > > > > > the leader epoch sequence received from remote storage to > [LSO, > > > > ELO]. > > > > > > > (LSO > > > > > > > > > > > > > > > > = log start offset)." We need to do the same thing for the > > > producer > > > > > > > > snapshot. However, it's hard to cut the producer snapshot to > an > > > > earlier > > > > > > > > offset. Another option is to simply take the lastOffset from > the > > > > remote > > > > > > > > segment and use that as the starting fetch offset in the > > > follower. > > > > This > > > > > > > > avoids the need for cutting. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Right, this was mentioned in the “transactional support” > section > > > > about > > > > > > > > adding these details. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 606. ListOffsets: Since we need a version bump, could you > > > document > > > > it > > > > > > > > under a protocol change section? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sure, we will update the KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 607. "LogStartOffset of a topic can point to either of local > > > > segment or > > > > > > > > remote segment but it is initialised and maintained in the > Log > > > > class like > > > > > > > > now. This is already maintained in `Log` class while loading > the > > > > logs and > > > > > > > > it can also be fetched from RemoteLogMetadataManager." What > will > > > > happen > > > > > > > to > > > > > > > > the existing logic (e.g. log recovery) that currently > depends on > > > > > > > > logStartOffset but assumes it's local? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > They use a field called localLogStartOffset which is the > local > > > log > > > > start > > > > > > > > offset.. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 608. Handle expired remote segment: How does it pick up new > > > > > > > logStartOffset > > > > > > > > from deleteRecords? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Good point. This was not addressed in the KIP. Will update > the > > > KIP > > > > on how > > > > > > > > the RLM task handles this scenario. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 609. RLMM message format: > > > > > > > > 609.1 It includes both MaxTimestamp and EventTimestamp. Where > > > does > > > > it get > > > > > > > > both since the message in the log only contains one > timestamp? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > `EventTimeStamp` is the timestamp at which that segment > metadata > > > > event is > > > > > > > > generated. This is more for audits. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 609.2 If we change just the state (e.g. to DELETE_STARTED), > it > > > > seems it's > > > > > > > > wasteful to have to include all other fields not changed. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is a good point. We thought about incremental updates. > But > > > we > > > > want > > > > > > > to > > > > > > > > make sure all the events are in the expected order and take > > > action > > > > based > > > > > > > > on the latest event. Will think through the approaches in > detail > > > > and > > > > > > > > update here. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 609.3 Could you document which process makes the following > > > > transitions > > > > > > > > DELETE_MARKED, DELETE_STARTED, DELETE_FINISHED? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Okay, will document more details. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 610. remote.log.reader.max.pending.tasks: "Maximum remote log > > > > reader > > > > > > > > thread pool task queue size. If the task queue is full, > broker > > > > will stop > > > > > > > > reading remote log segments." What does the broker do if the > > > queue > > > > is > > > > > > > > full? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It returns an error for this topic partition. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 611. What do we return if the request offset/epoch doesn't > exist > > > > in the > > > > > > > > following API? > > > > > > > > RemoteLogSegmentMetadata > remoteLogSegmentMetadata(TopicPartition > > > > > > > > topicPartition, long offset, int epochForOffset) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This returns null. But we prefer to update the return type as > > > > Optional > > > > > > > and > > > > > > > > return Empty if that does not exist. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Satish. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 1, 2020 at 9:45 AM Jun Rao < jun@ confluent. io > ( > > > > > > > > j...@confluent.io ) > wrote: > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > >> Hi, Satish, > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> Thanks for the updated KIP. Made another pass. A few more > > > comments > > > > > > > below. > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 600. The topic deletion logic needs more details. > > > > > > > >> 600.1 The KIP mentions "The controller considers the topic > > > > partition is > > > > > > > >> deleted only when it determines that there are no log > segments > > > > for that > > > > > > > >> topic partition by using RLMM". How is this done? 600.2 "If > the > > > > delete > > > > > > > >> option is enabled then the leader will stop RLM task and > stop > > > > processing > > > > > > > >> and it sets all the remote log segment metadata of that > > > partition > > > > with a > > > > > > > >> delete marker and publishes them to RLMM." We discussed this > > > > earlier. > > > > > > > When > > > > > > > >> a topic is being deleted, there may not be a leader for the > > > > deleted > > > > > > > >> partition. > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 601. Unclean leader election > > > > > > > >> 601.1 Scenario 1: new empty follower > > > > > > > >> After step 1, the follower restores up to offset 3. So why > does > > > > it have > > > > > > > >> LE-2 <https://issues.apache.org/jira/browse/LE-2> at > offset 5? > > > > > > > >> 601.2 senario 5: After Step 3, leader A has inconsistent > data > > > > between > > > > > > > its > > > > > > > >> local and the tiered data. For example. offset 3 has msg 3 > LE-0 > > > > <https://issues.apache.org/jira/browse/LE-0> locally, > > > > > > > >> but msg 5 LE-1 <https://issues.apache.org/jira/browse/LE-1> > in > > > > the remote store. While it's ok for the unclean leader > > > > > > > >> to lose data, it should still return consistent data, > whether > > > > it's from > > > > > > > >> the local or the remote store. > > > > > > > >> 601.3 The follower picks up log start offset using the > following > > > > api. > > > > > > > >> Suppose that we have 3 remote segments (LE, > SegmentStartOffset) > > > > as (2, > > > > > > > >> 10), > > > > > > > >> (3, 20) and (7, 15) due to an unclean leader election. > Using the > > > > > > > following > > > > > > > >> api will cause logStartOffset to go backward from 20 to 15. > How > > > > do we > > > > > > > >> prevent that? > > > > > > > >> earliestLogOffset(TopicPartition topicPartition, int > > > leaderEpoch) > > > > 601.4 > > > > > > > It > > > > > > > >> seems that retention is based on > > > > > > > >> listRemoteLogSegments(TopicPartition topicPartition, long > > > > leaderEpoch). > > > > > > > >> When there is an unclean leader election, it's possible for > the > > > > new > > > > > > > leader > > > > > > > >> to not to include certain epochs in its epoch cache. How are > > > > remote > > > > > > > >> segments associated with those epochs being cleaned? 601.5 > The > > > KIP > > > > > > > >> discusses the handling of unclean leader elections for user > > > > topics. What > > > > > > > >> about unclean leader elections on > > > > > > > >> __remote_log_segment_metadata? > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 602. It would be useful to clarify the limitations in the > > > initial > > > > > > > release. > > > > > > > >> The KIP mentions not supporting compacted topics. What about > > > JBOD > > > > and > > > > > > > >> changing the configuration of a topic from delete to compact > > > after > > > > > > > remote. > > > > > > > >> log. storage. enable ( http://remote.log.storage.enable/ ) > is > > > > enabled? > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 603. RLM leader tasks: > > > > > > > >> 603.1"It checks for rolled over LogSegments (which have the > last > > > > message > > > > > > > >> offset less than last stable offset of that topic > partition) and > > > > copies > > > > > > > >> them along with their offset/time/transaction indexes and > leader > > > > epoch > > > > > > > >> cache to the remote tier." It needs to copy the producer > > > snapshot > > > > too. > > > > > > > >> 603.2 "Local logs are not cleaned up till those segments are > > > > copied > > > > > > > >> successfully to remote even though their retention > time/size is > > > > reached" > > > > > > > >> This seems weird. If the tiering stops because the remote > store > > > > is not > > > > > > > >> available, we don't want the local data to grow forever. > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 604. "RLM maintains a bounded cache(possibly LRU) of the > index > > > > files of > > > > > > > >> remote log segments to avoid multiple index fetches from the > > > > remote > > > > > > > >> storage. These indexes can be used in the same way as local > > > > segment > > > > > > > >> indexes are used." Could you provide more details on this? > Are > > > the > > > > > > > indexes > > > > > > > >> cached in memory or on disk? If on disk, where are they > stored? > > > > Are the > > > > > > > >> cached indexes bound by a certain size? > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 605. BuildingRemoteLogAux > > > > > > > >> 605.1 In this section, two options are listed. Which one is > > > > chosen? > > > > > > > 605.2 > > > > > > > >> In option 2, it says "Build the local leader epoch cache by > > > > cutting the > > > > > > > >> leader epoch sequence received from remote storage to [LSO, > > > ELO]. > > > > (LSO > > > > > > > >> = log start offset)." We need to do the same thing for the > > > > producer > > > > > > > >> snapshot. However, it's hard to cut the producer snapshot > to an > > > > earlier > > > > > > > >> offset. Another option is to simply take the lastOffset > from the > > > > remote > > > > > > > >> segment and use that as the starting fetch offset in the > > > > follower. This > > > > > > > >> avoids the need for cutting. > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 606. ListOffsets: Since we need a version bump, could you > > > > document it > > > > > > > >> under a protocol change section? > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 607. "LogStartOffset of a topic can point to either of local > > > > segment or > > > > > > > >> remote segment but it is initialised and maintained in the > Log > > > > class > > > > > > > like > > > > > > > >> now. This is already maintained in `Log` class while > loading the > > > > logs > > > > > > > and > > > > > > > >> it can also be fetched from RemoteLogMetadataManager." What > will > > > > happen > > > > > > > to > > > > > > > >> the existing logic (e.g. log recovery) that currently > depends on > > > > > > > >> logStartOffset but assumes it's local? > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 608. Handle expired remote segment: How does it pick up new > > > > > > > logStartOffset > > > > > > > >> from deleteRecords? > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 609. RLMM message format: > > > > > > > >> 609.1 It includes both MaxTimestamp and EventTimestamp. > Where > > > > does it > > > > > > > get > > > > > > > >> both since the message in the log only contains one > timestamp? > > > > 609.2 If > > > > > > > we > > > > > > > >> change just the state (e.g. to DELETE_STARTED), it seems > it's > > > > wasteful > > > > > > > to > > > > > > > >> have to include all other fields not changed. 609.3 Could > you > > > > document > > > > > > > >> which process makes the following transitions DELETE_MARKED, > > > > > > > >> DELETE_STARTED, DELETE_FINISHED? > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 610. remote.log.reader.max.pending.tasks: "Maximum remote > log > > > > reader > > > > > > > >> thread pool task queue size. If the task queue is full, > broker > > > > will stop > > > > > > > >> reading remote log segments." What does the broker do if the > > > > queue is > > > > > > > >> full? > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> 611. What do we return if the request offset/epoch doesn't > exist > > > > in the > > > > > > > >> following API? > > > > > > > >> RemoteLogSegmentMetadata > remoteLogSegmentMetadata(TopicPartition > > > > > > > >> topicPartition, long offset, int epochForOffset) > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> Jun > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> On Mon, Aug 31, 2020 at 11:19 AM Satish Duggana < satish. > > > duggana@ > > > > > > > gmail. com > > > > > > > >> ( satish.dugg...@gmail.com ) > wrote: > > > > > > > >> > > > > > > > >> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> KIP is updated with > > > > > > > >>> - Remote log segment metadata topic message format/schema. > > > > > > > >>> - Added remote log segment metadata state transitions and > > > > explained how > > > > > > > >>> the deletion of segments is handled, including the case of > > > > partition > > > > > > > >>> deletions. > > > > > > > >>> - Added a few more limitations in the "Non goals" section. > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> Thanks, > > > > > > > >>> Satish. > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> On Thu, Aug 27, 2020 at 12:42 AM Harsha Ch < harsha. ch@ > > > gmail. > > > > com ( > > > > > > > >>> harsha...@gmail.com ) > wrote: > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> Updated the KIP with Meeting Notes section > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ > > > > > > > KIP-405 <https://issues.apache.org/jira/browse/KIP-405> > > > > %3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-MeetingNotes > > > > > > > >>> ( > > > > > > > >>> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-MeetingNotes > > > > > > > >>> ) > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> On Tue, Aug 25, 2020 at 1:03 PM Jun Rao < jun@ > confluent. io > > > ( > > > > > > > >>>> j...@confluent.io ) > wrote: > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> Hi, Harsha, > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> Thanks for the summary. Could you add the summary and the > > > > recording > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> link to > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> the last section of > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ > > > > > > > Kafka+Improvement+Proposals > > > > > > > >>> ( > > > > > > > >>> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > >>> ) > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> ? > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> Jun > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> On Tue, Aug 25, 2020 at 11:12 AM Harsha Chintalapani < > kafka@ > > > > > > > harsha. io ( > > > > > > > >>>>> ka...@harsha.io ) > wrote: > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> Thanks everyone for attending the meeting today. > > > > > > > >>>>>> Here is the recording > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> https:/ / drive. google. com/ file/ d/ > > > > > > > 14PRM7U0OopOOrJR197VlqvRX5SXNtmKj/ view?usp=sharing > > > > > > > >>> ( > > > > > > > >>> > > > > > > > > > > > > > > > https://drive.google.com/file/d/14PRM7U0OopOOrJR197VlqvRX5SXNtmKj/view?usp=sharing > > > > > > > >>> ) > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> Notes: > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> 1. KIP is updated with follower fetch protocol and > ready to > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> reviewed > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> 2. Satish to capture schema of internal metadata topic > in > > > the > > > > KIP > > > > > > > >>>>>> 3. We will update the KIP with details of different > cases > > > > > > > >>>>>> 4. Test plan will be captured in a doc and will add to > the > > > KIP > > > > > > > >>>>>> 5. Add a section "Limitations" to capture the > capabilities > > > > that > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> will > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> be > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> introduced with this KIP and what will not be covered in > > > this > > > > KIP. > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> Please add to it I missed anything. Will produce a > formal > > > > meeting > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> notes > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> from next meeting onwards. > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> Thanks, > > > > > > > >>>>>> Harsha > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> On Mon, Aug 24, 2020 at 9:42 PM, Ying Zheng < yingz@ > uber. > > > > com. > > > > > > > invalid ( > > > > > > > >>>>>> yi...@uber.com.invalid ) > wrote: > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> We did some basic feature tests at Uber. The test > cases and > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> results are > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> shared in this google doc: > > > > > > > >>>>>>> https:/ / docs. google. com/ spreadsheets/ d/ ( > > > > > > > >>>>>>> https://docs.google.com/spreadsheets/d/ ) > > > > > > > >>>>>>> > > > 1XhNJqjzwXvMCcAOhEH0sSXU6RTvyoSf93DHF-YMfGLk/edit?usp=sharing > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> The performance test results were already shared in > the KIP > > > > last > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> month. > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> On Mon, Aug 24, 2020 at 11:10 AM Harsha Ch < harsha. > ch@ > > > > gmail. > > > > > > > com ( > > > > > > > >>>>>>> harsha...@gmail.com ) > > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> wrote: > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> "Understand commitments towards driving design & > > > > implementation of > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> the > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> KIP > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> further and how it aligns with participant interests in > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> contributing to > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> the > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> efforts (ex: in the context of Uber’s Q3/Q4 roadmap)." > What > > > > is that > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> about? > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> On Mon, Aug 24, 2020 at 11:05 AM Kowshik Prakasam < > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> kprakasam@ confluent. io ( kpraka...@confluent.io ) > > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> wrote: > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Hi Harsha, > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> The following google doc contains a proposal for > temporary > > > > agenda > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> for > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> the > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> KIP-405 <https://issues.apache.org/jira/browse/KIP-405> > < > > > > https:/ / issues. apache. org/ jira/ browse/ KIP-405 > > > > <https://issues.apache.org/jira/browse/KIP-405> ( > > > > > > > >>>>>>> https://issues.apache.org/jira/browse/KIP-405 ) > sync > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> meeting > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> tomorrow: > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> https:/ / docs. google. com/ document/ d/ ( > > > > > > > >>>>>>> https://docs.google.com/document/d/ ) > > > > > > > >>>>>>> 1pqo8X5LU8TpwfC_iqSuVPezhfCfhGkbGN2TqiPA3LBU/edit > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> . > > > > > > > >>>>>>> Please could you add it to the Google calendar invite? > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Thank you. > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Cheers, > > > > > > > >>>>>>> Kowshik > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> On Thu, Aug 20, 2020 at 10:58 AM Harsha Ch < harsha. > ch@ > > > > gmail. > > > > > > > com ( > > > > > > > >>>>>>> harsha...@gmail.com ) > > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> wrote: > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Hi All, > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Scheduled a meeting for Tuesday 9am - 10am. I can > record > > > and > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> upload for > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> community to be able to follow the discussion. > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Jun, please add the required folks on confluent side. > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Thanks, > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Harsha > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> On Thu, Aug 20, 2020 at 12:33 AM, Alexandre Dupriez < > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> alexandre.dupriez@ > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> gmail. com ( http://gmail.com/ ) > wrote: > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Hi Jun, > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Many thanks for your initiative. > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> If you like, I am happy to attend at the time you > > > suggested. > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Many thanks, > > > > > > > >>>>>>> Alexandre > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Le mer. 19 août 2020 à 22:00, Harsha Ch < harsha. ch@ > > > > gmail. com ( > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> harsha. > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> ch@ gmail. com ( c...@gmail.com ) ) > a écrit : > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Hi Jun, > > > > > > > >>>>>>> Thanks. This will help a lot. Tuesday will work for us. > > > > > > > >>>>>>> -Harsha > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> On Wed, Aug 19, 2020 at 1:24 PM Jun Rao < jun@ > confluent. > > > > io ( > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> jun@ > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> confluent. io ( http://confluent.io/ ) ) > wrote: > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Hi, Satish, Ying, Harsha, > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Do you think it would be useful to have a regular > virtual > > > > meeting > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> to > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>>> > > > > > > > >>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> discuss this KIP? The goal of the meeting will be > sharing > > > > > > > >>>>>>> design/development progress and discussing any open > issues > > > to > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> accelerate > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> this KIP. If so, will every Tuesday (from next week) > > > 9am-10am > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> PT > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> work for you? I can help set up a Zoom meeting, invite > > > > everyone who > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> might > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> be interested, have it recorded and shared, etc. > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Thanks, > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Jun > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> On Tue, Aug 18, 2020 at 11:01 AM Satish Duggana < > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> satish. duggana@ gmail. com ( satish. duggana@ gmail. > com > > > ( > > > > > > > >>>>>>> satish.dugg...@gmail.com ) ) > > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> wrote: > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Hi Kowshik, > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> Thanks for looking into the KIP and sending your > comments. > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> 5001. Under the section "Follower fetch protocol in > > > detail", > > > > the > > > > > > > >>>>>>> next-local-offset is the offset upto which the > segments are > > > > copied > > > > > > > >>>>>>> > > > > >
