Hey Alexey, Thanks for your review and the alternative approach. Here is my understanding of your patch. kafka's background threads are used to move data between replicas. When data movement is triggered, the log will be rolled and the new logs will be put in the new directory, and background threads will move segment from old directory to new directory.
It is important to note that KIP-112 is intended to work with KIP-113 to support JBOD. I think your solution is definitely simpler and better under the current kafka implementation that a broker will fail if any disk fails. But I am not sure if we want to allow broker to run with partial disks failure. Let's say the a replica is being moved from log_dir_old to log_dir_new and then log_dir_old stops working due to disk failure. How would your existing patch handles it? To make the scenario a bit more complicated, let's say the broker is shtudown, log_dir_old's disk fails, and the broker starts. In this case broker doesn't even know if log_dir_new has all the data needed for this replica. It becomes a problem if the broker is elected leader of this partition in this case. The solution presented in the KIP attempts to handle it by replacing replica in an atomic version fashion after the log in the new dir has fully caught up with the log in the old dir. At at time the log can be considered to exist on only one log directory. And to answer your question, yes topicpartition.log refers to topic-paritition/segment.log. Thanks, Dong On Fri, Jan 13, 2017 at 4:12 AM, Alexey Ozeritsky <aozerit...@yandex.ru> wrote: > Hi, > > We have the similar solution that have been working in production since > 2014. You can see it here: https://github.com/resetius/ka > fka/commit/20658593e246d2184906879defa2e763c4d413fb > The idea is very simple > 1. Disk balancer runs in a separate thread inside scheduler pool. > 2. It does not touch empty partitions > 3. Before it moves a partition it forcibly creates new segment on a > destination disk > 4. It moves segment by segment from new to old. > 5. Log class works with segments on both disks > > Your approach seems too complicated, moreover it means that you have to > patch different components of the system > Could you clarify what do you mean by topicPartition.log? Is it > topic-paritition/segment.log ? > > 12.01.2017, 21:47, "Dong Lin" <lindon...@gmail.com>: > > Hi all, > > > > We created KIP-113: Support replicas movement between log directories. > > Please find the KIP wiki in the link > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-113% > 3A+Support+replicas+movement+between+log+directories > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-113% > 3A+Support+replicas+movement+between+log+directories>.* > > > > This KIP is related to KIP-112 > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-112% > 3A+Handle+disk+failure+for+JBOD>: > > Handle disk failure for JBOD. They are needed in order to support JBOD in > > Kafka. Please help review the KIP. You feedback is appreciated! > > > > Thanks, > > Dong >
