If you want to implement version control at the tablet level, then first, you need to design a way for FE to be able to clearly distinguish between “missing versions” and “unimported versions”.
Let's say two tablet A and B under the same partition. tablet A is with version 3 and tablet B is with version 2. Then how can we determine if B is a missing version of a tablet, or a tablet that has not been imported. Before implementing this part of the code, I recommend a more detailed design to ensure that the currently existing feature does not receive any impact or has a manageable impact. -- 此致！Best Regards 陈明雨 Mingyu Chen Email: chenmin...@apache.org At 2022-02-22 13:28:00, "王博" <wangbo13...@gmail.com> wrote: >Problems and solutions are relatively clear. >I would like to do some discussion on the problems this solution may face. >First, It seems that you want to commit transactions based on tablet >granularity, I think multiple tablets in one partition should >keep consistency in one steam load. >In one stream load, if some tablets commits success, some tablets commits >failed, how to re-load the failed tablet data. >From the user's point of view, how to deal the case which the steam load >with partial tablet load failed. > >Second, this may be a discussion on the details. Is the transaction lock >granularity still table of tablet? > >Finally, I think this is a very valuable project for Doris Load,Can you >provide a brief project plan? Including what are the parts of the whole, >and what does each part do? This can help newcomers to quickly understand >the project and participate in the development. > > >蔡聪辉 <caiconghui2...@163.com> 于2022年2月21日周一 19:37写道： > >> Now，we want to improve the performance and stability of loading data，one >> way is only load some tablets in the partition instead of all tablets which >> may cause many little files and bring instability，and for stability I would >> gradually change it, the first step is to support tablet with version >> property and then enable be to submit some tablets of one partition to >> commit transaction and finally publish them. >> >> >> The main advantages of this feature is : each load may only involve a >> small number of be nodes, which can greatly enhance the stability of the >> loading data. >> >> >> >>
