Problems and solutions are relatively clear. I would like to do some discussion on the problems this solution may face. First, It seems that you want to commit transactions based on tablet granularity, I think multiple tablets in one partition should keep consistency in one steam load. In one stream load, if some tablets commits success, some tablets commits failed, how to re-load the failed tablet data. >From the user's point of view, how to deal the case which the steam load with partial tablet load failed.
Second, this may be a discussion on the details. Is the transaction lock granularity still table of tablet? Finally, I think this is a very valuable project for Doris Load,Can you provide a brief project plan? Including what are the parts of the whole, and what does each part do? This can help newcomers to quickly understand the project and participate in the development. 蔡聪辉 <caiconghui2...@163.com> 于2022年2月21日周一 19:37写道： > Now，we want to improve the performance and stability of loading data，one > way is only load some tablets in the partition instead of all tablets which > may cause many little files and bring instability，and for stability I would > gradually change it, the first step is to support tablet with version > property and then enable be to submit some tablets of one partition to > commit transaction and finally publish them. > > > The main advantages of this feature is : each load may only involve a > small number of be nodes, which can greatly enhance the stability of the > loading data. > > > >
