I checked the code of be. The config 'trash_file_expire_time_sec' is a unified configuration. It means that the expiration time of all tablets are the same. In other words, each tablet does not have its own expiration time.
I wonder if the code I understand is correct? If I understand correctly, we cannot force the tablet to expire immediately by force drop stmt. Ling Miao ling miao <lingm...@apache.org> 于2020年11月23日周一 下午8:25写道： > It is true that the metadata can be cleared immediately, but the data > cannot be cleared immediately. > Or do we set the expire time of the tablet to a small value while force? > > Ling Miao > > 蔡聪辉 <caiconghui2...@163.com> 于2020年11月23日周一 上午10:46写道： > >> Now, Doris support Drop partition, Drop Table, Drop database force >> operation, >> so if we doesn't want to recover metadata anymore, we can simple execute >> drop force operation. >> >> >> From Conghui Cai >> At 2020-11-23 10:05:46, "ling miao" <lingm...@apache.org> wrote: >> >Or can we have a new command to clear metadata and data directly? >> >Instead of adjusting the parameters before issuing the command. >> >Because if you want to adjust the parameters, it may affect some data >> that >> >you don't want to delete immediately. >> >Instead, our repair function is affected. >> > >> >Ling Miao >> > >> >陈明雨 <morning...@163.com> 于2020年11月22日周日 下午3:33写道： >> > >> >> Yes, there will be a problem that once the user recover the partition >> by >> >> executing `RECOVER` statement, >> >> it will only recover the meta data, but the data is lost. >> >> >> >> >> >> I think we can implement a command to view the meta data in Catalog >> >> Recycle Bin, and can drop them manually >> >> with some properties to tell the BE to remove the data along with. >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> 此致！Best Regards >> >> 陈明雨 Mingyu Chen >> >> >> >> Email: >> >> chenmin...@apache.org >> >> >> >> >> >> >> >> >> >> >> >> 在 2020-11-22 10:05:13，"ling miao" <lingm...@apache.org> 写道： >> >> >Now we often just drop the partition and go directly to clean up the >> files >> >> >under trash >> >> >However, Doris continues to keep the catalog as you said, will there >> be no >> >> >problem? >> >> > >> >> >Ling Miao >> >> > >> >> >陈明雨 <morning...@163.com> 于2020年11月21日周六 下午11:12写道： >> >> > >> >> >> First, after table is dropped, it will stay in Catalog Recycle bin >> for a >> >> >> while, configured be FE config "catalog_trash_expire_second", >> default is >> >> >> 86400. >> >> >> After that, the table will be dropped from Catalog, and data will be >> >> moved >> >> >> to BE's trash. And then, the BE's trash will be cleaned later, >> >> >> configued by BE config "trash_file_expire_time_sec", default is >> 259200. >> >> >> >> >> >> >> >> >> If we want to clean the data immediately, we can first modify the FE >> >> >> config "catalog_trash_expire_second" to a smaller value. >> >> >> And we can implement a new command to view and clean BE's trash. >> >> >> >> >> >> >> >> >> For example, we can add a new column in result of "SHOW BACKENDS" to >> >> view >> >> >> the trash data size of each BE. >> >> >> And a new command such as: "ADMIN CLEAN BACKEND TRASH 'xxx'" to >> clean >> >> the >> >> >> trash immediately. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> 此致！Best Regards >> >> >> 陈明雨 Mingyu Chen >> >> >> >> >> >> Email: >> >> >> chenmin...@apache.org >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> At 2020-11-20 18:11:31, "ling miao" <lingm...@apache.org> wrote: >> >> >> >Hi folks, >> >> >> > >> >> >> >Currently, when user drop table or partition, Doris will first put >> the >> >> >> data >> >> >> >in the trash directory, and then wait for the trash directory to be >> >> >> emptied. >> >> >> > >> >> >> >But during operation and maintenance, I found that when I need to >> >> clean up >> >> >> >cluster data to release disk resources, I need the following steps: >> >> >> >1. Drop table or partition >> >> >> >2. Log in to the machine >> >> >> >3. Clean up the trash directory >> >> >> > >> >> >> >If there are too many BE nodes in the cluster, the entire process >> of >> >> >> >releasing disk resources will be extremely slow and cumbersome. >> >> >> > >> >> >> >So I propose whether it is possible to add a function for forcibly >> >> >> dropping >> >> >> >data. After this function is called, the data will be deleted >> directly >> >> >> >instead of placing the data in the trash directory. >> >> >> > >> >> >> >Looking forward to your reply (*^ω^*) >> >> >> > >> >> >> > >> >> >> >Ling Miao >> >> >> >> >> >> >
