Now, Doris support Drop partition, Drop Table, Drop database force operation, so if we doesn't want to recover metadata anymore, we can simple execute drop force operation.
From Conghui Cai At 2020-11-23 10:05:46, "ling miao" <lingm...@apache.org> wrote: >Or can we have a new command to clear metadata and data directly? >Instead of adjusting the parameters before issuing the command. >Because if you want to adjust the parameters, it may affect some data that >you don't want to delete immediately. >Instead, our repair function is affected. > >Ling Miao > >陈明雨 <morning...@163.com> 于2020年11月22日周日 下午3:33写道： > >> Yes, there will be a problem that once the user recover the partition by >> executing `RECOVER` statement, >> it will only recover the meta data, but the data is lost. >> >> >> I think we can implement a command to view the meta data in Catalog >> Recycle Bin, and can drop them manually >> with some properties to tell the BE to remove the data along with. >> >> >> >> >> -- >> >> 此致！Best Regards >> 陈明雨 Mingyu Chen >> >> Email: >> chenmin...@apache.org >> >> >> >> >> >> 在 2020-11-22 10:05:13，"ling miao" <lingm...@apache.org> 写道： >> >Now we often just drop the partition and go directly to clean up the files >> >under trash >> >However, Doris continues to keep the catalog as you said, will there be no >> >problem? >> > >> >Ling Miao >> > >> >陈明雨 <morning...@163.com> 于2020年11月21日周六 下午11:12写道： >> > >> >> First, after table is dropped, it will stay in Catalog Recycle bin for a >> >> while, configured be FE config "catalog_trash_expire_second", default is >> >> 86400. >> >> After that, the table will be dropped from Catalog, and data will be >> moved >> >> to BE's trash. And then, the BE's trash will be cleaned later, >> >> configued by BE config "trash_file_expire_time_sec", default is 259200. >> >> >> >> >> >> If we want to clean the data immediately, we can first modify the FE >> >> config "catalog_trash_expire_second" to a smaller value. >> >> And we can implement a new command to view and clean BE's trash. >> >> >> >> >> >> For example, we can add a new column in result of "SHOW BACKENDS" to >> view >> >> the trash data size of each BE. >> >> And a new command such as: "ADMIN CLEAN BACKEND TRASH 'xxx'" to clean >> the >> >> trash immediately. >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> 此致！Best Regards >> >> 陈明雨 Mingyu Chen >> >> >> >> Email: >> >> chenmin...@apache.org >> >> >> >> >> >> >> >> >> >> >> >> At 2020-11-20 18:11:31, "ling miao" <lingm...@apache.org> wrote: >> >> >Hi folks, >> >> > >> >> >Currently, when user drop table or partition, Doris will first put the >> >> data >> >> >in the trash directory, and then wait for the trash directory to be >> >> emptied. >> >> > >> >> >But during operation and maintenance, I found that when I need to >> clean up >> >> >cluster data to release disk resources, I need the following steps: >> >> >1. Drop table or partition >> >> >2. Log in to the machine >> >> >3. Clean up the trash directory >> >> > >> >> >If there are too many BE nodes in the cluster, the entire process of >> >> >releasing disk resources will be extremely slow and cumbersome. >> >> > >> >> >So I propose whether it is possible to add a function for forcibly >> >> dropping >> >> >data. After this function is called, the data will be deleted directly >> >> >instead of placing the data in the trash directory. >> >> > >> >> >Looking forward to your reply (*^ω^*) >> >> > >> >> > >> >> >Ling Miao >> >> >>
