zhengshengjun commented on issue #16424: URL: https://github.com/apache/doris/issues/16424#issuecomment-1418406096
FE Log: ``` 2023-02-04 09:12:46,169 INFO (tablet scheduler|62) [TabletScheduler.schedulePendingTablets():457] add clone task to agent task queue: tablet id: 338714, replica id: 542836, schema hash: 156405523, storageMedium: HDD, visible version(hash): 7295011, src backend: xxxx:xxxx:x:xxx::57, src path hash: 4248943622730157877, src backend: xxxx:xxxx:x:xxx::57, src path hash: 4248943622730157877, dest backend: 10003, dest path hash: 9102637428361925408 2023-02-04 10:50:00,686 INFO (tablet scheduler|62) [TabletScheduler.removeTabletCtx():1425] remove the tablet 338714. because: timeout 2023-02-04 10:50:06,708 INFO (tablet scheduler|62) [TabletScheduler.schedulePendingTablets():457] add clone task to agent task queue: tablet id: 338714, replica id: 542836, schema hash: 156405523, storageMedium: HDD, visible version(hash): 7297643, src backend: xxxx:xxxx:x:xxx::40, src path hash: -2641615873456264123, src backend: xxxx:xxxx:x:xxx::40, src path hash: -2641615873456264123, dest backend: 10003, dest path hash: 9102637428361925408 2023-02-04 12:27:26,814 INFO (tablet scheduler|62) [TabletScheduler.removeTabletCtx():1425] remove the tablet 338714. because: timeout ``` BE Log: I0204 09:12:46.169904 2756190 engine_clone_task.cpp:105] clone to existed tablet. missed_versions_size=202, allow_incremental_clone=0, signature=338714, tablet_id=338714, committed_version=7295011 I0204 09:12:46.174667 2756190 engine_clone_task.cpp:228] successfully make snapshot in remote BE|host=xxxx:xxxx:x:443::57|port=9060|tablet=338714|snapshot_path=/data08/doris_data/doris_data_hl/be.HDD/snapshot/20230204091246.917.5834/|signature=338714|missed_versions=[7256165-7256165][7256166-7256166][7256167-7256167][7256168-7256168][7256169-7256169][7256170-7256170][7256171-7256171][7256172-7256172][7256173-7256173][7256174-7256174][7256175-7256175][7256176-7256176][7256177-7256177][7256178-7256178][7256179-7256179][7256180-7256180][7256181-7256181][7256182-7256182][7256183-7256183][7256184-7256184][7256185-7256185][7256186-7256186][7256187-7256187][7256188-7256188][7256189-7256189][7256190-7256190][7256191-7256191][7256192-7256192][7256193-7256193][7256194-7256194][7256195-7256195][7256196-7256196][7256197-7256197][7256198-7256198][7256199-7256199][7256200-7256200][7256201-7256201][7256202-7256202][7256203-7256203][7256204-7256204][7256205-7256205][7256206-7256206][7256207-72562 07][7256208-7256208][7256209-7256209][7256210-7256210][7256211-7256211][7256212-7256212][7256213-7256213][7256214-7256214][7256215-7256215][7256216-7256216][7256217-7256217][7256218-7256218][7256219-7256219][7256220-7256220][7256221-7256221][7256222-7256222][7256223-7256223][7256224-7256224][7256225-7256225][7256226-7256226][7256227-7256227][7256228-7256228][7256229-7256229][7256230-7256230][7256231-7256231][7256232-7256232][7256233-7256233][7256234-7256234][7256235-7256235][7256236-7256236][7256237-7256237][7256238-7256238][7256239-7256239][7256240-7256240][7256241-7256241][7256242-7256242][7256243-7256243][7256244-7256244][7256245-7256245][7256246-7256246][7256247-7256247][7256248-7256248][7256249-7256249][7256250-7256250][7256251-7256251][7256252-7256252][7256253-7256253][7256254-7256254][7256255-7256255][7256256-7256256][7256257-7256257][7256258-7256258][7256259-7256259][7256260-7256260][7256261-7256261][7256262-7256262][7256263-7256263][7256264-7256264][7256265-7256265][7256266 -7256266][7256267-7256267][7256268-7256268][7256269-7256269][7256270-7256270][7256271-7256271][7256272-7256272][7256273-7256273][7256274-7256274][7256275-7256275][7256276-7256276][7256277-7256277][7256278-7256278][7256279-7256279][7256280-7256280][7256281-7256281][7256282-7256282][7256283-7256283][7256284-7256284][7256285-7256285][7256286-7256286][7256287-7256287][7256288-7256288][7256289-7256289][7256290-7256290][7256291-7256291][7256292-7256292][7256293-7256293][7256294-7256294][7256295-7256295][7256296-7256296][7256297-7256297][7256298-7256298][7256299-7256299][7256300-7256300][7256301-7256301][7256302-7256302][7256303-7256303][7256304-7256304][7256305-7256305][7256306-7256306][7256307-7256307][7256308-7256308][7256309-7256309][7256310-7256310][7256311-7256311][7256312-7256312][7256313-7256313][7256314-7256314][7256315-7256315][7256316-7256316][7256317-7256317][7256318-7256318][7256319-7256319][7256320-7256320][7256321-7256321][7256322-7256322][7256323-7256323][7256324-7256324][7 256325-7256325][7256326-7256326][7256327-7256327][7256328-7256328][7256329-7256329][7256330-7256330][7256331-7256331][7256332-7256332][7256333-7256333][7256334-7256334][7256335-7256335][7256336-7256336][7256337-7256337][7256338-7256338][7256339-7256339][7256340-7256340][7256341-7256341][7256342-7256342][7256343-7256343][7256344-7256344][7256345-7256345][7256346-7256346][7256347-7256347][7256348-7256348][7256349-7256349][7256350-7256350][7256351-7256351][7256352-7256352][7256353-7256353][7256354-7256354][7256355-7256355][7256356-7256356][7256357-7256357][7256358-7256358][7256359-7256359][7256360-7256360][7256361-7256361][7256362-7256362][7256363-7256363][7256364-7256364][7256365-7256365][7256366-7256366] I0204 09:12:46.175863 2756190 engine_clone_task.cpp:408] clone begin to download file from: http://[xxxx:xxxx:x:xxx::57]:8040/api/_tablet/_download?token=ae7e81ff-d3f4-490f-8608-9a3096f00d3c&file=/data08/doris_data/aurora_data_hl/be.HDD/snapshot/20230204091246.917.5834//338714/156405523/020000000061024c0349f83cebdd79b9dcaa3797c7a8cc90_22.dat to: /data25/doris_data/aurora_data_hl/be.SSD/data/134/338714/156405523/clone/020000000061024c0349f83cebdd79b9dcaa3797c7a8cc90_22.dat. size(B): 236173615, timeout(s): 4612 ... ... I0204 09:14:49.279039 2756190 engine_clone_task.cpp:446] succeed to copy tablet 338714, total file size: 6055467667 B, cost: 123103 ms, rate: 49.1903 MB/s I0204 09:14:49.381126 2756190 engine_clone_task.cpp:573] begin to finish full clone. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, cloned_max_version=[7295003-7295012] I0204 09:14:49.381153 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[0-7247166] I0204 09:14:49.381157 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[0-7247166] I0204 09:14:49.381160 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7247167-7256164] I0204 09:14:49.381161 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[7247167-7256164] I0204 09:14:49.381163 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7256367-7265993] I0204 09:14:49.381165 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[7256367-7265993] I0204 09:14:49.381166 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7265994-7277617] I0204 09:14:49.381167 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[7265994-7277617] I0204 09:14:49.381170 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7277618-7290348] I0204 09:14:49.381171 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[7277618-7290348] I0204 09:14:49.381173 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7290349-7294744] I0204 09:14:49.381175 2756190 engine_clone_task.cpp:625] version exist in local tablet, no need to clone. delete it from clone tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, version='[7290349-7294744] I0204 09:14:49.381197 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7294745-7295001] I0204 09:14:49.381198 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[7294745-7295001] I0204 09:14:49.381201 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7295002-7295066] W0204 09:14:52.411546 2756190 task_worker_pool.cpp:925] failed to clone tablet|signature=338714|tablet_id=338714|error=Internal error: version cross src latest. cloned_max_version=[7295003-7295012], local_version=[7295002-7295066] and the repeated err message in be is: W0204 09:14:52.411546 2756190 task_worker_pool.cpp:925] failed to clone tablet|signature=338714|tablet_id=338714|error=Internal error: version cross src latest. cloned_max_version=[7295003-7295012], local_version=[7295002-7295066] W0204 10:52:15.685870 2756189 task_worker_pool.cpp:925] failed to clone tablet|signature=338714|tablet_id=338714|error=Internal error: version cross src latest. cloned_max_version=[7297643-7297643], local_version=[7297537-7297677] W0204 12:29:55.985198 2756189 task_worker_pool.cpp:925] failed to clone tablet|signature=338714|tablet_id=338714|error=Internal error: version cross src latest. cloned_max_version=[7300464-7300464], local_version=[7300336-7300467] ... W0204 22:17:34.976135 2756189 task_worker_pool.cpp:925] failed to clone tablet|signature=338714|tablet_id=338714|error=Internal error: version cross src latest. cloned_max_version=[7316482-7316484], local_version=[7316296-7316502] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
