Hi Jose, On Sat, Sep 4, 2021 at 1:58 PM José Manuel Blanco <[email protected]> wrote:
> Hi all, > > We have a problem with tasks (written in PHP) doing a lot of file > renaming/moving (even several renames per second). > > The pattern is always the same: the task rename files with variable > filename to the SAME final file (that is: the destination filename is > ALWAYS the same). > > Problem: according to glusterfs and task logs, some rename are successful > but other don't and we don't understand why or what is causing the error in > the failing renames because the error is "File exists", but the rename > supossedly must address these situations correctly and "overwrite" the > destination file if it exists (it uses the PHP rename() function) > > More info: > > - We use a distributed-replicated volume in a 3 node cluster > > gluster volume info moodle-cv > > Volume Name: moodle-cv > Type: Distributed-Replicate > Volume ID: 1eef9714-3943-4d77-b42f-6b1144389c56 > Status: Started > Snapshot Count: 0 > Number of Bricks: 5 x (2 + 1) = 15 > Transport-type: tcp > Bricks: > Brick1: moodle2017-n2:/glusterfs/moodle-cv/brick1/datos > Brick2: moodle2017-n3:/glusterfs/moodle-cv/brick1/replica > Brick3: moodle2017-n1:/glusterfs/moodle-cv/brick1/arbiter (arbiter) > Brick4: moodle2017-n3:/glusterfs/moodle-cv/brick2/datos > Brick5: moodle2017-n2:/glusterfs/moodle-cv/brick2/replica > Brick6: moodle2017-n1:/glusterfs/moodle-cv/brick2/arbiter (arbiter) > Brick7: moodle2017-n2:/glusterfs/moodle-cv/brick3/datos > Brick8: moodle2017-n3:/glusterfs/moodle-cv/brick3/replica > Brick9: moodle2017-n1:/glusterfs/moodle-cv/brick3/arbiter (arbiter) > Brick10: moodle2017-n3:/glusterfs/moodle-cv/brick4/datos > Brick11: moodle2017-n2:/glusterfs/moodle-cv/brick4/replica > Brick12: moodle2017-n1:/glusterfs/moodle-cv/brick4/arbiter (arbiter) > Brick13: moodle2017-n2:/glusterfs/moodle-cv/brick5/datos > Brick14: moodle2017-n3:/glusterfs/moodle-cv/brick5/replica > Brick15: moodle2017-n1:/glusterfs/moodle-cv/brick5/arbiter (arbiter) > Options Reconfigured: > nfs.disable: on > storage.fips-mode-rchecksum: on > performance.open-behind: off > performance.lazy-open: no > cluster.self-heal-daemon: enable > > - The nodes use Oracle Linux 7.9 (RedHat clone) and GlusterFS 8.5 > > - The tasks use the PHP rename() function > > - The tasks renaming files are executed IN ONE NODE OF THE CLUSTER that > mounts the volume using the FUSE client > > - At the end of the message I paste a (very) little fragment of the volume > log > > > Any ideas of the possible cause of the problem and/or suggestions to avoid > it? > Most probably the issue is caused by special internal files needed by Gluster that are used to reference the correct location of a file in a distributed volume (they are referred as linkto files). I would say the error happens because those files already exist when it's expected that they don't exist (I think it's a similar problem as https://github.com/gluster/glusterfs/issues/1723). Are you using FUSE mounts ? are you doing the renames from more than one client ? Regards, Xavi Regards. > > > This is the fragment of the log: > > * I've "separated" the lines by "rename operation" but all the lines > appear together in the log > > --->SUCCESSFUL RENAME: > > [2021-09-03 09:04:02.478321] I [MSGID: 109066] > [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e50272f326.07618090.temp > (4a875eac-5389-400c-9359-458e151054f7) > (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) => > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache > (11fdf019-16f8-48ae-bd1f-05b670b29ec1) > (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-2) > > --->UNSUCCESSFUL RENAME: > > [2021-09-03 09:04:02.497845] I [MSGID: 109066] > [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp > (350698df-35d9-4489-90cc-faa004bfc861) > (hash=moodle-cv-replicate-0/cache=moodle-cv-replicate-0) => > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache > (4a875eac-5389-400c-9359-458e151054f7) > (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) > > [2021-09-03 09:04:02.503271] W [MSGID: 114031] > [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-0: remote > operation failed. > [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp}, > {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, > {errno=17}, {error=File exists}] > > [2021-09-03 09:04:02.503339] W [MSGID: 114031] > [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-1: remote > operation failed. > [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp}, > {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, > {errno=17}, {error=File exists}] > > [2021-09-03 09:04:02.503393] W [MSGID: 114031] > [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-2: remote > operation failed. > [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp}, > {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, > {errno=17}, {error=File exists}] > > [2021-09-03 09:04:02.504997] W [fuse-bridge.c:2500:fuse_rename_cbk] > 0-glusterfs-fuse: 781438412: > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp > -> > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache > => -1 (File exists) > > --->UNSUCCESSFUL RENAME: > > [2021-09-03 09:04:02.729314] I [MSGID: 109066] > [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp > (5bff6629-80b8-4d9c-8e37-abfa7691124e) > (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) => > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache > (4a875eac-5389-400c-9359-458e151054f7) > (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) > > [2021-09-03 09:04:02.735631] W [MSGID: 114031] > [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-5: remote > operation failed. > [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp}, > {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, > {errno=17}, {error=File exists}] > > [2021-09-03 09:04:02.735709] W [MSGID: 114031] > [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-4: remote > operation failed. > [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp}, > {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, > {errno=17}, {error=File exists}] > > [2021-09-03 09:04:02.735747] W [MSGID: 114031] > [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-3: remote > operation failed. > [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp}, > {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, > {errno=17}, {error=File exists}] > > [2021-09-03 09:04:02.737356] W [fuse-bridge.c:2500:fuse_rename_cbk] > 0-glusterfs-fuse: 781438861: > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp > -> > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache > => -1 (File exists) > > --->SUCCESSFUL RENAME: > > [2021-09-03 09:04:02.799776] I [MSGID: 109066] > [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502c175a4.68869416.temp > (82725bab-b888-4d24-b239-31b86f50425f) > (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) => > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache > (4a875eac-5389-400c-9359-458e151054f7) > (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) > > --->UNSUCCESFULL RENAME: > > [2021-09-03 09:04:02.963545] I [MSGID: 109066] > [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp > (b3c5435c-f79b-402f-ba00-0e78dcdb6cd2) > (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) => > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache > (82725bab-b888-4d24-b239-31b86f50425f) > (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) > > [2021-09-03 09:04:02.968665] W [MSGID: 114031] > [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-5: remote > operation failed. > [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp}, > {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, > {errno=17}, {error=File exists}] > > [2021-09-03 09:04:02.968801] W [MSGID: 114031] > [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-4: remote > operation failed. > [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp}, > {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, > {errno=17}, {error=File exists}] > > [2021-09-03 09:04:02.968847] W [MSGID: 114031] > [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-3: remote > operation failed. > [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp}, > {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, > {errno=17}, {error=File exists}] > > [2021-09-03 09:04:02.970438] W [fuse-bridge.c:2500:fuse_rename_cbk] > 0-glusterfs-fuse: 781439391: > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp > -> > /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache > => -1 (File exists) > > > -- > > <http://www.uva.es> > <http://www.uva.es> [image: Escudo UVa] <http://www.uva.es> > > Jose Manuel Blanco > > Analista Sistemas · [email protected] > Tfno.: (+34) 983 18 6411 > > Universidad de Valladolid > Servicio de las TIC > > Edificio Alfonso VIII · C/ Real de Burgos s/n > 47011 · Valladolid · España (*Spain*) > > | Facebook > <https://www.facebook.com/Universidad-de-Valladolid-187763507920209/> | > Twitter <https://twitter.com/uva_es> | UVa <http://www.uva.es> > > Este mensaje puede contener información confidencial, sometida al secreto > profesional, cuya divulgación no está permitida por la ley. Si usted no es > su destinatario, por favor, notifíquelo al remitente y borre este correo de > su sistema. A los efectos de la protección de datos y el RGPD, consulte: > protección > de datos en la UVa <http://www.uva.es/protecciondedatos/>. El emisor no > garantiza la integridad, rapidez o seguridad del presente correo, ni se > responsabiliza de posibles perjuicios derivados de la captura, > incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por > terceros. Piensa en verde: lee en la pantalla. > > This message may contain confidential information covered by the > obligation of professional secrecy, the disclosure of which would be > contrary to the law. If you are not the intended recipient, please advise > the sender and delete this e-mail from your system. For the purposes of > data protection and GDPR, read: protección de datos en la UVa > <http://www.uva.es/protecciondedatos/>. The sender does not guarantee the > integrity, the accuracy, the swift delivery or the security of this email > transmission, and assumes no responsibility for any possible damage > incurred through data capture, virus incorporation or any manipulation > carried out by third parties. Think green: read on the screen. > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > [email protected] > https://lists.gluster.org/mailman/listinfo/gluster-users >
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
