Problem setting buffer size for gfortran ( v 11.2)
Hi Investigating some sporadical performance drop when using Fortran IO on our Lustre shared file system, we realized that the buffer size used as default by the gnu fortran compiler is 8 kiB only. We wanted to change this default and according to the documentation, one has just to set the environment variables: GFORTRAN_UNFORMATTED_BUFFER_SIZE GFORTRAN_FORMATTED_BUFFER_SIZE= GFORTRAN_UNBUFFERED_ALL=n to increase the buffer size during runtime. Unfortunately isetting these variables does not change the default buffer size. I tried with gfortran 8, 10, and the new 11.2. Looking a the code, i found in io/unix.c that the default is set to the following value, static const int FORMATTED_BUFFER_SIZE_DEFAULT = 8192; static const int UNFORMATTED_BUFFER_SIZE_DEFAULT = 128*1024; Of course, changing the default value and recompiling works and change accordingly the buffer size It should nevertheless be possible to change these values without recompiling. Is there something one should be aware of when setting these variables? Best regards, Denis - Denis Bertini Abteilung: CIT Ort: SB3 2.265a Tel: +49 6159 71 2240 Fax: +49 6159 71 2986 E-Mail: d.bert...@gsi.de GSI Helmholtzzentrum für Schwerionenforschung GmbH Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 Managing Directors / Geschäftsführung: Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats: Ministerialdirigent Dr. Volkmar Dietz
Re: Problem setting buffer size for gfortran ( v 11.2)
Hi Denis, Am 23.02.22 um 17:10 schrieb Bertini, Denis Dr. via Fortran: Hi Investigating some sporadical performance drop when using Fortran IO on our Lustre shared file system, we realized that the buffer size used as default by the gnu fortran compiler is 8 kiB only. We wanted to change this default and according to the documentation, one has just to set the environment variables: GFORTRAN_UNFORMATTED_BUFFER_SIZE GFORTRAN_FORMATTED_BUFFER_SIZE= GFORTRAN_UNBUFFERED_ALL=n to increase the buffer size during runtime. this works for me on gcc-12 development for the following test program on Linux/x86: open(10,file="bigfile") write(10,*) "data" close(10) end Compile, link, and run under strace with GFORTRAN_FORMATTED_BUFFER_SIZE=4194304 strace ./a.out [...] openat(AT_FDCWD, "bigfile", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0 mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x146b69b1c000 write(3, " data\n", 6) = 6 ftruncate(3, 6) = 0 munmap(0x146b69b1c000, 4198400) = 0 close(3)= 0 Unfortunately isetting these variables does not change the default buffer size. I tried with gfortran 8, 10, and the new 11.2. Looking a the code, i found in io/unix.c that the default is set to the following value, static const int FORMATTED_BUFFER_SIZE_DEFAULT = 8192; static const int UNFORMATTED_BUFFER_SIZE_DEFAULT = 128*1024; Of course, changing the default value and recompiling works and change accordingly the buffer size It should nevertheless be possible to change these values without recompiling. Is there something one should be aware of when setting these variables? You said you're using Lustre. I guess you use the default blocksize (4MB). Does your Lustre client support mmap? What does strace report? I remember I once saw a system where we had to disable the use of mmap in a library. Cheers, Harald Best regards, Denis - Denis Bertini Abteilung: CIT Ort: SB3 2.265a Tel: +49 6159 71 2240 Fax: +49 6159 71 2986 E-Mail: d.bert...@gsi.de GSI Helmholtzzentrum für Schwerionenforschung GmbH Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 Managing Directors / Geschäftsführung: Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats: Ministerialdirigent Dr. Volkmar Dietz
Re: Problem setting buffer size for gfortran ( v 11.2)
Hi Harald, In attachment you will find a typical strace output when running with gfortran8 on lustre filsystem. As i said for gfortran version 8.x, 10.x, 11.2 setting the runtime env. variable to change buffer size did not work for me as you could see in the strace output. You could also see that "mmap" is used by the lustre client. The optimal blocksize given back by the STAT() routine indeed favor a 4 MiB buffer size. Best Denis From: Harald Anlauf Sent: Wednesday, February 23, 2022 7:26 PM To: Bertini, Denis Dr.; fortran@gcc.gnu.org Subject: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Denis, Am 23.02.22 um 17:10 schrieb Bertini, Denis Dr. via Fortran: > Hi > Investigating some sporadical performance drop when using Fortran IO on our > Lustre > shared file system, we realized that the buffer size used as default by the > gnu fortran > compiler is 8 kiB only. > We wanted to change this default and according to the documentation, one has > just to set > the environment variables: > > GFORTRAN_UNFORMATTED_BUFFER_SIZE > GFORTRAN_FORMATTED_BUFFER_SIZE= > GFORTRAN_UNBUFFERED_ALL=n > > to increase the buffer size during runtime. this works for me on gcc-12 development for the following test program on Linux/x86: open(10,file="bigfile") write(10,*) "data" close(10) end Compile, link, and run under strace with GFORTRAN_FORMATTED_BUFFER_SIZE=4194304 strace ./a.out [...] openat(AT_FDCWD, "bigfile", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0 mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x146b69b1c000 write(3, " data\n", 6) = 6 ftruncate(3, 6) = 0 munmap(0x146b69b1c000, 4198400) = 0 close(3)= 0 > Unfortunately isetting these variables does not change the default buffer > size. > I tried with gfortran 8, 10, and the new 11.2. > Looking a the code, i found in > io/unix.c > that the default is set to the following value, > > static const int FORMATTED_BUFFER_SIZE_DEFAULT = 8192; > static const int UNFORMATTED_BUFFER_SIZE_DEFAULT = 128*1024; > > Of course, changing the default value and recompiling works and change > accordingly the buffer size > It should nevertheless be possible to change these values without recompiling. > Is there something one should be aware of when setting these variables? You said you're using Lustre. I guess you use the default blocksize (4MB). Does your Lustre client support mmap? What does strace report? I remember I once saw a system where we had to disable the use of mmap in a library. Cheers, Harald > Best regards, > Denis > > - > Denis Bertini > Abteilung: CIT > Ort: SB3 2.265a > > Tel: +49 6159 71 2240 > Fax: +49 6159 71 2986 > E-Mail: d.bert...@gsi.de > > GSI Helmholtzzentrum für Schwerionenforschung GmbH > Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de > > Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 > Managing Directors / Geschäftsführung: > Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock > Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats: > Ministerialdirigent Dr. Volkmar Dietz >
Re: Problem setting buffer size for gfortran ( v 11.2)
Hi Denis, Am 23.02.22 um 21:28 schrieb Bertini, Denis Dr. via Fortran: Hi Harald, In attachment you will find a typical strace output when running with gfortran8 on lustre filsystem. unfortunately I do not see any attachment to your mail. Note: GFORTRAN_FORMATTED_BUFFER_SIZE & friends was added in 10.1, and possibly backported to some 9.y, but *not* to 8.x. As i said for gfortran version 8.x, 10.x, 11.2 setting the runtime env. variable to change buffer size did not work for me as you could see in the strace output. You could also see that "mmap" is used by the lustre client. The optimal blocksize given back by the STAT() routine indeed favor a 4 MiB buffer size. It might make sense to show your issue to the vendor of your system. Parallel file systems - not only Lustre - require special handling not only to get optimum performance, but also correct behavior. (E.g. the HDF5 library is known to hit bugs in our Lustre.) Cheers, Harald Best Denis From: Harald Anlauf Sent: Wednesday, February 23, 2022 7:26 PM To: Bertini, Denis Dr.; fortran@gcc.gnu.org Subject: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Denis, Am 23.02.22 um 17:10 schrieb Bertini, Denis Dr. via Fortran: Hi Investigating some sporadical performance drop when using Fortran IO on our Lustre shared file system, we realized that the buffer size used as default by the gnu fortran compiler is 8 kiB only. We wanted to change this default and according to the documentation, one has just to set the environment variables: GFORTRAN_UNFORMATTED_BUFFER_SIZE GFORTRAN_FORMATTED_BUFFER_SIZE= GFORTRAN_UNBUFFERED_ALL=n to increase the buffer size during runtime. this works for me on gcc-12 development for the following test program on Linux/x86: open(10,file="bigfile") write(10,*) "data" close(10) end Compile, link, and run under strace with GFORTRAN_FORMATTED_BUFFER_SIZE=4194304 strace ./a.out [...] openat(AT_FDCWD, "bigfile", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0 mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x146b69b1c000 write(3, " data\n", 6) = 6 ftruncate(3, 6) = 0 munmap(0x146b69b1c000, 4198400) = 0 close(3)= 0 Unfortunately isetting these variables does not change the default buffer size. I tried with gfortran 8, 10, and the new 11.2. Looking a the code, i found in io/unix.c that the default is set to the following value, static const int FORMATTED_BUFFER_SIZE_DEFAULT = 8192; static const int UNFORMATTED_BUFFER_SIZE_DEFAULT = 128*1024; Of course, changing the default value and recompiling works and change accordingly the buffer size It should nevertheless be possible to change these values without recompiling. Is there something one should be aware of when setting these variables? You said you're using Lustre. I guess you use the default blocksize (4MB). Does your Lustre client support mmap? What does strace report? I remember I once saw a system where we had to disable the use of mmap in a library. Cheers, Harald Best regards, Denis - Denis Bertini Abteilung: CIT Ort: SB3 2.265a Tel: +49 6159 71 2240 Fax: +49 6159 71 2986 E-Mail: d.bert...@gsi.de GSI Helmholtzzentrum für Schwerionenforschung GmbH Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 Managing Directors / Geschäftsführung: Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats: Ministerialdirigent Dr. Volkmar Dietz
Re: Problem setting buffer size for gfortran ( v 11.2)
Hi Harald, I ran also the same with gfortran v11.2. together with singularity and did not work as well. Then i change the defintion of the buffer size in unix.c to default at 1 MiB and it works. What do you mean by vendor? Do you think the problem could be linked to running fortran within singularity container? Best Denis From: Harald Anlauf Sent: Wednesday, February 23, 2022 10:18:17 PM To: Bertini, Denis Dr. Cc: fortran@gcc.gnu.org Subject: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Denis, if I'm reading the strace output correctly, you're running the app with gfortran-8 (too old) under singularity (virtualization software). I'm afraid you need to contact your vendor or system folks. Cheers, Harald Gesendet: Mittwoch, 23. Februar 2022 um 21:59 Uhr Von: "Bertini, Denis Dr." An: "Harald Anlauf" , "fortran@gcc.gnu.org" Betreff: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Harald, I do not know why the attachment was not sent with my email I try again Best Denis
Re: Problem setting buffer size for gfortran ( v 11.2)
Hi Denis, if editing of unix.c works: is it possible that you either misspelled the name of the environment variable, forgot to export it, or if you are running under MPI, you need to make sure that the variable is MPI-exported to all ranks? Cheers, Harald Gesendet: Mittwoch, 23. Februar 2022 um 22:23 Uhr Von: "Bertini, Denis Dr." An: "Harald Anlauf" Cc: "fortran@gcc.gnu.org" Betreff: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Harald, I ran also the same with gfortran v11.2. together with singularity and did not work as well. Then i change the defintion of the buffer size in unix.c to default at 1 MiB and it works. What do you mean by vendor? Do you think the problem could be linked to running fortran within singularity container? Best Denis From: Harald Anlauf Sent: Wednesday, February 23, 2022 10:18:17 PM To: Bertini, Denis Dr. Cc: fortran@gcc.gnu.org Subject: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Denis, if I'm reading the strace output correctly, you're running the app with gfortran-8 (too old) under singularity (virtualization software). I'm afraid you need to contact your vendor or system folks. Cheers, Harald Gesendet: Mittwoch, 23. Februar 2022 um 21:59 Uhr Von: "Bertini, Denis Dr." An: "Harald Anlauf" , "fortran@gcc.gnu.org" Betreff: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Harald, I do not know why the attachment was not sent with my email I try again Best Denis
[PATCH] PR fortran/84519 - [F2018] STOP and ERROR STOP statements with QUIET specifier
Dear Fortranners, Fortran 2018 added a QUIET= specifier to STOP and ERROR STOP statements. Janne already implemented the library side code four (4!) years ago, but so far the frontend implementation was missing. Furthermore, F2018 allows for non-default-integer stopcode expressions (finally!). The attached patch provides this implementation. That was not too much fun for the following reasons: - fixed format vs. free format - F95 and F2003 apparently did not require a blank between STOP and stopcode, while F2008+ do require it. This should explain for the three testcases. Regtested on x86_64-pc-linux-gnu. OK for mainline? One step closer to F2018! Thanks, Harald From 66e80a9847b3e16d4c619ba8da9f3dba891cff34 Mon Sep 17 00:00:00 2001 From: Harald Anlauf Date: Wed, 23 Feb 2022 23:08:29 +0100 Subject: [PATCH] Fortran: frontend code for F2018 QUIET specifier to STOP and ERROR STOP Fortran 2018 allows for a QUIET specifier to the STOP and ERROR STOP statements. Whilst the gfortran library code provides support for this specifier for quite some time, the frontend implementation was missing. gcc/fortran/ChangeLog: PR fortran/84519 * dump-parse-tree.cc (show_code_node): Dump QUIET specifier when present. * match.cc (gfc_match_stopcode): Implement parsing of F2018 QUIET specifier. F2018 stopcodes may have non-default integer kind. * trans-stmt.cc (gfc_trans_stop): Pass QUIET specifier to call of library function. gcc/testsuite/ChangeLog: PR fortran/84519 * gfortran.dg/stop_1.f90: New test. * gfortran.dg/stop_2.f: New test. * gfortran.dg/stop_3.f90: New test. --- gcc/fortran/dump-parse-tree.cc | 5 +++ gcc/fortran/match.cc | 62 +++- gcc/fortran/trans-stmt.cc| 21 -- gcc/testsuite/gfortran.dg/stop_1.f90 | 44 gcc/testsuite/gfortran.dg/stop_2.f | 31 ++ gcc/testsuite/gfortran.dg/stop_3.f90 | 22 ++ 6 files changed, 172 insertions(+), 13 deletions(-) create mode 100644 gcc/testsuite/gfortran.dg/stop_1.f90 create mode 100644 gcc/testsuite/gfortran.dg/stop_2.f create mode 100644 gcc/testsuite/gfortran.dg/stop_3.f90 diff --git a/gcc/fortran/dump-parse-tree.cc b/gcc/fortran/dump-parse-tree.cc index 2a2f9901b08..322416e6556 100644 --- a/gcc/fortran/dump-parse-tree.cc +++ b/gcc/fortran/dump-parse-tree.cc @@ -2370,6 +2370,11 @@ show_code_node (int level, gfc_code *c) show_expr (c->expr1); else fprintf (dumpfile, "%d", c->ext.stop_code); + if (c->expr2 != NULL) + { + fputs (" QUIET=", dumpfile); + show_expr (c->expr2); + } break; diff --git a/gcc/fortran/match.cc b/gcc/fortran/match.cc index 8edfe4a3a2d..715a74eba51 100644 --- a/gcc/fortran/match.cc +++ b/gcc/fortran/match.cc @@ -2978,6 +2978,13 @@ Fortran 2008 has R856 allstop-stmt is ALL STOP [ stop-code ] R857 stop-code is scalar-default-char-constant-expr or scalar-int-constant-expr +Fortran 2018 has + + R1160 stop-stmt is STOP [ stop-code ] [ , QUIET = scalar-logical-expr] + R1161 error-stop-stmt is + ERROR STOP [ stop-code ] [ , QUIET = scalar-logical-expr] + R1162 stop-code is scalar-default-char-expr + or scalar-int-expr For free-form source code, all standards contain a statement of the form: @@ -2994,8 +3001,10 @@ static match gfc_match_stopcode (gfc_statement st) { gfc_expr *e = NULL; + gfc_expr *quiet = NULL; match m; bool f95, f03, f08; + char c; /* Set f95 for -std=f95. */ f95 = (gfc_option.allow_std == GFC_STD_OPT_F95); @@ -3006,11 +3015,16 @@ gfc_match_stopcode (gfc_statement st) /* Set f08 for -std=f2008. */ f08 = (gfc_option.allow_std == GFC_STD_OPT_F08); - /* Look for a blank between STOP and the stop-code for F2008 or later. */ - if (gfc_current_form != FORM_FIXED && !(f95 || f03)) -{ - char c = gfc_peek_ascii_char (); + /* Plain STOP statement? */ + if (gfc_match_eos () == MATCH_YES) +goto checks; + + /* Look for a blank between STOP and the stop-code for F2008 or later. + But allow for F2018's ,QUIET= specifier. */ + c = gfc_peek_ascii_char (); + if (gfc_current_form != FORM_FIXED && !(f95 || f03) && c != ',') +{ /* Look for end-of-statement. There is no stop-code. */ if (c == '\n' || c == '!' || c == ';') goto done; @@ -3023,7 +3037,12 @@ gfc_match_stopcode (gfc_statement st) } } - if (gfc_match_eos () != MATCH_YES) + if (c == ' ') +{ + gfc_gobble_whitespace (); + c = gfc_peek_ascii_char (); +} + if (c != ',') { int stopcode; locus old_locus; @@ -3053,11 +3072,20 @@ gfc_match_stopcode (gfc_statement st) goto cleanup; if (m == MATCH_NO) goto syntax; +} - if (gfc_match_eos () != MATCH_YES) - goto syntax; + if (gfc_match (" , quiet = %e", &quiet) == MATCH_YES) +{ + if (!gfc_notify_std (GFC_STD_F2018, "QUIET= specifier for
*Ping* [PATCH] PR fortran/104573 - ICE in resolve_structure_cons, at fortran/resolve.cc:1299
Am 16.02.22 um 22:20 schrieb Harald Anlauf via Gcc-patches: Dear Fortranners, while we detect invalid uses of type(*), we may run into other issues later when the declared variable is used, leading to an ICE due to a NULL pointer dereference. This is demonstrated by Gerhard's testcase. Steve and I came to rather similar fixes, see PR. Mine is attached. Regtested on x86_64-pc-linux-gnu. OK for mainline? Thanks, Harald
Re: [PATCH] PR fortran/84519 - [F2018] STOP and ERROR STOP statements with QUIET specifier
On 2/23/22 2:21 PM, Harald Anlauf via Fortran wrote: Dear Fortranners, Fortran 2018 added a QUIET= specifier to STOP and ERROR STOP statements. Janne already implemented the library side code four (4!) years ago, but so far the frontend implementation was missing. Furthermore, F2018 allows for non-default-integer stopcode expressions (finally!). The attached patch provides this implementation. That was not too much fun for the following reasons: - fixed format vs. free format - F95 and F2003 apparently did not require a blank between STOP and stopcode, while F2008+ do require it. This should explain for the three testcases. Regtested on x86_64-pc-linux-gnu. OK for mainline? One step closer to F2018! Thanks, Harald A minor comment. Is there a way to also have a run-time test case? OK to commit now and additional test case can be added if necessary later. Regards, Jerry
Re: Problem setting buffer size for gfortran ( v 11.2)
Hi Harald, Good point. I export both environment variables export GFORTRAN_UNFORMATTED_BUFFER_SIZE=1024000 export GFORTRAN_UNBUFFERED_ALL=n directly in the script wich is used within the sbatch main submit script. I added ( in case ) the export also in my .bashrc profile. So i guess all ranks got the exported variables. Additonnaly i ran interactively my test program without slurm on with just one MPI process on one of the node in our cluster. Tracing it then shows as well the fixed buffer size of 8kiB. So i guess there is something not working with setting of the runtime variable, may be not from io/unix.c but from runtime/environ.c where this variables should be set ? I am now testing with gcc11.2 that i modified and re-compile from source. Will send then the output trace Regards, Denis From: Harald Anlauf Sent: Wednesday, February 23, 2022 10:38:20 PM To: Bertini, Denis Dr. Cc: fortran@gcc.gnu.org Subject: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Denis, if editing of unix.c works: is it possible that you either misspelled the name of the environment variable, forgot to export it, or if you are running under MPI, you need to make sure that the variable is MPI-exported to all ranks? Cheers, Harald Gesendet: Mittwoch, 23. Februar 2022 um 22:23 Uhr Von: "Bertini, Denis Dr." An: "Harald Anlauf" Cc: "fortran@gcc.gnu.org" Betreff: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Harald, I ran also the same with gfortran v11.2. together with singularity and did not work as well. Then i change the defintion of the buffer size in unix.c to default at 1 MiB and it works. What do you mean by vendor? Do you think the problem could be linked to running fortran within singularity container? Best Denis From: Harald Anlauf Sent: Wednesday, February 23, 2022 10:18:17 PM To: Bertini, Denis Dr. Cc: fortran@gcc.gnu.org Subject: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Denis, if I'm reading the strace output correctly, you're running the app with gfortran-8 (too old) under singularity (virtualization software). I'm afraid you need to contact your vendor or system folks. Cheers, Harald Gesendet: Mittwoch, 23. Februar 2022 um 21:59 Uhr Von: "Bertini, Denis Dr." An: "Harald Anlauf" , "fortran@gcc.gnu.org" Betreff: Re: Problem setting buffer size for gfortran ( v 11.2) Hi Harald, I do not know why the attachment was not sent with my email I try again Best Denis