Rashun, Support for RHEL 8.10 only just arrived in 2.15.5, so performance of clients on that distro version can't have regressed since there is no prior data point to compare it against. I might suggest you use RHEL 8.9 clients, and then perf test 2.15.4 and 2.15.5 Lustre client modules against the exact same backing Lustre cluster. This will provide much more convincing evidence that something regressed between those two versions.
I can't speak to any of the systems involved here as they do not appear to be Azure Managed Lustre. Best, ellis From: lustre-discuss <lustre-discuss-boun...@lists.lustre.org> On Behalf Of Andreas Dilger via lustre-discuss Sent: Thursday, August 1, 2024 11:34 PM To: Baucum, Rashun <rashun.bau...@td.com> Cc: lustre-discuss@lists.lustre.org Subject: [EXTERNAL] Re: [lustre-discuss] Rhel8.10 Lustre Kernel Performance Decrease Sorry, you'll have to ask someone at Azure about this. I don't know anything about what "Premium Lustre" or "Lustre V2" means, and I can't speak to any kind of performance for their systems. On Jul 30, 2024, at 08:48, Baucum, Rashun <rashun.bau...@td.com<mailto:rashun.bau...@td.com>> wrote: Good morning, Andreas. I apologize for the delay. Yes, I will provide additional information. I initially asked the question as a general inquiry. The crux of the performance issue we see is that the newer RHEL 8.10 - 2.15.5 lustre with similar builds do not hit the same average performance as we have seen previously. Which version of Lustre for the old and new kernel and was it the same before upgrading to RHEL8.10? * Previous Kernel : 3.10.0-1160.49.1.el7_lustre, RHEL 7.9 * Current Kernel : 4.18.0-553.5.1.el8_lustre, RHEL 8.10 * It was not the same, we waited for the last minor version of RHEL8. Which RHEL version are you comparing against, RHEL 8.9? * Comparison Version : RHEL 8.8 * Lustre Client Version : 4.18.0-425.3.1.el8_lustre Have you upgraded both the clients and servers to RHEL 8.10, or only the clients? * Currently both are upgraded to the RHEL 8.10. Results of FIO testing Original Production: Initial baseline established ~1 year ago. This build is no longer in use, but performance of non "premium" builds should be approximately around this. Lustre V2: Current builds and are currently being used as a direct reference point. These used RHEL8 clients while being RHEL7 lustres. RHEL8.10 - 2.15.5: Builds being tested before we push updates to higher environments. Between Original Production, Lustre V2 36, and the RHEL8.10 lustres there are minimal changes between builds. In this specific case the only difference between Lustre V2 36 and RHEL8.10 builds are the lustre versions and the RHEL 8.10 Premium Lustre uses Azure's premium SSDs as disks for OSTs. All other builds use standard HDDs for OSTs. Write Throughput Original Production Lustre V2 36 Lustre V2 108 RHEL 8.10 - 2.15.5 - Premium Lustre RHEL 8.10 - 2.15.5 - Standard Lustre Latency (sec) Avg 1.485 1.482 1.338 1.596 2.968 IOPS Avg 689 690 765 640 640 Bandwidth (MB/s) 723 724 803 672 362 IO (GB) 4340 4346 4818 4037 2173 Write IOPS Original Production Lusre V2 36 Lustre V2 108 RHEL 8.10 - 2.15.5 - Premium Lustre RHEL 8.10 - 2.15.5 - Standard Lustre Latency (sec) Avg 10.354 18.709 4.911 5.839 11.821 Avg IOPS 24 13 52 43 21 Bandwidth (MB/s) 26 14 55 45 23 IO (GB) 156 86 328 276 137 Read Throughput Original Production Lustre V2 36 Lustre V2 108 RHEL 8.10 - 2.15.5 - Premium Lustre RHEL 8.10 - 2.15.5 - Standard Lustre Latency (sec) Avg 1.292 1.223 2.307 1.598 3.022 Avg IOPS 792 836 754 640 338 Bandwidth (MB/s) 831 877 791 672 355 IO (GB) 4986 5269 4750 4033 2134 Read IOPS Original Production Lustre V2 36 Lustre V2 108 RHEL 8.10 - 2.15.5 - Premium Lustre RHEL 8.10 - 2.15.5 - Standard Lustre Latency (sec) Avg 4.190 5.887 5.076 5.954 11.924 Avg IOPS 61 43 50 42 21 Bandwidth (MB/s) 64 45 53 45 23 IO (GB) 384 274 318 271 135 Thanks, Rashun Baucum Internal From: Andreas Dilger <adil...@whamcloud.com<mailto:adil...@whamcloud.com>> Sent: Thursday, July 4, 2024 12:40 AM To: Baucum, Rashun <rashun.bau...@td.com<mailto:rashun.bau...@td.com>> Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> Subject: Re: [lustre-discuss] Rhel8.10 Lustre Kernel Performance Decrease CAUTION: EXTERNAL MAIL. DO NOT CLICK ON LINKS OR OPEN ATTACHMENTS YOU DO NOT TRUST ATTENTION : COURRIEL EXTERNE. NE CLIQUEZ PAS SUR DES LIENS ET N'OUVREZ PAS DE PIÈCES JOINTES AUXQUELS VOUS NE FAITES PAS CONFIANCE On Jul 3, 2024, at 13:12, Baucum, Rashun via lustre-discuss <lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>> wrote: Good afternoon, We have recently started executing performance testing on the new rhel 8.10 lustre kernel. We have noticed an drop in performance in our initial testing. Its roughly a 30-40% drop in total IO observed with our FIO testing. My question is has anyone else noticed any performance decreases? Hi Rashun, could you please be more specific about what you are comparing? Which version of Lustre for the old and new kernel, and was it the same before upgrading to RHEL8.10? Which RHEL version are you comparing against, RHEL 8.9? Have you upgraded both the clients and servers to RHEL8.10, or only the clients? Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Whamcloud If you wish to unsubscribe from receiving commercial electronic messages from TD Bank Group, please click here<http://www.td.com/tdoptout> or go to the following web address: www.td.com/tdoptout<http://www.td.com/tdoptout> Si vous souhaitez vous désabonner des messages électroniques de nature commerciale envoyés par Groupe Banque TD veuillez cliquer ici<http://www.td.com/tddesab> ou vous rendre à l'adresse www.td.com/tddesab<http://www.td.com/tddesab> NOTICE: Confidential message which may be privileged. Unauthorized use/disclosure prohibited. If received in error, please go to www.td.com/legal<http://www.td.com/legal> for instructions. AVIS : Message confidentiel dont le contenu peut être privilégié. Utilisation/divulgation interdites sans permission. Si reçu par erreur, prière d'aller au www.td.com/francais/avis_juridique<http://www.td.com/francais/avis_juridique> pour des instructions. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Whamcloud
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
