Very strange issue indeed Eric, I have never heard of a specific client causing slowdowns, I would understand if it where a massive amount of oracle sessions starting but if it's really platform and not session count related it would have to be something that IBM needs to look into I guess. May I ask, is there a specific reason you are running fairly old server code in the environment?
On Tue, Aug 27, 2019 at 1:01 PM Loon, Eric van (ITOP NS) - KLM < eric-van.l...@klm.com> wrote: > Hi Stefan, > > There is no noticable impact on the TSM performance when we use the > tsmdiskperf tool on the database volumes. > We recently discovered the issue seems to be related to the TDP for Oracle > clients. When none are running the server response is OK, even during a > reasonable load. But as soon as the TDP for Oracle clients start kicking > in, the server becomes slow. We brought it to the attention of the > developers, but no response yet... > Thanks for your help! > > Kind regards, > Eric van Loon > Air France/KLM Storage & Backup > > -----Original Message----- > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of > Stefan Folkerts > Sent: maandag 26 augustus 2019 10:39 > To: ADSM-L@VM.MARIST.EDU > Subject: Re: TSM server performance continuing > > Eric, > > What happens when you benchmark the DB volumes using the tool provided > with the blueprints on an idle system, does the system also slow down with > commands such as q stgpool or does it stay fast when the benchmark is > running on all volumes? > Also, what kind of a result does the benchmark give you and what do you > use for your database storage? > > Regards, > Stefan > > > > On Thu, Aug 22, 2019 at 3:36 PM PAC Brion Arnaud < > arnaud.br...@panalpina.com> > wrote: > > > Hi Eric, > > > > These seem to be default values when installing Red Hat Enterprise > > Linux Server 7.3. > > I checked several other machines, all are having the same value. > > > > And following page > > https://stackoverflow.com/questions/55428812/how-the-values-of-kernel- > > parameters-are-defined > > seems to confirm my statement. > > > > Cheers. > > > > Arnaud > > > > > > > > ********************************************************************** > > ********************************************************** > > Backup and Recovery Systems Administrator Panalpina Management Ltd., > > Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 > > Basel/CH > > Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 > > Direct: +41 (61) 226 19 78 > > e-mail: arnaud.br...@panalpina.com > > This electronic message transmission contains information from > > Panalpina and is confidential or privileged. > > This information is intended only for the person (s) named above. If > > you are not the intended recipient, any disclosure, copying, > > distribution or use or any other action based on the contents of this > > information is strictly prohibited. > > > > If you receive this electronic transmission in error, please notify > > the sender by e-mail, telephone or fax at the numbers listed above. > Thank you. > > > > ********************************************************************** > > ********************************************************** > > www.panalpina.com > > > > > > > > -----Original Message----- > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf > > Of Loon, Eric van (ITOP NS) - KLM > > Sent: Thursday, August 22, 2019 2:31 PM > > To: ADSM-L@VM.MARIST.EDU > > Subject: Re: TSM server performance continuing > > > > Hi Arnoud, > > > > I do not understand why your max seg size (kbytes) = 18014398509465599 > > and max total shared memory (kbytes) = 18014397435740096 are so huge. > > If you have 256 GB RAM, like I do, I would expect max seg size > > (kbytes) = > > 268435456 and max total shared memory (kbytes) = 268435456. > > Thanks for your help! > > > > Kind regards, > > Eric van Loon > > Air France/KLM Storage & Backup > > > > > > -----Original Message----- > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf > > Of PAC Brion Arnaud > > Sent: donderdag 22 augustus 2019 09:02 > > To: ADSM-L@VM.MARIST.EDU > > Subject: Re: TSM server performance continuing > > > > Hi Eric, > > > > Running 4 TSM servers at 8.1.6.1 (PowerLinux servers with 256 GB RAM), > > and making use of directory-container storage pools only. > > No server performance or responsiveness issues here anymore, since we > > changed our storage to make use of IBM Isilon. > > > > Here my values : > > > > vmstat 1 > > procs -----------memory---------- ---swap-- -----io---- -system-- > > ------cpu----- > > r b swpd free buff cache si so bi bo in cs us sy > id > > wa st > > 13 0 1420032 1152064 1429248 243848064 0 0 7500 3766 0 0 > > 24 3 61 11 0 > > 13 1 1420032 1046016 1428288 243943488 0 0 614256 80 22427 > 42342 > > 50 4 39 7 0 > > 12 0 1420032 1144640 1426688 243861952 0 0 542840 0 19109 > 33340 > > 49 2 43 6 0 > > 12 4 1420032 883904 1427136 244133824 0 0 577108 0 19777 34954 > > 49 2 44 6 0 > > 12 1 1420032 936576 1425344 244072768 0 0 613152 372 21110 37559 > > 49 2 43 6 0 > > 11 5 1420032 1028096 1425600 243977344 0 0 549024 0 24599 > 47296 > > 48 2 42 8 0 > > 12 0 1420032 1028032 1424832 243983424 0 0 566180 16 21693 > 39976 > > 49 2 43 6 0 > > > > > > ipcs -l > > > > ------ Messages Limits -------- > > max queues system wide = 250880 > > max size of message (bytes) = 65536 > > default max size of queue (bytes) = 65536 > > > > ------ Shared Memory Limits -------- > > max number of segments = 62720 > > max seg size (kbytes) = 18014398509465599 max total shared memory > > (kbytes) = 18014397435740096 min seg size (bytes) = 1 > > > > ------ Semaphore Limits -------- > > max number of arrays = 62720 > > max semaphores per array = 250 > > max semaphores system wide = 256000 > > max ops per semop call = 32 > > semaphore max value = 32767 > > > > > > Specific tuning done : disabling CPU virtualization (led to sever > > hangs when active) > > > > ppc64_cpu --smt=off > > > > > > Cheers. > > > > Arnaud > > > > > > ********************************************************************** > > ********************************************************** > > Backup and Recovery Systems Administrator Panalpina Management Ltd., > > Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 > > Basel/CH > > Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 > > Direct: +41 (61) 226 19 78 > > e-mail: arnaud.br...@panalpina.com > > This electronic message transmission contains information from > > Panalpina and is confidential or privileged. > > This information is intended only for the person (s) named above. If > > you are not the intended recipient, any disclosure, copying, > > distribution or use or any other action based on the contents of this > > information is strictly prohibited. > > > > If you receive this electronic transmission in error, please notify > > the sender by e-mail, telephone or fax at the numbers listed above. > Thank you. > > > > ********************************************************************** > > ********************************************************** > > www.panalpina.com > > > > > > -----Original Message----- > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf > > Of Loon, Eric van (ITOP NS) - KLM > > Sent: Wednesday, August 21, 2019 5:39 PM > > To: ADSM-L@VM.MARIST.EDU > > Subject: TSM server performance continuing > > > > Hi guys, > > > > A few weeks ago I already wrote about the severe performance issues we > > have with our TSM 7.1 servers. In the 'old days' we used to back up > > our clients to TSM 6.3 servers with Data Domains attached. Smaller > > clients backed up through the LAN, large ones through the SAN. > > Our newer servers use LAN-only with directory containers and the > > performance of these servers really sucks. Setting up a session takes > > sometimes almost one minute and a q stg also takes 30 to 50 seconds. I > > noticed that performance is OK when there are no TDP for Oracle > > sessions running, but as soon as they are started the performance > > starts to drop drastically. > > We are really lost on where to look for the cause. I sent numerous > > logs and traces to IBM, but I guess they are out of ideas too since I > > don't hear anything back from them lately. The only thing is that > > supports notices delays in DB2, but they don't know why... > > What I noticed on my TSM server is that as soon as there is a load on > > the server, the blocked queue starts to rise. I would like to know if > > that's something to focus on or not. > > Can some of you please run the "vmstat 1" command on their Linux > > server (preferably one with directory containers too) and let me know > > if you too see values other than 0 in the B column? > > Thank you very much for your help in advance! > > > > Kind regards, > > Eric van Loon > > Air France/KLM Storage & Backup > > ******************************************************** > > For information, services and offers, please visit our web site: > > http://www.klm.com. This e-mail and any attachment may contain > > confidential and privileged material intended for the addressee only. > > If you are not the addressee, you are notified that no part of the > > e-mail or any attachment may be disclosed, copied or distributed, and > > that any other action related to this e-mail or attachment is strictly > > prohibited, and may be unlawful. If you have received this e-mail by > > error, please notify the sender immediately by return e-mail, and delete > this message. > > > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or > > its employees shall not be liable for the incorrect or incomplete > > transmission of this e-mail or any attachments, nor responsible for any > delay in receipt. > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal > > Dutch > > Airlines) is registered in Amstelveen, The Netherlands, with > > registered number 33014286 > > ******************************************************** > > ******************************************************** > > For information, services and offers, please visit our web site: > > http://www.klm.com. This e-mail and any attachment may contain > > confidential and privileged material intended for the addressee only. > > If you are not the addressee, you are notified that no part of the > > e-mail or any attachment may be disclosed, copied or distributed, and > > that any other action related to this e-mail or attachment is strictly > > prohibited, and may be unlawful. If you have received this e-mail by > > error, please notify the sender immediately by return e-mail, and delete > this message. > > > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or > > its employees shall not be liable for the incorrect or incomplete > > transmission of this e-mail or any attachments, nor responsible for any > delay in receipt. > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal > > Dutch > > Airlines) is registered in Amstelveen, The Netherlands, with > > registered number 33014286 > > ******************************************************** > > > ******************************************************** > For information, services and offers, please visit our web site: > http://www.klm.com. This e-mail and any attachment may contain > confidential and privileged material intended for the addressee only. If > you are not the addressee, you are notified that no part of the e-mail or > any attachment may be disclosed, copied or distributed, and that any other > action related to this e-mail or attachment is strictly prohibited, and may > be unlawful. If you have received this e-mail by error, please notify the > sender immediately by return e-mail, and delete this message. > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its > employees shall not be liable for the incorrect or incomplete transmission > of this e-mail or any attachments, nor responsible for any delay in receipt. > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch > Airlines) is registered in Amstelveen, The Netherlands, with registered > number 33014286 > ******************************************************** >
