Maurice Drane - Engineer - ( Data Centres ) Portfolio Management and IT Services Cardiff University Data Centre Operations 02920876159 dr...@cardiff.ac.uk
-----Original Message----- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Strilecky, Douglas Sent: 08 December 2016 12:34 To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] ADSM-L Digest - 6 Dec 2016 to 7 Dec 2016 (#2016-241) Unsubscribe please -----Original Message----- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of ADSM-L automatic digest system Sent: Thursday, December 08, 2016 12:00 AM To: ADSM-L@VM.MARIST.EDU Subject: ADSM-L Digest - 6 Dec 2016 to 7 Dec 2016 (#2016-241) There are 7 messages totalling 489 lines in this issue. Topics of the day: 1. 3 Generation (GFS) Backup Schedule setup using TSM (2) 2. problem after 7.1.5 upgrade: ANS1708E ... Only root can do this operation (2) 3. Antwort: [ADSM-L] problem after 7.1.5 upgrade: ANS1708E ... Only root can do this operation (2) 4. Strange problem with the same ACTIVELOG files opened multiple times ---------------------------------------------------------------------- Date: Wed, 7 Dec 2016 07:32:34 +0100 From: Markus Engelhard <markus.engelh...@bundesbank.de> Subject: 3 Generation (GFS) Backup Schedule setup using TSM Hello Andreas, sorry to hear tat this sort of thing is still happening this century... Have you had a look at container pools and dedup? It would ease the storage-side pain quite a bit, with drawbacks in taking copies to tape. I have been testing the dedup + compression feature, an apart from handling of copies of the data, Data reduction has been impressive even with incrforever data with limited influence on performance. Test was with SP 7.1.6.0. Kind regards, Markus ------------------------------ Date: Wed, 7 Dec 2016 07:18:33 +0000 From: =?iso-8859-1?Q?Anders_R=E4ntil=E4?= <and...@rantila.com> Subject: Re: 3 Generation (GFS) Backup Schedule setup using TSM Hi, Question: 19 copies of the "same" data. Each copy 500GB. Tape library use= d a storage. Would like a cheaper solution. Let's start: 19*500GB is 9=BDTB. Disk drives are up to 6TB today.=20 =09 The data will fit in 2 disk drives - without using dedup and compression. Apply dedup and compression and the data will probably fit in a USB memory= stick (1TB). The obvious answer is: This is 2016 and tape is dead since many years! =20 If you skip the tape library and use to modern design with container pools = you will save a lot of money. Both on the hardware (tape libs are not cheap= ) and most important on the cost of licenses. /Anders R=E4ntil=E4 =09 =20 ------------------------------ Date: Wed, 7 Dec 2016 14:45:40 +0000 From: "Rhodes, Richard L." <rrho...@firstenergycorp.com> Subject: problem after 7.1.5 upgrade: ANS1708E ... Only root can do this operation Just upgrade a TSM server from 6.3.5 to 7.1.5 on AIX. This morning our Oracle DBA's are reporting this error: ANS1708E Backup operation failed. Only a root user can do this operation= <http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> And on the TSM server we see: ANR1890W Backup operation for session 8549 node xxxx failed due to root-o= nly restriction<http://www-01.ibm.com/support/docview.wss?uid=3Dswg216511= 08> The Oracle DBA's run Oracle backups to disk, then call dsmc to push the b= ackup to TSM. They do this AS THE ORACLE USER, not root. I found a IBM doc that said to set the node parm "Users allowed to back <= http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> up" from roo= t to ALL. I checked, and all the nodes that had this problem _ARE_ set t= o ALL. doc: http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108 Any thougts are welcome! cat z.ksh for i in <http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> <l= ist of nodes> do<http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> ./run_cmd.ksh -l tsm1 "q node $i f=3Dd" | grep -i "Users allowed to b= ack up"<http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> done<http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> =2E/z.ksh<http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> Users allowed to back up: All<http://www-01.ibm.com/support/docview.wss?u= id=3Dswg21651108> -----------------------------------------=0D The information contained in this message is intended only for the person= al and confidential use of the recipient(s) named above. If the reader of= this message is not the intended recipient or an agent responsible for d= elivering it to the intended recipient, you are hereby notified that you = have received this document in error and that any review, dissemination, = distribution, or copying of this message is strictly prohibited. If you h= ave received this communication in error, please notify us immediately, a= nd delete the original message. ------------------------------ Date: Wed, 7 Dec 2016 15:53:09 +0100 From: Rainer Holzinger <rainer.holzin...@empalis.com> Subject: Antwort: [ADSM-L] problem after 7.1.5 upgrade: ANS1708E ... Only root can do this operation Hi Richard, option 'backupinitiation off' in dsmserv.opt and recycle of the instance die help at my site some months ago. HTH, Rainer Von: "Rhodes, Richard L." <rrho...@firstenergycorp.com> An: ADSM-L@VM.MARIST.EDU Datum: 07.12.2016 15:47 Betreff: [ADSM-L] problem after 7.1.5 upgrade: ANS1708E ... Only root can do this operation Gesendet von: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> Just upgrade a TSM server from 6.3.5 to 7.1.5 on AIX. This morning our Oracle DBA's are reporting this error: ANS1708E Backup operation failed. Only a root user can do this operation< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> And on the TSM server we see: ANR1890W Backup operation for session 8549 node xxxx failed due to root-only restriction< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> The Oracle DBA's run Oracle backups to disk, then call dsmc to push the backup to TSM. They do this AS THE ORACLE USER, not root. I found a IBM doc that said to set the node parm "Users allowed to back < http://www-01.ibm.com/support/docview.wss?uid=swg21651108> up" from root to ALL. I checked, and all the nodes that had this problem _ARE_ set to ALL. doc: http://www-01.ibm.com/support/docview.wss?uid=swg21651108 Any thougts are welcome! cat z.ksh for i in <http://www-01.ibm.com/support/docview.wss?uid=swg21651108> <list of nodes> do<http://www-01.ibm.com/support/docview.wss?uid=swg21651108> ./run_cmd.ksh -l tsm1 "q node $i f=d" | grep -i "Users allowed to back up"<http://www-01.ibm.com/support/docview.wss?uid=swg21651108> done<http://www-01.ibm.com/support/docview.wss?uid=swg21651108> ./z.ksh<http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=swg21651108> ----------------------------------------- The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message. ------------------------------ Date: Wed, 7 Dec 2016 15:04:25 +0000 From: chris rees <rees_ch...@hotmail.com> Subject: Re: problem after 7.1.5 upgrade: ANS1708E ... Only root can do this operation SGkuLg0KDQpjaGVjayBzZXJ2ZXIgcGFyYW1ldGVyLg0KDQpCYWNrdXBpbml0aWF0aW9ucm9vdA0K DQpJbiBxIG9wdA0KDQpJIGhpdCB0aGUgc2FtZSB0aGluZyB3aXRoIG9yYWNsZSBiYWNrdXBzIHJl Y2VudGx5IGFmdGVyIGEgNi4zIHRvIDcuMSB1cGdyYWRlDQo= ------------------------------ Date: Wed, 7 Dec 2016 15:08:22 +0000 From: "Rhodes, Richard L." <rrho...@firstenergycorp.com> Subject: Re: Antwort: [ADSM-L] problem after 7.1.5 upgrade: ANS1708E ... Only root can do this operation I see. I just found the option and set it live via "setopt backupinitiationroot = off". That fixed the problem live, and from your comment I checked the d= smserv.opt and it now has a "BACKUPINITIATIONROOT OFF" in it, so TSM stuc= k it in. Thanks! Rick -----Original Message----- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of = Rainer Holzinger Sent: Wednesday, December 07, 2016 9:53 AM To: ADSM-L@VM.MARIST.EDU Subject: *EXTERNAL* Antwort: [ADSM-L] problem after 7.1.5 upgrade: ANS170= 8E ... Only root can do this operation Hi Richard, option 'backupinitiation off' in dsmserv.opt and recycle of the instanc= e die help at my site some months ago. HTH, Rainer Von: "Rhodes, Richard L." <rrho...@firstenergycorp.com> An: ADSM-L@VM.MARIST.EDU Datum: 07.12.2016 15:47 Betreff: [ADSM-L] problem after 7.1.5 upgrade: ANS1708E ... Only root can do this operation Gesendet von: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> Just upgrade a TSM server from 6.3.5 to 7.1.5 on AIX. This morning our Oracle DBA's are reporting this error: ANS1708E Backup operation failed. Only a root user can do this operation= < http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> And on the TSM server we see: ANR1890W Backup operation for session 8549 node xxxx failed due to root-only restriction< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> The Oracle DBA's run Oracle backups to disk, then call dsmc to push the backup to TSM. They do this AS THE ORACLE USER, not root. I found a IBM doc that said to set the node parm "Users allowed to back < http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> up" from roo= t to ALL. I checked, and all the nodes that had this problem _ARE_ set to ALL. doc: http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108 Any thougts are welcome! cat z.ksh for i in <http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> <l= ist of nodes> do<http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> ./run_cmd.ksh -l tsm1 "q node $i f=3Dd" | grep -i "Users allowed to b= ack up"<http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> done<http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> =2E/z.ksh<http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> Users allowed to back up: All< http://www-01.ibm.com/support/docview.wss?uid=3Dswg21651108> ----------------------------------------- The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message. -----------------------------------------=0D The information contained in this message is intended only for the person= al and confidential use of the recipient(s) named above. If the reader of= this message is not the intended recipient or an agent responsible for d= elivering it to the intended recipient, you are hereby notified that you = have received this document in error and that any review, dissemination, = distribution, or copying of this message is strictly prohibited. If you h= ave received this communication in error, please notify us immediately, a= nd delete the original message. ------------------------------ Date: Wed, 7 Dec 2016 10:48:09 -0500 From: "Dury, John C." <jd...@duqlight.com> Subject: Re: Strange problem with the same ACTIVELOG files opened multiple times Well at least I feel a little better that it is happening to someone else a= lso. We were also having a problem with the DB Backup completely hanging. I= thought maybe it was just stuck, but it actually happened on a Friday and = was still running as of Monday morning but making no progress. Originally t= he DB backup was going to a an NFS mounted filesystem but after moving to a= local RAID 1 group and changing some other recommendations in /etc/sysctl.= conf, the hung DB backup issue seems to be solved. Our DB and ACTIVELOG fil= es are all on local SSD also. The BACKUPPOOL volumes are on a NFS (10G) mou= nted filesystem that lives on a Data Domain. I do have an open PMR with IBM= about the excessive number of times each /tsmactivelog S00* file is open. = It has been sent to development for investigation. I will post any updates = here. John Graham Stewart<https://www.mail-archive.com/search?l=3dads...@vm.marist.edu= &q=3Dfrom:%22Graham+Stewart%22> Tue, 06 Dec 2016 11:09:00 -0800<https://www= .mail-archive.com/search?l=3dads...@vm.marist.edu&q=3Ddate:20161206> We see something very similar (TSM 7.1.6.0, RHEL 7.2): -83570 open active S00*.log files, under db2sysc -193 S00*.log files -DB backups sometimes (but not always) slow down, showing no progress, but eventually get going again. We've never rebooted under this scenario, and the DB backups always complete successfully, but some days take twice as long as other days. No NFS. All local disk: SSD (DB and logs) and SATA (storage pools). No deduplication. We haven't yet placed a support call with IBM about the variance in DB backup times, but intend to. -- Graham Stewart Network and Storage Services Manager Information Technology Services University of Toronto Libraries 416-978-6337 On 12/06/2016 01:41 PM, Dury, John C. wrote: We continue to see this on all 4 TSM 7.x servers. I do not see this behavio= r on the TSM 6.x servers at all. Anyone else running TSM 7.x on RHEL 7.x? The /tsmactivelog/NODE0000/LOGSTREAM0000/S0018048.LOG is open simultaneousl= y a total of 386 times. This can't be normal behavior. We have two TSM v7.1.7.0 server running on different RHEL 7.x servers. The primary storage pool is BACKUPPOOL which has it's volumes in the local OS mounted as NFS volumes across a 10g network connection. The volumes live on= the Data domain which does it's own deduplication in the background. We have a schedule that does a full TSM DB backup daily. The target is a separate fil= e system but is also mounted as a NFS on the Data Domain across a 10g network connection. The TSM Active log is mounted on local disk. The TSM DB is also mounted locally on a RAID of SSD drives for performance. The issue I am seeing is that although there are only 261 S00*.LOG files in /tsmactivelog, they appear to all be open multiple times. "lsof|grep -i tsmactive|wc -l" command tells me that there are 94576 files opened in /tsmactivelog. The process that has the /TSMACTIVELOG files opened is db2sysc. I've never seen this on our TSM 6.x server. It's almost as if the active log file is opened= but then never closed. It isn't a gradual climb to a high number of mounts eith= er. 10 minutes after booting the server, there are an excessive number of files mounted in /tsmactivelog. This behavior is happening even when the server i= s extremely idle with very few (or none) sessions and/or processes running. The issue we keep seeing every few days is processes like the TSM full DB backup, runs for a few minutes, and then progress just stops. After cancell= ing it, it never comes down so I am forced to HALT the server and then reboot i= t. It seems like the excessive number of opened files in /tsmactivelog and the hanging DB Backup feel related but I am not sure. I've been working with IBM on the hanging processes but so far they are als= o stumped but they agree the two issues seem like they should be related. I'm hoping someone out there might have some ideas. ------------------------------ End of ADSM-L Digest - 6 Dec 2016 to 7 Dec 2016 (#2016-241) *********************************************************** Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ”Contact Us” forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email.
