Here is the 6.3.6.000 expiration problem APAR description: IT17642: SLOW EXPIRATION AFTER UPGRADE TO 6.3.6.000 SERVER
APAR status - OPEN - *Error description* After upgrading the server to 6.3.6.000 (or higher) and 7.1.3(or higher) expiration might "hang" on a node while expiring backup data only. Expiration is not actually hung it is still processing but very slowly due to a non-optimized SQL/Select. A change to this SQL/Select occurred between 6.3.5.0 and 6.3.6.0. This code change was implemented in servers 6.3.6.0+ and 7.1.3.0 to 7.1.7. There have been no reported problems at 7.1.3 or above. *Customer/L2 Diagnostics* (if applicable) >From the server activity log - review expiration entries. Expiration processes by node/filespace. If it is seen that 1 node does not complete expiration in a timely manner your server may be experiencing this issue. By default Expiration restarts where it left off. If it starts on the same node repeatedly this may also be another indicator you are hitting this issue. Servermon.pl output can be gathered for 1 hour and needs to collect the db2 information. From the *db2.txt output search for the "ELAPSED_TIME_SEC" section. If you are experiencing this issue you will find the following select and a high Elapsed execution time - in this example 25310 seconds (which is a little over 7 hrs): ELAPSED_TIME_SEC STATE STMT_TEXT ---------------- ----- 25310 EXEC SELECT IMBK.OBJID, IMBK.BFSIZE, CASE WHEN IMBK.GROUPTYPE IS NOT NULL AND BITAND(IMBK.GROUPTYPE,458752)>0 THEN 1 END AS ISIMGLLEAD, CASE WHEN IMBK.GROUPTYPE IS NOT NULL AND BITAND(IMBK.GROUPTYPE,7)>0 THEN 1 END AS ISIMGLMEMB FROM TSMDB1.BACKUP_OBJECTS IMBK WHERE IMBK.NODEID=? AND IMBK.FSID=? AND IMBK.OBJTYPE=? AND IMBK.STATE=2 AND IMBK.MCID=? AND (BITAND(IMBK.GROUPTYPE,CAST(? AS INT))=0 OR IMBK.GROUPTYPE=131072 OR IMBK.GROUPTYPE IS NULL ) AND ( IMBK.DEACDATE<='1956-10-11 00:00:00' OR ( IMBK.DEACDATE<=? AND 1 record(s) selected. *NOTE* The ELAPSED_TIME_SEC select has been truncated in the DB2 collection and this is how it will appear in the output. *Platforms affected:* Windows, Unix/Linux 6.3.6, 7.1.3 and higher *Initial Impact:* Medium *Additional Keywords:* IT07612 |MDVREGR 6.3.6.0| |MDVREGR 7.1.3.0| *Local fix* 1) Contact Tivoli Storage Manager Support for a "Diagnostic" server (6.3.6.000 only) which has been made available to correct this problem until the fix is available 2) If you can not apply the DIAG server the work-around would be to run expiration: excluding the "slow" node(s) and restarting expiration at the first node. This can be done by adding the 2 following undocumented parameters to the Expire Inventory command: restart=no excludenodes=node1,node2,etc (where node1, node2 are the names of the slow nodes) On Thu, Nov 10, 2016 at 12:14 PM, Zoltan Forray <zfor...@vcu.edu> wrote: > Not sure. We have always used "expire inventory resource=8" with no > issues until 6.3.6.000 > > Since the APAR info is behind IBM's support wall (don't get me started on > the inability to get firmware updates for my IBM TS3500 library from IBM), > I don't know if I can post it here. Hopefully someone from IBM can > chime-in. > > > > On Thu, Nov 10, 2016 at 11:34 AM, Henrik Ahlgren <pa...@seestieto.com> > wrote: > >> Thanks Zoltan, – I somehow missed your earlier posting about this topic. >> I'll try to get my hands on the IT17642 article (why on earth does IBM >> show it only to "authorized" users? I hate this vendor BS.). >> >> But in addition to slowness, it seems that EXPIRE INVENTORY NODE=* at >> least attempts to expire all nodes, unlike EXPIRE INVENTORY without the >> NODE or DOMAIN options. Have you seen this? >> >> On Thu, Nov 10, 2016, at 18:07, Zoltan Forray wrote: >> > Henrik, >> > >> > If you check recent posts here, you will see there is a major issue with >> > 6.3.6.000 server and expirations. I have been fighting it since I >> > upgrade >> > one server. IBM has a hotfix to address it or you can simply upgrade to >> > 7.1.6, which is what I did. You must contact IBM support for it. They >> > say >> > the impact has been minimal and hit-or-miss (not everyone who upgrades >> to >> > 6.3.6.000 has the problem) so an official patch has not been released so >> > far. >> > >> > Search Google for IT17642. >> > >> > On Thu, Nov 10, 2016 at 10:50 AM, Henrik Ahlgren <pa...@seestieto.com> >> > wrote: >> > >> > > The documentation for EXPIRE INVENTORY command says "If you do not >> > > specify either NODE or DOMAIN with value, data for all nodes is >> > > processes". That is in fact how it works up to 6.3.6.100, but it seems >> > > that there is a bug in 6.3.6 that causes it to only expire a handful >> of >> > > randomly selected nodes, unless you specify "NODE=*". >> > > >> > > Have any of you run into this? If you're using 6.3.6, I would urge you >> > > to check the expire history, i.e. >> > > >> > > select date(start_time) as date,count(*) as nodes,sum(affected) as >> > > expired >> > > from summary >> > > where activity='EXPIRED' >> > > group by date(start_time) >> > > >> > >> > >> > >> > -- >> > *Zoltan Forray* >> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator >> > Xymon Monitor Administrator >> > VMware Administrator (in training) >> > Virginia Commonwealth University >> > UCC/Office of Technology Services >> > www.ucc.vcu.edu >> > zfor...@vcu.edu - 804-828-4807 >> > Don't be a phishing victim - VCU and other reputable organizations will >> > never use email to request that you reply with your password, social >> > security number or confidential personal information. For more details >> > visit http://infosecurity.vcu.edu/phishing.html >> > > > > -- > *Zoltan Forray* > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > Xymon Monitor Administrator > VMware Administrator (in training) > Virginia Commonwealth University > UCC/Office of Technology Services > www.ucc.vcu.edu > zfor...@vcu.edu - 804-828-4807 > Don't be a phishing victim - VCU and other reputable organizations will > never use email to request that you reply with your password, social > security number or confidential personal information. For more details > visit http://infosecurity.vcu.edu/phishing.html > -- *Zoltan Forray* Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator (in training) Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://infosecurity.vcu.edu/phishing.html
