Thanks for the confirmations. It took multiple attempts over 3-days to finally delete all of the systemstate objects for these 4-nodes, totalling over 1.2B objects deleted. The deletes kept constantly failing with errors like this:
2/26/2019 2:24:27 PM ANR0106E imfsdel.c(2723): Unexpected error 4522 fetching row in table "Backup.Objects". 2/26/2019 2:24:27 PM ANR1880W Server transaction was canceled because of a conflicting lock on table BACKUP_OBJECTS. 2/26/2019 2:24:27 PM ANR1893E Process 2399 for DELETE FILESPACE completed with a completion state of FAILURE. 2/26/2019 2:46:04 PM ANR0106E imfsdel.c(2723): Unexpected error 4522 fetching row in table "Backup.Objects". 2/26/2019 2:46:04 PM ANR1880W Server transaction was canceled because of a conflicting lock on table BACKUP_OBJECTS. 2/26/2019 2:46:04 PM ANR1893E Process 2400 for DELETE FILESPACE completed with a completion state of FAILURE. 2/26/2019 5:03:16 PM ANR0106E imfsdel.c(2723): Unexpected error 4522 fetching row in table "Backup.Objects". 2/26/2019 5:03:16 PM ANR1880W Server transaction was canceled because of a conflicting lock on table BACKUP_OBJECTS. 2/26/2019 5:03:16 PM ANR1893E Process 2404 for DELETE FILESPACE completed with a completion state of FAILURE. On Tue, Feb 26, 2019 at 4:55 PM Harris, Steven < steven.har...@btfinancialgroup.com> wrote: > Zoltan > > I had something similar. Prod node of application had a reasonable number > of systemstate objects. Two nonprod nodes of same application had huge > numbers of systemstate. This first came to my attention when daily > expiration was taking 3 or more days instead of the usual 30 minutes. > > As far as I can tell the nonprod nodes were set up in a lazy manner and > the database and application were dumped on the C: drive in a place that is > part of systemstate. I excluded the systemstate for these two nodes and > deleted the filespaces - which took a week or more. The local ticket is > still open with the application people, almost a year on. > > Hope that helps > > Steve. > > Steven Harris > TSM Admin/Consultant > Canberra Australia. > > > -----Original Message----- > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of > Deschner, Roger Douglas > Sent: Wednesday, 27 February 2019 6:49 AM > To: ADSM-L@VM.MARIST.EDU > Subject: Re: [ADSM-L] Bottomless pit > > I set up a cron job that does filespace and node deletions in a batch on > weekends. Then if it takes a long time, I don't care. I set this up back on > server version 5, when deletions took a REALLY long time, and I kept it in > V6 to deal with exactly this issue with System State. We've been using > Client Option Sets to prevent System State backup for several years now, > but sometimes one slips through. Also we now have actual data filespaces > with many millions of stored objects, and sometimes one of those needs to > be deleted, another reason to batch deletions on weekends. > > Roger Deschner > University of Illinois at Chicago > "I have not lost my mind -- it is backed up on tape somewhere." > ________________________________________ > From: Sasa Drnjevic <sasa.drnje...@srce.hr> > Sent: Monday, February 25, 2019 15:25 > Subject: Re: Bottomless pit > > FYI, > same here...but my range/ratio was: > > ~2 mil occ to 25 mil deleted objects... > > Never solved the mystery... gave up :-> > > > -- > Sasa Drnjevic > www.srce.unizg.hr/en/ > > > > > On 2019-02-25 20:05, Zoltan Forray wrote: > > Here is a new one....... > > > > We turned off backing up SystemState last week. Now I am going > > through and deleted the Systemstate filesystems. > > > > Since I wanted to see how many objects would be deleted, I did a "Q > > OCCUPANCY" and preserved the file count numbers for all Windows nodes > > on this server. > > > > For 4-nodes, the delete of their systemstate filespaces has been > > running for 5-hours. A "Q PROC" shows: > > > > 2019-02-25 08:52:05 Deleting file space > > ORION-POLL-WEST\SystemState\NULL\System State\SystemState (fsId=1) > > (backup > > data) for node ORION-POLL-WEST: *105,511,859 objects deleted*. > > > > Considering the occupancy for this node was *~5-Million objects*, how > > has it deleted *105-Million* objects (and counting). The other > > 3-nodes in question are also up to *>100-Million objects deleted* and > > none of them had more than *6M objects* in occupancy? > > > > At this rate, the deleting objects count for 4-nodes systemstate will > > exceed 50% of the total occupancy objects on this server that houses > > the backups for* 263-nodes*? > > > > I vaguely remember some bug/APAR about systemstate backups being > > large/slow/causing performance problems with expiration but these > > nodes client levels are fairly current (8.1.0.2 - staying below the > > 8.1.2/SSL/TLS enforcement levels) and the ISP server is 7.1.7.400. > > All of these are Windows 2016, if that matters. > > > > -- > > *Zoltan Forray* > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon > > Monitor Administrator VMware Administrator Virginia Commonwealth > > University UCC/Office of Technology Services www.ucc.vcu.edu > > zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and > > other reputable organizations will never use email to request that you > > reply with your password, social security number or confidential > > personal information. For more details visit http://phishing.vcu.edu/ > > > > This message and any attachment is confidential and may be privileged or > otherwise protected from disclosure. You should immediately delete the > message if you are not the intended recipient. If you have received this > email by mistake please delete it from your system; you should not copy the > message or disclose its content to anyone. > > This electronic communication may contain general financial product advice > but should not be relied upon or construed as a recommendation of any > financial product. The information has been prepared without taking into > account your objectives, financial situation or needs. You should consider > the Product Disclosure Statement relating to the financial product and > consult your financial adviser before making a decision about whether to > acquire, hold or dispose of a financial product. > > For further details on the financial product please go to > http://www.bt.com.au > > Past performance is not a reliable indicator of future performance. > -- *Zoltan Forray* Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://phishing.vcu.edu/
