Hi Andy, We do not have any policy/managementclass with versions higher than 5 across our complex and retain-extra copies is 30-days. I do not know what these servers/applications do so I will ask about the Crypto Keys you referred to in the link.
For the first 3-with this issue, the client is 8.1.0.2 and the OS is Windows 2016. The current one that just successfully finished, is Windows 2012R2 and the client is 7.1.4.4 2/26/2019 12:40:13 PM ANR0806I The ORIONADDWEB\SystemState\NULL\System State\SystemState (fsId=1) file space was deleted for node ORIONADDWEB: 300,346,169 objects were deleted. 2/26/2019 12:40:13 PM ANR0987I Process 1382 for DELETE FILESPACE running in the BACKGROUND processed 300,346,169 items with a completion state of SUCCESS at 12:40:09 PM. On Tue, Feb 26, 2019 at 12:19 PM Andrew Raibeck <stor...@us.ibm.com> wrote: > Hi Zoltan, > > What policy was system state bound to for the nodes that exceed 200 million > objects? Is it possible that the number of versions retained was very > large? > > Another thought is whether the OS of each of these nodes might be affected > by this issue: > > > https://social.technet.microsoft.com/Forums/en-US/e2632b6e-76b9-4640-85b9-698fb55199d8/cprogramdatamicrosoftcryptorsamachinekeys-is-filling-my-disk-space?forum=winservergen > > We have seen huge system state backups that can occur due to that issue, so > if it occurred for those nodes, that is another explanation. > > Regards, > > Andy > > > ____________________________________________________________________________ > > Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2019-02-26 > 11:59:43: > > > From: Zoltan Forray <zfor...@vcu.edu> > > To: ADSM-L@VM.MARIST.EDU > > Date: 2019-02-26 12:01 > > Subject: Re: Bottomless pit > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > > > Hi Andy, > > > > Thank you for clarifying things - a bit. However, why would certain > > nodes have enormously large numbers vs the average when all things are > > equal as far as policies are concerned? > > > > I do see average systemstate object delete counts in the *1-2M range* but > > these 4-nodes are exceeding *200M* each. On this server, I deleted the > > systemstate for 60-nodes. Only 9-exceeded 1M and of those 1-exceeded 2M. > > > > This last node deletion is still running after 3-hours of deletion. > > > > 2019-02-26 08:57:56 Deleting file space ORIONADDWEB\SystemState\NULL > \System > > State\SystemState (fsId=1) (backup data) for node ORIONADDWEB: > 243,029,858 > > objects deleted. > > > > Even the 3-nodes whose deletion failed (after deleting >110M objects > each) > > due to some kind of bitfile error - after restarting the deletions are up > > to 50M+ each and still running? > > > > > > On Tue, Feb 26, 2019 at 11:30 AM Andrew Raibeck <stor...@us.ibm.com> > wrote: > > > > > Hi Zoltan, > > > > > > The large number of objects is normal for system state file spaces. > System > > > state backup uses grouping, with each backed up object being a member > of > > > the group. If the same object is included in multiple groups, then it > will > > > be counted more than once. Each system state backup creates a new > group, so > > > as the number of retained backup versions grows, so does the number of > > > groups, and thus the total object count can grow very large. > > > > > > Best regards, > > > > > > Andy > > > > > > > > > > > ____________________________________________________________________________ > > > > > > > Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com > > > > > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2019-02-26 > > > 10:30:04: > > > > > > > From: Zoltan Forray <zfor...@vcu.edu> > > > > To: ADSM-L@VM.MARIST.EDU > > > > Date: 2019-02-26 10:30 > > > > Subject: Re: Bottomless pit > > > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > > > > > > > Just found another node with a similar issue on a different ISP > server > > > with > > > > different software levels (client=7.1.4.4 and OS=Windows 2012R2). > The > > > node > > > > name is the same so I think the application is, as well. > > > > > > > > 2019-02-26 08:57:56 Deleting file space ORIONADDWEB\SystemState\NULL > > > \System > > > > State\SystemState (fsId=1) (backup data) for node ORIONADDWEB: > > > *129,785,134 > > > > objects deleted*. > > > > > > > > > > > > On Tue, Feb 26, 2019 at 9:15 AM Sasa Drnjevic <sasa.drnje...@srce.hr > > > > > wrote: > > > > > > > > > On 26.2.2019. 15:01, Zoltan Forray wrote: > > > > > > Since all of these systemstate deletes crashed/failed, I > restarted > > > them > > > > > and > > > > > > 2-of the 3 are already up to 5M objects after running for > 30-minutes. > > > > > Will > > > > > > this ever end successfully? > > > > > > > > > > All of mine did finish successfully... > > > > > > > > > > But, none of them had more than 25 mil files deleted. > > > > > > > > > > Wish you luck ;-) > > > > > > > > > > Rgds, > > > > > > > > > > -- > > > > > Sasa Drnjevic > > > > > www.srce.unizg.hr/en/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 25, 2019 at 4:25 PM Sasa Drnjevic > <sasa.drnje...@srce.hr > > > > > > > > > wrote: > > > > > > > > > > > >> FYI, > > > > > >> same here...but my range/ratio was: > > > > > >> > > > > > >> ~2 mil occ to 25 mil deleted objects... > > > > > >> > > > > > >> Never solved the mystery... gave up :-> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> Sasa Drnjevic > > > > > >> www.srce.unizg.hr/en/ > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> On 2019-02-25 20:05, Zoltan Forray wrote: > > > > > >>> Here is a new one....... > > > > > >>> > > > > > >>> We turned off backing up SystemState last week. Now I am going > > > through > > > > > >> and > > > > > >>> deleted the Systemstate filesystems. > > > > > >>> > > > > > >>> Since I wanted to see how many objects would be deleted, I did > a "Q > > > > > >>> OCCUPANCY" and preserved the file count numbers for all Windows > > > nodes > > > > > on > > > > > >>> this server. > > > > > >>> > > > > > >>> For 4-nodes, the delete of their systemstate filespaces has > been > > > > > running > > > > > >>> for 5-hours. A "Q PROC" shows: > > > > > >>> > > > > > >>> 2019-02-25 08:52:05 Deleting file space > > > > > >>> ORION-POLL-WEST\SystemState\NULL\System State\SystemState > (fsId=1) > > > > > >> (backup > > > > > >>> data) for node ORION-POLL-WEST: *105,511,859 objects deleted*. > > > > > >>> > > > > > >>> Considering the occupancy for this node was *~5-Million > objects*, > > > how > > > > > has > > > > > >>> it deleted *105-Million* objects (and counting). The other > 3-nodes > > > in > > > > > >>> question are also up to *>100-Million objects deleted* and none > of > > > them > > > > > >> had > > > > > >>> more than *6M objects* in occupancy? > > > > > >>> > > > > > >>> At this rate, the deleting objects count for 4-nodes > systemstate > > > will > > > > > >>> exceed 50% of the total occupancy objects on this server that > > > houses > > > > > the > > > > > >>> backups for* 263-nodes*? > > > > > >>> > > > > > >>> I vaguely remember some bug/APAR about systemstate backups > being > > > > > >>> large/slow/causing performance problems with expiration but > these > > > nodes > > > > > >>> client levels are fairly current (8.1.0.2 - staying below the > > > > > >> 8.1.2/SSL/TLS > > > > > >>> enforcement levels) and the ISP server is 7.1.7.400. All of > these > > > are > > > > > >>> Windows 2016, if that matters. > > > > > >>> > > > > > >>> -- > > > > > >>> *Zoltan Forray* > > > > > >>> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > > > >>> Xymon Monitor Administrator > > > > > >>> VMware Administrator > > > > > >>> Virginia Commonwealth University > > > > > >>> UCC/Office of Technology Services > > > > > >>> www.ucc.vcu.edu > > > > > >>> zfor...@vcu.edu - 804-828-4807 > > > > > >>> Don't be a phishing victim - VCU and other reputable > organizations > > > will > > > > > >>> never use email to request that you reply with your password, > > > social > > > > > >>> security number or confidential personal information. For more > > > details > > > > > >>> visit https://urldefense.proofpoint.com/v2/url? > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > > > > > > > > > > > > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e= > > > > > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > *Zoltan Forray* > > > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > > > > Xymon Monitor Administrator > > > > > > VMware Administrator > > > > > > Virginia Commonwealth University > > > > > > UCC/Office of Technology Services > > > > > > www.ucc.vcu.edu > > > > > > zfor...@vcu.edu - 804-828-4807 > > > > > > Don't be a phishing victim - VCU and other reputable > organizations > > > will > > > > > > never use email to request that you reply with your password, > social > > > > > > security number or confidential personal information. For more > > > details > > > > > > visit https://urldefense.proofpoint.com/v2/url? > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > > > > > > > > > > > > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e= > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Zoltan Forray* > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > > Xymon Monitor Administrator > > > > VMware Administrator > > > > Virginia Commonwealth University > > > > UCC/Office of Technology Services > > > > www.ucc.vcu.edu > > > > zfor...@vcu.edu - 804-828-4807 > > > > Don't be a phishing victim - VCU and other reputable organizations > will > > > > never use email to request that you reply with your password, social > > > > security number or confidential personal information. For more > details > > > > visit https://urldefense.proofpoint.com/v2/url? > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > > > > > > > > > > > > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e= > > > > > > > > > > > > > > > > > -- > > *Zoltan Forray* > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > Xymon Monitor Administrator > > VMware Administrator > > Virginia Commonwealth University > > UCC/Office of Technology Services > > www.ucc.vcu.edu > > zfor...@vcu.edu - 804-828-4807 > > Don't be a phishing victim - VCU and other reputable organizations will > > never use email to request that you reply with your password, social > > security number or confidential personal information. For more details > > visit https://urldefense.proofpoint.com/v2/url? > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=hQoTgXdTWRp- > > > > c4tE7WF3GSnI5KTnZ9r6rwUQJqLtLu0&s=1kN_Gmc2Qiaab36WLtyjbl66sf5H54wTfT4PyG9zCj8&e= > > > > -- *Zoltan Forray* Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://phishing.vcu.edu/
