Hi Zoltan, There is nothing in the client code you are using that would cause excessive backups. If you happen to have backup logs going far back enough, those might show an excessive number of system state objects. Normally I would expect the number of backup objects N to be 90000 < N < 150000. that is a roughly estimated range, depends on the individual system, and maybe N could be a little higher... but my radar would certainly be alerted if it was more than 200,000 and growing daily.
Regards, Andy ____________________________________________________________________________ Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2019-02-26 12:52:27: > From: Zoltan Forray <zfor...@vcu.edu> > To: ADSM-L@VM.MARIST.EDU > Date: 2019-02-26 12:52 > Subject: Re: Bottomless pit > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > Hi Andy, > > We do not have any policy/managementclass with versions higher than 5 > across our complex and retain-extra copies is 30-days. I do not know what > these servers/applications do so I will ask about the Crypto Keys you > referred to in the link. > > For the first 3-with this issue, the client is 8.1.0.2 and the OS is > Windows 2016. The current one that just successfully finished, is Windows > 2012R2 and the client is 7.1.4.4 > > 2/26/2019 12:40:13 PM ANR0806I The ORIONADDWEB\SystemState\NULL\System > State\SystemState (fsId=1) file space was deleted for node ORIONADDWEB: > 300,346,169 objects were deleted. > 2/26/2019 12:40:13 PM ANR0987I Process 1382 for DELETE FILESPACE running in > the BACKGROUND processed 300,346,169 items with a completion state of > SUCCESS at 12:40:09 PM. > > > On Tue, Feb 26, 2019 at 12:19 PM Andrew Raibeck <stor...@us.ibm.com> wrote: > > > Hi Zoltan, > > > > What policy was system state bound to for the nodes that exceed 200 million > > objects? Is it possible that the number of versions retained was very > > large? > > > > Another thought is whether the OS of each of these nodes might be affected > > by this issue: > > > > > > https://urldefense.proofpoint.com/v2/url? > u=https-3A__social.technet.microsoft.com_Forums_en-2DUS_e2632b6e-2D76b9-2D4640-2D85b9-2D698fb55199d8_cprogramdatamicrosoftcryptorsamachinekeys-2Dis-2Dfilling-2Dmy-2Ddisk-2Dspace-3Fforum-3Dwinservergen&d=DwIBaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=N9AQefDr2d0WLgIG8G7ONWkk-3POpjbBbVM3mI2lzuM&s=rNhhFizJITYew56twqREPr482w- > Hss5ZsFp5-UY_W58&e= > > > > We have seen huge system state backups that can occur due to that issue, so > > if it occurred for those nodes, that is another explanation. > > > > Regards, > > > > Andy > > > > > > ____________________________________________________________________________ > > > > Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com > > > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2019-02-26 > > 11:59:43: > > > > > From: Zoltan Forray <zfor...@vcu.edu> > > > To: ADSM-L@VM.MARIST.EDU > > > Date: 2019-02-26 12:01 > > > Subject: Re: Bottomless pit > > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > > > > > Hi Andy, > > > > > > Thank you for clarifying things - a bit. However, why would certain > > > nodes have enormously large numbers vs the average when all things are > > > equal as far as policies are concerned? > > > > > > I do see average systemstate object delete counts in the *1-2M range* but > > > these 4-nodes are exceeding *200M* each. On this server, I deleted the > > > systemstate for 60-nodes. Only 9-exceeded 1M and of those 1-exceeded 2M. > > > > > > This last node deletion is still running after 3-hours of deletion. > > > > > > 2019-02-26 08:57:56 Deleting file space ORIONADDWEB\SystemState\NULL > > \System > > > State\SystemState (fsId=1) (backup data) for node ORIONADDWEB: > > 243,029,858 > > > objects deleted. > > > > > > Even the 3-nodes whose deletion failed (after deleting >110M objects > > each) > > > due to some kind of bitfile error - after restarting the deletions are up > > > to 50M+ each and still running? > > > > > > > > > On Tue, Feb 26, 2019 at 11:30 AM Andrew Raibeck <stor...@us.ibm.com> > > wrote: > > > > > > > Hi Zoltan, > > > > > > > > The large number of objects is normal for system state file spaces. > > System > > > > state backup uses grouping, with each backed up object being a member > > of > > > > the group. If the same object is included in multiple groups, then it > > will > > > > be counted more than once. Each system state backup creates a new > > group, so > > > > as the number of retained backup versions grows, so does the number of > > > > groups, and thus the total object count can grow very large. > > > > > > > > Best regards, > > > > > > > > Andy > > > > > > > > > > > > > > > > ____________________________________________________________________________ > > > > > > > > > > Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com > > > > > > > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2019-02-26 > > > > 10:30:04: > > > > > > > > > From: Zoltan Forray <zfor...@vcu.edu> > > > > > To: ADSM-L@VM.MARIST.EDU > > > > > Date: 2019-02-26 10:30 > > > > > Subject: Re: Bottomless pit > > > > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > > > > > > > > > Just found another node with a similar issue on a different ISP > > server > > > > with > > > > > different software levels (client=7.1.4.4 and OS=Windows 2012R2). > > The > > > > node > > > > > name is the same so I think the application is, as well. > > > > > > > > > > 2019-02-26 08:57:56 Deleting file space ORIONADDWEB\SystemState \NULL > > > > \System > > > > > State\SystemState (fsId=1) (backup data) for node ORIONADDWEB: > > > > *129,785,134 > > > > > objects deleted*. > > > > > > > > > > > > > > > On Tue, Feb 26, 2019 at 9:15 AM Sasa Drnjevic <sasa.drnje...@srce.hr > > > > > > > wrote: > > > > > > > > > > > On 26.2.2019. 15:01, Zoltan Forray wrote: > > > > > > > Since all of these systemstate deletes crashed/failed, I > > restarted > > > > them > > > > > > and > > > > > > > 2-of the 3 are already up to 5M objects after running for > > 30-minutes. > > > > > > Will > > > > > > > this ever end successfully? > > > > > > > > > > > > All of mine did finish successfully... > > > > > > > > > > > > But, none of them had more than 25 mil files deleted. > > > > > > > > > > > > Wish you luck ;-) > > > > > > > > > > > > Rgds, > > > > > > > > > > > > -- > > > > > > Sasa Drnjevic > > > > > > www.srce.unizg.hr/en/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 25, 2019 at 4:25 PM Sasa Drnjevic > > <sasa.drnje...@srce.hr > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> FYI, > > > > > > >> same here...but my range/ratio was: > > > > > > >> > > > > > > >> ~2 mil occ to 25 mil deleted objects... > > > > > > >> > > > > > > >> Never solved the mystery... gave up :-> > > > > > > >> > > > > > > >> > > > > > > >> -- > > > > > > >> Sasa Drnjevic > > > > > > >> www.srce.unizg.hr/en/ > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> On 2019-02-25 20:05, Zoltan Forray wrote: > > > > > > >>> Here is a new one....... > > > > > > >>> > > > > > > >>> We turned off backing up SystemState last week. Now I am going > > > > through > > > > > > >> and > > > > > > >>> deleted the Systemstate filesystems. > > > > > > >>> > > > > > > >>> Since I wanted to see how many objects would be deleted, I did > > a "Q > > > > > > >>> OCCUPANCY" and preserved the file count numbers for all Windows > > > > nodes > > > > > > on > > > > > > >>> this server. > > > > > > >>> > > > > > > >>> For 4-nodes, the delete of their systemstate filespaces has > > been > > > > > > running > > > > > > >>> for 5-hours. A "Q PROC" shows: > > > > > > >>> > > > > > > >>> 2019-02-25 08:52:05 Deleting file space > > > > > > >>> ORION-POLL-WEST\SystemState\NULL\System State\SystemState > > (fsId=1) > > > > > > >> (backup > > > > > > >>> data) for node ORION-POLL-WEST: *105,511,859 objects deleted*. > > > > > > >>> > > > > > > >>> Considering the occupancy for this node was *~5-Million > > objects*, > > > > how > > > > > > has > > > > > > >>> it deleted *105-Million* objects (and counting). The other > > 3-nodes > > > > in > > > > > > >>> question are also up to *>100-Million objects deleted* and none > > of > > > > them > > > > > > >> had > > > > > > >>> more than *6M objects* in occupancy? > > > > > > >>> > > > > > > >>> At this rate, the deleting objects count for 4-nodes > > systemstate > > > > will > > > > > > >>> exceed 50% of the total occupancy objects on this server that > > > > houses > > > > > > the > > > > > > >>> backups for* 263-nodes*? > > > > > > >>> > > > > > > >>> I vaguely remember some bug/APAR about systemstate backups > > being > > > > > > >>> large/slow/causing performance problems with expiration but > > these > > > > nodes > > > > > > >>> client levels are fairly current (8.1.0.2 - staying below the > > > > > > >> 8.1.2/SSL/TLS > > > > > > >>> enforcement levels) and the ISP server is 7.1.7.400. All of > > these > > > > are > > > > > > >>> Windows 2016, if that matters. > > > > > > >>> > > > > > > >>> -- > > > > > > >>> *Zoltan Forray* > > > > > > >>> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > > > > >>> Xymon Monitor Administrator > > > > > > >>> VMware Administrator > > > > > > >>> Virginia Commonwealth University > > > > > > >>> UCC/Office of Technology Services > > > > > > >>> www.ucc.vcu.edu > > > > > > >>> zfor...@vcu.edu - 804-828-4807 > > > > > > >>> Don't be a phishing victim - VCU and other reputable > > organizations > > > > will > > > > > > >>> never use email to request that you reply with your password, > > > > social > > > > > > >>> security number or confidential personal information. For more > > > > details > > > > > > >>> visit https://urldefense.proofpoint.com/v2/url? > > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > > > > > > > > > > > > > > > > > > > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e= > > > > > > > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *Zoltan Forray* > > > > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > > > > > Xymon Monitor Administrator > > > > > > > VMware Administrator > > > > > > > Virginia Commonwealth University > > > > > > > UCC/Office of Technology Services > > > > > > > www.ucc.vcu.edu > > > > > > > zfor...@vcu.edu - 804-828-4807 > > > > > > > Don't be a phishing victim - VCU and other reputable > > organizations > > > > will > > > > > > > never use email to request that you reply with your password, > > social > > > > > > > security number or confidential personal information. For more > > > > details > > > > > > > visit https://urldefense.proofpoint.com/v2/url? > > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > > > > > > > > > > > > > > > > > > > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e= > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *Zoltan Forray* > > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > > > Xymon Monitor Administrator > > > > > VMware Administrator > > > > > Virginia Commonwealth University > > > > > UCC/Office of Technology Services > > > > > www.ucc.vcu.edu > > > > > zfor...@vcu.edu - 804-828-4807 > > > > > Don't be a phishing victim - VCU and other reputable organizations > > will > > > > > never use email to request that you reply with your password, social > > > > > security number or confidential personal information. For more > > details > > > > > visit https://urldefense.proofpoint.com/v2/url? > > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > > > > > > > > > > > > > > > > > > > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e= > > > > > > > > > > > > > > > > > > > > > > > > -- > > > *Zoltan Forray* > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > Xymon Monitor Administrator > > > VMware Administrator > > > Virginia Commonwealth University > > > UCC/Office of Technology Services > > > www.ucc.vcu.edu > > > zfor...@vcu.edu - 804-828-4807 > > > Don't be a phishing victim - VCU and other reputable organizations will > > > never use email to request that you reply with your password, social > > > security number or confidential personal information. For more details > > > visit https://urldefense.proofpoint.com/v2/url? > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=hQoTgXdTWRp- > > > > > > > > c4tE7WF3GSnI5KTnZ9r6rwUQJqLtLu0&s=1kN_Gmc2Qiaab36WLtyjbl66sf5H54wTfT4PyG9zCj8&e= > > > > > > > > > > -- > *Zoltan Forray* > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > Xymon Monitor Administrator > VMware Administrator > Virginia Commonwealth University > UCC/Office of Technology Services > www.ucc.vcu.edu > zfor...@vcu.edu - 804-828-4807 > Don't be a phishing victim - VCU and other reputable organizations will > never use email to request that you reply with your password, social > security number or confidential personal information. For more details > visit https://urldefense.proofpoint.com/v2/url? > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=N9AQefDr2d0WLgIG8G7ONWkk-3POpjbBbVM3mI2lzuM&s=qJtGBK9WeZFU_1uZjZJN_jNGKZUQNnnB9MIn6JPDPC8&e= >
