Hi Andy,

We do not have any policy/managementclass with versions higher than 5
across our complex and retain-extra copies is 30-days. I do not know what
these servers/applications do so I will ask about the Crypto Keys you
referred to in the link.

For the first 3-with this issue, the client is 8.1.0.2 and the OS is
Windows 2016.  The current one that just successfully finished, is Windows
2012R2 and the client is 7.1.4.4

2/26/2019 12:40:13 PM ANR0806I The ORIONADDWEB\SystemState\NULL\System
State\SystemState (fsId=1) file space was deleted for node ORIONADDWEB:
300,346,169 objects were deleted.
2/26/2019 12:40:13 PM ANR0987I Process 1382 for DELETE FILESPACE running in
the BACKGROUND processed 300,346,169 items with a completion state of
SUCCESS at 12:40:09 PM.


On Tue, Feb 26, 2019 at 12:19 PM Andrew Raibeck <stor...@us.ibm.com> wrote:

> Hi Zoltan,
>
> What policy was system state bound to for the nodes that exceed 200 million
> objects? Is it possible that the number of versions retained was very
> large?
>
> Another thought is whether the OS of each of these nodes might be affected
> by this issue:
>
>
> https://social.technet.microsoft.com/Forums/en-US/e2632b6e-76b9-4640-85b9-698fb55199d8/cprogramdatamicrosoftcryptorsamachinekeys-is-filling-my-disk-space?forum=winservergen
>
> We have seen huge system state backups that can occur due to that issue, so
> if it occurred for those nodes, that is another explanation.
>
> Regards,
>
> Andy
>
>
> ____________________________________________________________________________
>
> Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com
>
> "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2019-02-26
> 11:59:43:
>
> > From: Zoltan Forray <zfor...@vcu.edu>
> > To: ADSM-L@VM.MARIST.EDU
> > Date: 2019-02-26 12:01
> > Subject: Re: Bottomless pit
> > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU>
> >
> > Hi Andy,
> >
> > Thank you for clarifying things - a bit.    However, why would certain
> > nodes have enormously large numbers vs the average when all things are
> > equal as far as policies are concerned?
> >
> > I do see average systemstate object delete counts in the *1-2M range* but
> > these 4-nodes are exceeding *200M* each.  On this server, I deleted the
> > systemstate for 60-nodes.  Only 9-exceeded 1M and of those 1-exceeded 2M.
> >
> > This last node deletion is still running after 3-hours of deletion.
> >
> > 2019-02-26 08:57:56 Deleting file space ORIONADDWEB\SystemState\NULL
> \System
> > State\SystemState (fsId=1) (backup data) for node ORIONADDWEB:
> 243,029,858
> > objects deleted.
> >
> > Even the 3-nodes whose deletion failed  (after deleting >110M objects
> each)
> > due to some kind of bitfile error - after restarting the deletions are up
> > to 50M+ each and still running?
> >
> >
> > On Tue, Feb 26, 2019 at 11:30 AM Andrew Raibeck <stor...@us.ibm.com>
> wrote:
> >
> > > Hi Zoltan,
> > >
> > > The large number of objects is normal for system state file spaces.
> System
> > > state backup uses grouping, with each backed up object being a member
> of
> > > the group. If the same object is included in multiple groups, then it
> will
> > > be counted more than once. Each system state backup creates a new
> group, so
> > > as the number of retained backup versions grows, so does the number of
> > > groups, and thus the total object count can grow very large.
> > >
> > > Best regards,
> > >
> > > Andy
> > >
> > >
> > >
>
> ____________________________________________________________________________
>
> > >
> > > Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com
> > >
> > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2019-02-26
> > > 10:30:04:
> > >
> > > > From: Zoltan Forray <zfor...@vcu.edu>
> > > > To: ADSM-L@VM.MARIST.EDU
> > > > Date: 2019-02-26 10:30
> > > > Subject: Re: Bottomless pit
> > > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU>
> > > >
> > > > Just found another node with a similar issue on a different ISP
> server
> > > with
> > > > different software levels (client=7.1.4.4 and OS=Windows 2012R2).
> The
> > > node
> > > > name is the same so I think the application is, as well.
> > > >
> > > > 2019-02-26 08:57:56 Deleting file space ORIONADDWEB\SystemState\NULL
> > > \System
> > > > State\SystemState (fsId=1) (backup data) for node ORIONADDWEB:
> > > *129,785,134
> > > > objects deleted*.
> > > >
> > > >
> > > > On Tue, Feb 26, 2019 at 9:15 AM Sasa Drnjevic <sasa.drnje...@srce.hr
> >
> > > wrote:
> > > >
> > > > > On 26.2.2019. 15:01, Zoltan Forray wrote:
> > > > > > Since all of these systemstate deletes crashed/failed, I
> restarted
> > > them
> > > > > and
> > > > > > 2-of the 3 are already up to 5M objects after running for
> 30-minutes.
> > > > > Will
> > > > > > this ever end successfully?
> > > > >
> > > > > All of mine did finish successfully...
> > > > >
> > > > > But, none of them had more than 25 mil files deleted.
> > > > >
> > > > > Wish you luck ;-)
> > > > >
> > > > > Rgds,
> > > > >
> > > > > --
> > > > > Sasa Drnjevic
> > > > > www.srce.unizg.hr/en/
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > On Mon, Feb 25, 2019 at 4:25 PM Sasa Drnjevic
> <sasa.drnje...@srce.hr
> > > >
> > > > > wrote:
> > > > > >
> > > > > >> FYI,
> > > > > >> same here...but my range/ratio was:
> > > > > >>
> > > > > >> ~2 mil occ to 25 mil deleted objects...
> > > > > >>
> > > > > >> Never solved the mystery... gave up :->
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Sasa Drnjevic
> > > > > >> www.srce.unizg.hr/en/
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On 2019-02-25 20:05, Zoltan Forray wrote:
> > > > > >>> Here is a new one.......
> > > > > >>>
> > > > > >>> We turned off backing up SystemState last week.  Now I am going
> > > through
> > > > > >> and
> > > > > >>> deleted the Systemstate filesystems.
> > > > > >>>
> > > > > >>> Since I wanted to see how many objects would be deleted, I did
> a "Q
> > > > > >>> OCCUPANCY" and preserved the file count numbers for all Windows
> > > nodes
> > > > > on
> > > > > >>> this server.
> > > > > >>>
> > > > > >>> For 4-nodes, the delete of their systemstate filespaces has
> been
> > > > > running
> > > > > >>> for 5-hours. A "Q PROC" shows:
> > > > > >>>
> > > > > >>> 2019-02-25 08:52:05 Deleting file space
> > > > > >>> ORION-POLL-WEST\SystemState\NULL\System State\SystemState
> (fsId=1)
> > > > > >> (backup
> > > > > >>> data) for node ORION-POLL-WEST: *105,511,859 objects deleted*.
> > > > > >>>
> > > > > >>> Considering the occupancy for this node was *~5-Million
> objects*,
> > > how
> > > > > has
> > > > > >>> it deleted *105-Million* objects (and counting).  The other
> 3-nodes
> > > in
> > > > > >>> question are also up to *>100-Million objects deleted* and none
> of
> > > them
> > > > > >> had
> > > > > >>> more than *6M objects* in occupancy?
> > > > > >>>
> > > > > >>> At this rate, the deleting objects count for 4-nodes
> systemstate
> > > will
> > > > > >>> exceed 50% of the total occupancy objects on this server that
> > > houses
> > > > > the
> > > > > >>> backups for* 263-nodes*?
> > > > > >>>
> > > > > >>> I vaguely remember some bug/APAR about systemstate backups
> being
> > > > > >>> large/slow/causing performance problems with expiration but
> these
> > > nodes
> > > > > >>> client levels are fairly current (8.1.0.2 - staying below the
> > > > > >> 8.1.2/SSL/TLS
> > > > > >>> enforcement levels) and the ISP server is 7.1.7.400.  All of
> these
> > > are
> > > > > >>> Windows 2016, if that matters.
> > > > > >>>
> > > > > >>> --
> > > > > >>> *Zoltan Forray*
> > > > > >>> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > > > >>> Xymon Monitor Administrator
> > > > > >>> VMware Administrator
> > > > > >>> Virginia Commonwealth University
> > > > > >>> UCC/Office of Technology Services
> > > > > >>> www.ucc.vcu.edu
> > > > > >>> zfor...@vcu.edu - 804-828-4807
> > > > > >>> Don't be a phishing victim - VCU and other reputable
> organizations
> > > will
> > > > > >>> never use email to request that you reply with your password,
> > > social
> > > > > >>> security number or confidential personal information. For more
> > > details
> > > > > >>> visit https://urldefense.proofpoint.com/v2/url?
> > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > >
> > >
> > >
> >
>
> siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e=
>
> > >
> > > > > >>>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Zoltan Forray*
> > > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > > > > Xymon Monitor Administrator
> > > > > > VMware Administrator
> > > > > > Virginia Commonwealth University
> > > > > > UCC/Office of Technology Services
> > > > > > www.ucc.vcu.edu
> > > > > > zfor...@vcu.edu - 804-828-4807
> > > > > > Don't be a phishing victim - VCU and other reputable
> organizations
> > > will
> > > > > > never use email to request that you reply with your password,
> social
> > > > > > security number or confidential personal information. For more
> > > details
> > > > > > visit https://urldefense.proofpoint.com/v2/url?
> > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > >
> > >
> > >
> >
>
> siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e=
>
> > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *Zoltan Forray*
> > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > > Xymon Monitor Administrator
> > > > VMware Administrator
> > > > Virginia Commonwealth University
> > > > UCC/Office of Technology Services
> > > > www.ucc.vcu.edu
> > > > zfor...@vcu.edu - 804-828-4807
> > > > Don't be a phishing victim - VCU and other reputable organizations
> will
> > > > never use email to request that you reply with your password, social
> > > > security number or confidential personal information. For more
> details
> > > > visit https://urldefense.proofpoint.com/v2/url?
> > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > >
> > >
> > >
> >
>
> siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e=
>
> > >
> > > >
> > >
> >
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator
> > VMware Administrator
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > www.ucc.vcu.edu
> > zfor...@vcu.edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit https://urldefense.proofpoint.com/v2/url?
> > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=hQoTgXdTWRp-
> >
>
> c4tE7WF3GSnI5KTnZ9r6rwUQJqLtLu0&s=1kN_Gmc2Qiaab36WLtyjbl66sf5H54wTfT4PyG9zCj8&e=
>
> >
>


--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zfor...@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/

Reply via email to