Hi Zoltan,

There is nothing in the client code you are using that would cause
excessive backups. If you happen to have backup logs going far back enough,
those might show an excessive number of system state objects. Normally I
would expect the number of backup objects N to be 90000 < N < 150000. that
is a roughly estimated range, depends on the individual system, and maybe N
could be a little higher... but my radar would certainly be alerted if it
was more than 200,000 and growing daily.

Regards,

Andy

____________________________________________________________________________

Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com

"ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2019-02-26
12:52:27:

> From: Zoltan Forray <zfor...@vcu.edu>
> To: ADSM-L@VM.MARIST.EDU
> Date: 2019-02-26 12:52
> Subject: Re: Bottomless pit
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU>
>
> Hi Andy,
>
> We do not have any policy/managementclass with versions higher than 5
> across our complex and retain-extra copies is 30-days. I do not know what
> these servers/applications do so I will ask about the Crypto Keys you
> referred to in the link.
>
> For the first 3-with this issue, the client is 8.1.0.2 and the OS is
> Windows 2016.  The current one that just successfully finished, is
Windows
> 2012R2 and the client is 7.1.4.4
>
> 2/26/2019 12:40:13 PM ANR0806I The ORIONADDWEB\SystemState\NULL\System
> State\SystemState (fsId=1) file space was deleted for node ORIONADDWEB:
> 300,346,169 objects were deleted.
> 2/26/2019 12:40:13 PM ANR0987I Process 1382 for DELETE FILESPACE running
in
> the BACKGROUND processed 300,346,169 items with a completion state of
> SUCCESS at 12:40:09 PM.
>
>
> On Tue, Feb 26, 2019 at 12:19 PM Andrew Raibeck <stor...@us.ibm.com>
wrote:
>
> > Hi Zoltan,
> >
> > What policy was system state bound to for the nodes that exceed 200
million
> > objects? Is it possible that the number of versions retained was very
> > large?
> >
> > Another thought is whether the OS of each of these nodes might be
affected
> > by this issue:
> >
> >
> > https://urldefense.proofpoint.com/v2/url?
>
u=https-3A__social.technet.microsoft.com_Forums_en-2DUS_e2632b6e-2D76b9-2D4640-2D85b9-2D698fb55199d8_cprogramdatamicrosoftcryptorsamachinekeys-2Dis-2Dfilling-2Dmy-2Ddisk-2Dspace-3Fforum-3Dwinservergen&d=DwIBaQ&c=jf_iaSHvJObTbx-

>
siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=N9AQefDr2d0WLgIG8G7ONWkk-3POpjbBbVM3mI2lzuM&s=rNhhFizJITYew56twqREPr482w-

> Hss5ZsFp5-UY_W58&e=
> >
> > We have seen huge system state backups that can occur due to that
issue, so
> > if it occurred for those nodes, that is another explanation.
> >
> > Regards,
> >
> > Andy
> >
> >
> >
____________________________________________________________________________

> >
> > Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com
> >
> > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2019-02-26
> > 11:59:43:
> >
> > > From: Zoltan Forray <zfor...@vcu.edu>
> > > To: ADSM-L@VM.MARIST.EDU
> > > Date: 2019-02-26 12:01
> > > Subject: Re: Bottomless pit
> > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU>
> > >
> > > Hi Andy,
> > >
> > > Thank you for clarifying things - a bit.    However, why would
certain
> > > nodes have enormously large numbers vs the average when all things
are
> > > equal as far as policies are concerned?
> > >
> > > I do see average systemstate object delete counts in the *1-2M range*
but
> > > these 4-nodes are exceeding *200M* each.  On this server, I deleted
the
> > > systemstate for 60-nodes.  Only 9-exceeded 1M and of those 1-exceeded
2M.
> > >
> > > This last node deletion is still running after 3-hours of deletion.
> > >
> > > 2019-02-26 08:57:56 Deleting file space ORIONADDWEB\SystemState\NULL
> > \System
> > > State\SystemState (fsId=1) (backup data) for node ORIONADDWEB:
> > 243,029,858
> > > objects deleted.
> > >
> > > Even the 3-nodes whose deletion failed  (after deleting >110M objects
> > each)
> > > due to some kind of bitfile error - after restarting the deletions
are up
> > > to 50M+ each and still running?
> > >
> > >
> > > On Tue, Feb 26, 2019 at 11:30 AM Andrew Raibeck <stor...@us.ibm.com>
> > wrote:
> > >
> > > > Hi Zoltan,
> > > >
> > > > The large number of objects is normal for system state file spaces.
> > System
> > > > state backup uses grouping, with each backed up object being a
member
> > of
> > > > the group. If the same object is included in multiple groups, then
it
> > will
> > > > be counted more than once. Each system state backup creates a new
> > group, so
> > > > as the number of retained backup versions grows, so does the number
of
> > > > groups, and thus the total object count can grow very large.
> > > >
> > > > Best regards,
> > > >
> > > > Andy
> > > >
> > > >
> > > >
> >
> >
____________________________________________________________________________

> >
> > > >
> > > > Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com
> > > >
> > > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on
2019-02-26
> > > > 10:30:04:
> > > >
> > > > > From: Zoltan Forray <zfor...@vcu.edu>
> > > > > To: ADSM-L@VM.MARIST.EDU
> > > > > Date: 2019-02-26 10:30
> > > > > Subject: Re: Bottomless pit
> > > > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU>
> > > > >
> > > > > Just found another node with a similar issue on a different ISP
> > server
> > > > with
> > > > > different software levels (client=7.1.4.4 and OS=Windows 2012R2).
> > The
> > > > node
> > > > > name is the same so I think the application is, as well.
> > > > >
> > > > > 2019-02-26 08:57:56 Deleting file space ORIONADDWEB\SystemState
\NULL
> > > > \System
> > > > > State\SystemState (fsId=1) (backup data) for node ORIONADDWEB:
> > > > *129,785,134
> > > > > objects deleted*.
> > > > >
> > > > >
> > > > > On Tue, Feb 26, 2019 at 9:15 AM Sasa Drnjevic
<sasa.drnje...@srce.hr
> > >
> > > > wrote:
> > > > >
> > > > > > On 26.2.2019. 15:01, Zoltan Forray wrote:
> > > > > > > Since all of these systemstate deletes crashed/failed, I
> > restarted
> > > > them
> > > > > > and
> > > > > > > 2-of the 3 are already up to 5M objects after running for
> > 30-minutes.
> > > > > > Will
> > > > > > > this ever end successfully?
> > > > > >
> > > > > > All of mine did finish successfully...
> > > > > >
> > > > > > But, none of them had more than 25 mil files deleted.
> > > > > >
> > > > > > Wish you luck ;-)
> > > > > >
> > > > > > Rgds,
> > > > > >
> > > > > > --
> > > > > > Sasa Drnjevic
> > > > > > www.srce.unizg.hr/en/
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On Mon, Feb 25, 2019 at 4:25 PM Sasa Drnjevic
> > <sasa.drnje...@srce.hr
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> FYI,
> > > > > > >> same here...but my range/ratio was:
> > > > > > >>
> > > > > > >> ~2 mil occ to 25 mil deleted objects...
> > > > > > >>
> > > > > > >> Never solved the mystery... gave up :->
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Sasa Drnjevic
> > > > > > >> www.srce.unizg.hr/en/
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On 2019-02-25 20:05, Zoltan Forray wrote:
> > > > > > >>> Here is a new one.......
> > > > > > >>>
> > > > > > >>> We turned off backing up SystemState last week.  Now I am
going
> > > > through
> > > > > > >> and
> > > > > > >>> deleted the Systemstate filesystems.
> > > > > > >>>
> > > > > > >>> Since I wanted to see how many objects would be deleted, I
did
> > a "Q
> > > > > > >>> OCCUPANCY" and preserved the file count numbers for all
Windows
> > > > nodes
> > > > > > on
> > > > > > >>> this server.
> > > > > > >>>
> > > > > > >>> For 4-nodes, the delete of their systemstate filespaces has
> > been
> > > > > > running
> > > > > > >>> for 5-hours. A "Q PROC" shows:
> > > > > > >>>
> > > > > > >>> 2019-02-25 08:52:05 Deleting file space
> > > > > > >>> ORION-POLL-WEST\SystemState\NULL\System State\SystemState
> > (fsId=1)
> > > > > > >> (backup
> > > > > > >>> data) for node ORION-POLL-WEST: *105,511,859 objects
deleted*.
> > > > > > >>>
> > > > > > >>> Considering the occupancy for this node was *~5-Million
> > objects*,
> > > > how
> > > > > > has
> > > > > > >>> it deleted *105-Million* objects (and counting).  The other
> > 3-nodes
> > > > in
> > > > > > >>> question are also up to *>100-Million objects deleted* and
none
> > of
> > > > them
> > > > > > >> had
> > > > > > >>> more than *6M objects* in occupancy?
> > > > > > >>>
> > > > > > >>> At this rate, the deleting objects count for 4-nodes
> > systemstate
> > > > will
> > > > > > >>> exceed 50% of the total occupancy objects on this server
that
> > > > houses
> > > > > > the
> > > > > > >>> backups for* 263-nodes*?
> > > > > > >>>
> > > > > > >>> I vaguely remember some bug/APAR about systemstate backups
> > being
> > > > > > >>> large/slow/causing performance problems with expiration but
> > these
> > > > nodes
> > > > > > >>> client levels are fairly current (8.1.0.2 - staying below
the
> > > > > > >> 8.1.2/SSL/TLS
> > > > > > >>> enforcement levels) and the ISP server is 7.1.7.400.  All
of
> > these
> > > > are
> > > > > > >>> Windows 2016, if that matters.
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> *Zoltan Forray*
> > > > > > >>> Spectrum Protect (p.k.a. TSM) Software & Hardware
Administrator
> > > > > > >>> Xymon Monitor Administrator
> > > > > > >>> VMware Administrator
> > > > > > >>> Virginia Commonwealth University
> > > > > > >>> UCC/Office of Technology Services
> > > > > > >>> www.ucc.vcu.edu
> > > > > > >>> zfor...@vcu.edu - 804-828-4807
> > > > > > >>> Don't be a phishing victim - VCU and other reputable
> > organizations
> > > > will
> > > > > > >>> never use email to request that you reply with your
password,
> > > > social
> > > > > > >>> security number or confidential personal information. For
more
> > > > details
> > > > > > >>> visit https://urldefense.proofpoint.com/v2/url?
> > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > > >
> > > >
> > > >
> > >
> >
> >
>
siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e=

> >
> > > >
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Zoltan Forray*
> > > > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware
Administrator
> > > > > > > Xymon Monitor Administrator
> > > > > > > VMware Administrator
> > > > > > > Virginia Commonwealth University
> > > > > > > UCC/Office of Technology Services
> > > > > > > www.ucc.vcu.edu
> > > > > > > zfor...@vcu.edu - 804-828-4807
> > > > > > > Don't be a phishing victim - VCU and other reputable
> > organizations
> > > > will
> > > > > > > never use email to request that you reply with your password,
> > social
> > > > > > > security number or confidential personal information. For
more
> > > > details
> > > > > > > visit https://urldefense.proofpoint.com/v2/url?
> > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > > >
> > > >
> > > >
> > >
> >
> >
>
siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e=

> >
> > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Zoltan Forray*
> > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > > > Xymon Monitor Administrator
> > > > > VMware Administrator
> > > > > Virginia Commonwealth University
> > > > > UCC/Office of Technology Services
> > > > > www.ucc.vcu.edu
> > > > > zfor...@vcu.edu - 804-828-4807
> > > > > Don't be a phishing victim - VCU and other reputable
organizations
> > will
> > > > > never use email to request that you reply with your password,
social
> > > > > security number or confidential personal information. For more
> > details
> > > > > visit https://urldefense.proofpoint.com/v2/url?
> > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > > >
> > > >
> > > >
> > >
> >
> >
>
siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=zUs0X2c-0aFkUJkwVOlc7YfGUwuUiaieLUaugGIXmQQ&s=qfYWZxPRiZPFOeph0XdxtVb3FtsGj1zoGEzhMmGCNqA&e=

> >
> > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Zoltan Forray*
> > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > Xymon Monitor Administrator
> > > VMware Administrator
> > > Virginia Commonwealth University
> > > UCC/Office of Technology Services
> > > www.ucc.vcu.edu
> > > zfor...@vcu.edu - 804-828-4807
> > > Don't be a phishing victim - VCU and other reputable organizations
will
> > > never use email to request that you reply with your password, social
> > > security number or confidential personal information. For more
details
> > > visit https://urldefense.proofpoint.com/v2/url?
> > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=hQoTgXdTWRp-
> > >
> >
> >
>
c4tE7WF3GSnI5KTnZ9r6rwUQJqLtLu0&s=1kN_Gmc2Qiaab36WLtyjbl66sf5H54wTfT4PyG9zCj8&e=

> >
> > >
> >
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zfor...@vcu.edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit https://urldefense.proofpoint.com/v2/url?
> u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=N9AQefDr2d0WLgIG8G7ONWkk-3POpjbBbVM3mI2lzuM&s=qJtGBK9WeZFU_1uZjZJN_jNGKZUQNnnB9MIn6JPDPC8&e=

>

Reply via email to