Appreciate the input Del, thanks !

Thank you,
-Rick Adamson


-----Original Message-----
From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of Del Hoobler
Sent: Friday, July 19, 2019 2:32 PM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] deletion performance of large deduplicated files

* This email originated outside of the organization. Use caution when opening 
attachments or clicking links. *

----------------------------------------------------------------------
Hi Eric and all,

We are aware of this situation and putting additional focus and attention on it.


Del

----------------------------------------------------

"ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 07/19/2019
09:35:37 AM:

> From: "Loon, Eric van (ITOP NS) - KLM" <eric-van.l...@klm.com>
> To: ADSM-L@VM.MARIST.EDU
> Date: 07/19/2019 09:36 AM
> Subject: [EXTERNAL] Re: deletion performance of large deduplicated files
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU>
>
> Hi Rick and others!
>
> I replicated the data of the test TDP client to multiple servers,
> running 7.1.7, 7.1.9 and even 8.1.8: the performance sucks on all
servers.
> We do not use client replication as part of our server protection.
> We need a real time replication over the datacenter and thus we rely
> on host-based replication for all our servers. I tested with and
> without replication: there is no noticeable difference.
> As a work-around I will install a non-deduplicated file stgpool on
> the worst performing server next week. I expect the server to
> perform better as soon as all container pool objects are expired, in
> about 3 weeks from now.
> In the meantime I will keep on pursuing IBM until it is fixed or
> else we might need to replace the product altogether...
>
> Kind regards,
> Eric van Loon
> Air France/KLM Storage & Backup
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On
> Behalf Of Rick Adamson
> Sent: vrijdag 19 juli 2019 14:45
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: deletion performance of large deduplicated files
>
> Eric, Michael,
> I have been working through similar struggles and as I read your
> posts had to wonder, can you provide some details on your server and
> client versions?
> Basically I now have experience/exposure to every version/
> maintenance pack/patch SP has put out since 7.1.3.x
>
> My servers are now running on  8.1.8 (windows) and has seemed to
> have stabilized (for now).
> One thing that was causing me a lot of grief was using directory
> storage pools with client side deduplication enabled, particularly
> on data protection products (all of them).
> Afterwards there was a lot of cleanup; auditing containers,
> confirming that protect storage was completing successfully, and
> finally doing the same for replicate node processes.
>
> I found that understandably server performance takes a severe nose
> dive if you are trying to process (protect/replicate) damaged
> containers, and most likely restores will be compromised as well.
>
>
> Thank you,
> -Rick Adamson
>
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of
Michael Prix
> Sent: Friday, July 19, 2019 6:11 AM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: [ADSM-L] deletion performance of large deduplicated files
>
> * This email originated outside of the organization. Use caution
> when opening attachments or clicking links. *
>
> ----------------------------------------------------------------------
> Hello Eric,
>
>
>
>   welcome to my nightmares. Take a seat, wanna have a drink?
>
>
>
> I had the pleasure of performance and data corruption PMRs during the
last two
>
> years with TDP Oracle. Yes, at first the customer got blamed for not
adhering
>
> completely to to blueprints, but after some weeks it boild down to  ...
>
> silence.
>
> Data corruption was because of what ended in IT28096 - now fixed.
>
> Performance is interesting, but resembles to what you have written. We
work
>
> with MAXPIECESIZE settings on RMAN to keep the backup pieces small and
got
>
> some interesting values, pending further observation, but we might be on
a
>
> cheerful way. I'm talking about database sizes of 50TB here, warehouse
style.
>
>   In between we moved the big DBs to a dedicated server to prove that
the
>
> performance drop is because of the big DBs, and the remaining "small"
DBs  -
>
> size of 500MB up to 5TB - didn't put any measurable stress on the DB in
terms
>
> of expiration and protect stgpool. Even the big DBs on their dedicated
server
>
> performed better in terms of expiration and protect stgpool, which might
have
>
> been a coincidence of these DBs holding nearly the same data and having
the
>
> same retention period.
>
>
>
> What I can't observe is a slowness of the DB. Queries are answered in
the
>
> normal time - depending on the query. a count(*) from backupobjects
naturally
>
> takes some time, considerably longer when you use dedup, but the daily
queries
>
> are answered in the "normal" timeframe.
>
>
>
> What helped immediately was some tuning:
>
> - More LUNS and filesystems for the TSM-DB
>
> - smaller disks, but more of them, for each filesystem.
>
>   changing the disks from 100GB to 2 x 50GB for each DB-filesystem got
me a
>
> performance boost of 200% in expiration and backup db. Unbelievable,but
true.
>
> Yes, I'm using SSD. And SVC. And multiple storage systems. Performance
isn't
>
> the problem, we are measuring 2ms respone time for write AND read.
>
> - stripeset for each fileset
>
>
>
>
>
> --
>
> Michael Prix
>
>
>
> On Fri, 2019-07-19 at 07:29 +0000, Loon, Eric van (ITOP NS) - KLM wrote:
>
> > Hi TSM/SP-ers,
>
> >
>
> > We are struggling with the performance of our TSM servers for months
now. We
>
> > are running several servers with hardware (Data Domain) dedup for
years
>
> > without any problems, but on our new servers with directory container
pools
>
> > performance is really, really bad.
>
> > The servers and storage are designed according to the Blueprints and
they
>
> > are working fine as long as you do not add large database (Oracle and
SAP)
>
> > client to them. As soon as you do, the overall server performance
becomes
>
> > very bad: client and admin session initiation takes 20 to 40 seconds,
SQL
>
> > queries run for minutes where they should take a few seconds and q
query
>
> > stgpool sometimes takes more than a minute to respond!
>
> > I have two cases open for this. In one case we focused a lot on the OS
and
>
> > disk performance, but during that process I noticed that the
performance is
>
> > most likely caused by the way TSM processes large (few hundred MB)
files. I
>
> > performed a large amount of tests and came to the conclusion that it
takes
>
> > TSM a huge amount of time to delete large deduplicated files, both in
>
> > container pools as deduplicated file pools. As test I use an TDP for
Oracle
>
> > client which uses a backup piece size of 900 MB. The client contains
about
>
> > 5000 files. Deleting the files from a container pool takes more than
an
>
> > hour. When you run a delete object for the files individually I see
that
>
> > most files take more than a second(!) to delete. If I put that same
data in
>
> > a non-deduplicated file pool, a delete filespace takes about 15
seconds...
>
> > The main issue is that the TDP clients are doing the exact same thing:
as
>
> > soon as a backup file is no longer needed, it's removed from the RMAN
>
> > catalog and deleted from TSM. Since we have several huge database
clients
>
> > (multiple TB's each) these Oracle delete jobs tend to run for hours.
These
>
> > delete jobs also seem to slow down each other, because when there are
>
> > several of those jobs running at the same time, they become even more
slow.
>
> > At this point I have one server where these jobs are running 24 hours
per
>
> > day! This server is at the moment the worst performing TSM server I
have
>
> > ever seen. On the other container pool servers I was able to move the
Oracle
>
> > and SAP server away to the old servers (the ones with the Data
Domain), but
>
> > on this one I can't because of Data Domain capacity reasons.
>
> > For this file deletion performance I also have a case open, but there
is
>
> > absolutely no progress. I proved IBM how bad the performance is and I
even
>
> > offered them a copy of our database so they can see for themselves,
but only
>
> > silence from development...
>
> > One thing I do not understand: I find it very hard to believe that we
are
>
> > the only one suffering from this issue. There must be dozens of TSM
users
>
> > out there that backup large databases to TSM container pools?
>
> >
>
> > Kind regards,
>
> > Eric van Loon
>
> > Air France/KLM Storage & Backup
>
> > ********************************************************
>
> > For information, services and offers, please visit our web site:
>
> > https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__www.klm.com&d=DwICaQ&c=AzgFQeXLLKhxSQaoFCm29A&r=eqh5PzQPIsPArLoI_uV1mKvhIpcNP1MsClDPSJjFfxw&m=OAsoTBlnuy-
> Km69JdbqTRXQFPP3_U4a9gj4CSO0lbFw&s=MSJs-
> FT5ZG_GgtiqMOCzRHNo0Bw0PiD3C7-s_xuNgBg&e= . This e-mail and any
> attachment may contain confidential
>
> > and privileged material intended for the addressee only. If you are
not the
>
> > addressee, you are notified that no part of the e-mail or any
attachment may
>
> > be disclosed, copied or distributed, and that any other action related
to
>
> > this e-mail or attachment is strictly prohibited, and may be unlawful.
If
>
> > you have received this e-mail by error, please notify the sender
immediately
>
> > by return e-mail, and delete this message.
>
> >
>
> > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or
its
>
> > employees shall not be liable for the incorrect or incomplete
transmission
>
> > of this e-mail or any attachments, nor responsible for any delay in
receipt.
>
> > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
Dutch
>
> > Airlines) is registered in Amstelveen, The Netherlands, with
registered
>
> > number 33014286
>
> > ********************************************************
>
>
> **CONFIDENTIALITY NOTICE** This electronic message contains
> information from Southeastern Grocers, Inc and is intended only for
> the use of the addressee. This message may contain information that
> is privileged, confidential and/or exempt from disclosure under
> applicable Law. This message may not be read, used, distributed,
> forwarded, reproduced or stored by any other than the intended
> recipient. If you are not the intended recipient, please delete and
> notify the sender.
> ********************************************************
> For information, services and offers, please visit our web site:
> https://urldefense.proofpoint.com/v2/url?
> u=http-3A__www.klm.com&d=DwIGaQ&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=0hq2JX5c3TEZNriHEs7Zf7HrkY2fNtONOrEOM8Txvk8&m=sawlFLhDPFJ01kK39aEsF5p-
> P9m243WQvtv2i8G_tG0&s=THD1AsuKVK6V1PFwHzYqwMwt5qW9rRgIaXYIEhTqyUk&e=
> . This e-mail and any attachment may contain confidential and
> privileged material intended for the addressee only. If you are not
> the addressee, you are notified that no part of the e-mail or any
> attachment may be disclosed, copied or distributed, and that any
> other action related to this e-mail or attachment is strictly
> prohibited, and may be unlawful. If you have received this e-mail by
> error, please notify the sender immediately by return e-mail, and
> delete this message.
>
> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/
> or its employees shall not be liable for the incorrect or incomplete
> transmission of this e-mail or any attachments, nor responsible for
> any delay in receipt.
> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
> Dutch Airlines) is registered in Amstelveen, The Netherlands, with
> registered number 33014286
> ********************************************************
>

**CONFIDENTIALITY NOTICE** This electronic message contains information from 
Southeastern Grocers, Inc and is intended only for the use of the addressee. 
This message may contain information that is privileged, confidential and/or 
exempt from disclosure under applicable Law. This message may not be read, 
used, distributed, forwarded, reproduced or stored by any other than the 
intended recipient. If you are not the intended recipient, please delete and 
notify the sender.

Reply via email to