Hi, but for example in VE backup (with legacy dedup stgpool - no container), i do client side dedup that combine also compression for best overall space saving. With deduplication server side i completely lose the benefits of compression.
On Sun, May 28, 2017 at 12:06 AM, Del Hoobler <hoob...@us.ibm.com> wrote: > Hi Robert, > > These recommendations are for directory and cloud container pools which > always use deduplication. They also happened to be our recommendations > for legacy FILE deduplication. > > > Del > > ---------------------------------------------------- > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 05/23/2017 > 12:52:07 AM: > > > From: "rou...@univ.haifa.ac.il" <rou...@univ.haifa.ac.il> > > To: ADSM-L@VM.MARIST.EDU > > Date: 05/23/2017 12:52 AM > > Subject: Re: Client-side dedup speed > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > > > Hello Del > > > > Did it's working too with Storage type DIRECTORY when on it > > deduplicate data is YES ? > > > > TSM server Version 8.1.1.0 > > > > Best Regards Robert > > > > -----Original Message----- > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On > > Behalf Of Del Hoobler > > Sent: Monday, May 22, 2017 8:55 PM > > To: ADSM-L@VM.MARIST.EDU > > Subject: Re: [ADSM-L] Client-side dedup speed > > > > In the deduplication best practices paper it is called out that in > > some cases the client dedup cache will slow backups down. This is > > mentioned in section 1.2.3.2. Then in section 4.3.1, the > > enablededupcache option is recommended to be set to 'no' in all > > cases except for backups across high latency networks and is > > recommend to always be set to 'no' for applications which use the > > Spectrum Protect API. > > > > > > Del > > > > > > > ----- Forwarded by Del Hoobler/Endicott/IBM on 05/21/2017 01:33 PM > > > ----- > > > > > > From: "Loon, Eric van (ITOPT3) - KLM" <eric-van.l...@klm.com> > > > To: ADSM-L@VM.MARIST.EDU > > > Date: 05/19/2017 07:28 AM > > > Subject: Re: Client-side dedup speed > > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > > > > > Hi Stefan! > > > Thanks for your reply! > > > I have done some testing on this client and I discovered that the > > > local deduplication cache is the bottleneck here. Apparently going > > > through the local cache will only be beneficial when it is not getting > > > > too large. As soon as I disabled the local cache, backup performance > > > is improved drastically. As soon as the RESOURCEUTILIZATION is raised > > > (from 2 to 5) the performance is improved even more! > > > I'm a happy guy again, for those of you interested, I included the > > > statistics of all test down below. > > > Kind regards, > > > Eric van Loon > > > Air France/KLM Storage Engineering > > > > > > Server-side deduplication: > > > Total number of objects backed up: 1,926,689 > > > Total number of bytes transferred: 227.43 GB > > > Total data reduction ratio: 0.05% > > > Elapsed processing time: 03:54:34 > > > > > > Client-side deduplication: > > > Total number of objects backed up: 1,934,305 > > > Total objects deduplicated: 656,414 > > > Total bytes before deduplication: 229.95 GB > > > Total bytes after deduplication: 1.31 GB > > > Deduplication reduction: 99.44% > > > Elapsed processing time: 12:02:03 > > > > > > Client-side deduplication, ENABLEDEDUPCACHE NO: > > > Total number of objects backed up: 1,956,018 > > > Total objects deduplicated: 661,502 > > > Total bytes before deduplication: 220.51 GB > > > Total bytes after deduplication: 0 B > > > Deduplication reduction: 100.00% > > > Total data reduction ratio: 99.67% > > > Elapsed processing time: 04:20:33 > > > > > > Client-side deduplication, ENABLEDEDUPCACHE NO RESOURCEUTILIZATION 5: > > > Total number of objects backed up: 1,973,335 > > > Total objects deduplicated: 670,950 > > > Total bytes before deduplication: 226.77 GB > > > Total bytes after deduplication: 0 B > > > Deduplication reduction: 100.00% > > > Total data reduction ratio: 99.68% > > > Elapsed processing time: 02:12:22 > > > > > > > > > -----Original Message----- > > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf > > > Of Stefan Folkerts > > > Sent: zaterdag 13 mei 2017 20:40 > > > To: ADSM-L@VM.MARIST.EDU > > > Subject: Re: Client-side dedup speed > > > > > > Hi Eric, > > > > > > I've seen client-side dedup be slower on 1Gb/s networks every time I > > > test it, every single time. > > > It doesn't surprise me at all just looking at the added latency to > > > check if a chunk is already in Spectrum Protect or not, sending it via > > > > the network, network latency (big one), reading form the database and > > > than all the way back to the client has got to be a LOT slower than > > > doing it all in the server, at least when the local cache isn't very > > > effective and with two million files I doubt it is very effective (but > > > > I don't know this, haven't done enough testing with different data > > > source types). > > > Network latency on a round trip alone probably adds a lot of time (% > > > wise) to just a local SSD database read alone, SSD's have a latency of > > > > 0.0something ms and everything stay's in compute with server side > > dedup. > > > > > > I only use client-side dedup with WAN networks or when I test and > > > would see that the speeds are comparable, I think this would be due to > > > > a very low latency network but I wonder if others here have done more > > > testing on this. > > > I believe that when you have very heavy loads you can put more in the > > > server in an night of backups because there is a smaller load on the > > > server because the clients do some work but again, I only use it when > > > sending data thru WAN connections, for those scenario's it's perfect > > > especially in combination with 8.1 B/A client-side compression. > > > > > > > > > > > > > > > On Fri, May 12, 2017 at 1:52 PM, Loon, Eric van (ITOPT3) - KLM < > > > eric-van.l...@klm.com> wrote: > > > > > > > Dear TSM-ers, > > > > We are almost in production with our first directory containerpool > > > > TSM > > > > > > server. We did a lot of testing with client vs server-side dedup and > > > > > overall client-side was faster. But not on one of our Linux clients. > > > > This machine has a filesystem with nearly two million files, total > > > > amount of data is 227 GB. The server backs up through 1Gb Ethernet > > > > and > > > > > > with server-side dedup the backup takes 03:54. With client-side > > > > dedup enabled, the backup runs for 12:02! The client contains 16 > > > > CPU's and average utilization is no more than 10%. No swapping > > during a backup. > > > > I opened a PMR, but the support offices tries to convince me that > > > > client-side dedup is slower (twice as slow) by default. I really > > > > have a very hard time believing this. What is your experience with > > > > client-side dedup in combination with a large amount of smaller > > > (approx. 1 Mb) files? > > > > Thanks for any help in advance! > > > > Kind regards, > > > > Eric van Loon > > > > Air France/KLM Storage Engineering > > > > ******************************************************** > > > > For information, services and offers, please visit our web site: > > > > http://www.klm.com. This e-mail and any attachment may contain > > > > confidential and privileged material intended for the addressee > only. > > > > If you are not the addressee, you are notified that no part of the > > > > e-mail or any attachment may be disclosed, copied or distributed, > > > > and that any other action related to this e-mail or attachment is > > > > strictly > > > > > > prohibited, and may be unlawful. If you have received this e-mail by > > > > > error, please notify the sender immediately by return e-mail, and > > > delete this message. > > > > > > > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries > > > > and/or its employees shall not be liable for the incorrect or > > > > incomplete transmission of this e-mail or any attachments, nor > > > > responsible > > > for any delay in receipt. > > > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal > > > > Dutch > > > > Airlines) is registered in Amstelveen, The Netherlands, with > > > > registered number 33014286 > > > > ******************************************************** > > > > > > > ******************************************************** > > > For information, services and offers, please visit our web site: > > > http://www.klm.com. This e-mail and any attachment may contain > > > confidential and privileged material intended for the addressee only. > > > If you are not the addressee, you are notified that no part of the > > > e-mail or any attachment may be disclosed, copied or distributed, and > > > that any other action related to this e-mail or attachment is strictly > > > > prohibited, and may be unlawful. If you have received this e-mail by > > > error, please notify the sender immediately by return e-mail, and > > > delete this message. > > > > > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/ or > > > > its employees shall not be liable for the incorrect or incomplete > > > transmission of this e-mail or any attachments, nor responsible for > > > any delay in receipt. > > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal > > > Dutch Airlines) is registered in Amstelveen, The Netherlands, with > > > registered number 33014286 > > > ******************************************************** > > >
