Re: Real world deduplication rates with TSM 7.1 and container pools

2016-03-21 Thread PAC Brion Arnaud
Michael, Ken and Stephan,

Thanks a lot for valuable input !

Based on your feedback, I believe that IBM is effectively overselling it's 
product  : best value announced is around 65 %, which means dedup factor around 
2.5 ...

Reason why I asked is that we are currently making use of Data Domain VTL's in 
our shop, which at present time have a dedup factor of 7.7, but are aging and 
should soon be retired.
I was wondering if their replacement with a combination of cheap disk storage 
and TSM deduped container would be a good idea ... 
So far I still need to be convinced : disk (IBM, Hitachi ...) is way cheaper 
than a VTL, but TSM dedup rates are seeming to be less than expected : this 
will probably force us to buy more disks, thus making such a solution less 
attractive.

Cheers.

Arnaud

-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Stefan 
Folkerts
Sent: Friday, March 18, 2016 5:32 PM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: Real world deduplication rates with TSM 7.1 and container pools

We see around 50-65% deduplication savings on the fileclass storagepools,
most common seems to be around 55%.
It requires what I call "deep reclaims" with very low values that need a
lot of time.
We are seeing 60-70% on containerpools but on average it is more like 65%
but that is based on a much smaller install base.
Both in heterogeneous environments.


On Fri, Mar 18, 2016 at 5:06 PM, Ken Bury  wrote:

> I have two 7.1.4 servers, one with devclass file with dedupe, and the other
> is using containers. The two servers are in a node replication pair so the
> data on each server is exactly the same. The workload is almost exclusively
> vmware backups with datamover dedupe and compression. The data reduction
> for both pools is 89%. I like what I am getting from container pools and
> replication.
> On Fri, Mar 18, 2016 at 11:35 Ryder, Michael S 
> wrote:
>
> > Hi Arnaud
> >
> > If IBM made that commitment in black and white, then you should hold
> their
> > feet to the fire.  But I am willing to bet this was a salesman promising
> > "similar performance."
> >
> > There is no technology I know where any deduplication factor can be
> > guaranteed.  Perhaps "UP to 4" for certain kinds of data...  And overall
> > reduction of storage is what you should be comparing, not simply the
> > deduplication percentage.
> >
> > Here, try reading at least the introduction of this document, " Effective
> > Planning and Use of TSM V6 and V7 Deduplication"
> >
> >
> >
> http://www.ibm.com/developerworks/community/wikis/form/anonymous/api/wiki/f731037e-c0cf-436e-88b5-862b9a6597c3/page/82e361b4-8e96-42cf-b559-0b77df9aed2c/attachment/5cf980b3-807f-464b-a1c0-b896b0cec7e6/media/TSM%20Dedup%20Best%20Practices%20-%20v2.1.pdf
> >
> > We haven't adopted the directory-container pools yet due to their lacking
> > of support for important features like migration and copy pools, but I
> have
> > no doubt that IBM will be delivering those abilities soon; otherwise,
> there
> > are very limited use-cases for directory-containers.
> >
> > Best regards,
> >
> > Mike
> > RMD IT Client Services
> >
> > On Fri, Mar 18, 2016 at 10:41 AM, PAC Brion Arnaud <
> > arnaud.br...@panalpina.com> wrote:
> >
> > > Hi All,
> > >
> > > We are currently testing TSM 7.1 deduplication feature, in conjunction
> > > with container based storage pools.
> > > So far, my test TSM instances, installed with such a setup are
> reporting
> > > dedup percentage of 45 %, means dedup factor around 1.81, using a
> sample
> > of
> > > clients which are representative of our production environment.
> > > This is unfortunately pretty far from what was promised by IBM (dedup
> > > factor of 4) ...
> > >
> > > I'm wondering if anybody making use of container based storage pools
> and
> > > deduplication would be sharing his deduplication factor, so that I
> could
> > > have a better appreciation of real world figures.
> > > If you would be so kind to share your information (possibly with the
> kind
> > > of backed-up data  i.e. VM, DB, NAS, Exchange, and retention values
> ...)
> > I
> > > would be very grateful !
> > >
> > > Thanks in advance for appreciated feedback.
> > >
> > > Cheers.
> > >
> > > Arnaud
> > >
> > >
> > >
> >
> **
> > > Backup and Recovery Systems Administrator
> > > Panalpina Management Ltd., Basle, Switzerland,
> > > CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH
> > > Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01
> > > Direct: +41 (61) 226 19 78
> > > e-mail: arnaud.br...@panalpina.com
> > > This electronic message transmission contains information from
> Panalpina
> > > and is confidential or privileged. This information is intended only
> for
> > > the person (s) named above. If you are not the intended recipient, any
> > > disclosure, copying,

Re: Real world deduplication rates with TSM 7.1 and container pools

2016-03-21 Thread Hans Christian Riksheim
We are also getting around 60% in dedup savings. When I compare similar
data going to tape I find that simple compression saves us about 40%.

Hans Chr.

On Mon, Mar 21, 2016 at 8:54 AM, PAC Brion Arnaud <
arnaud.br...@panalpina.com> wrote:

> Michael, Ken and Stephan,
>
> Thanks a lot for valuable input !
>
> Based on your feedback, I believe that IBM is effectively overselling it's
> product  : best value announced is around 65 %, which means dedup factor
> around 2.5 ...
>
> Reason why I asked is that we are currently making use of Data Domain
> VTL's in our shop, which at present time have a dedup factor of 7.7, but
> are aging and should soon be retired.
> I was wondering if their replacement with a combination of cheap disk
> storage and TSM deduped container would be a good idea ...
> So far I still need to be convinced : disk (IBM, Hitachi ...) is way
> cheaper than a VTL, but TSM dedup rates are seeming to be less than
> expected : this will probably force us to buy more disks, thus making such
> a solution less attractive.
>
> Cheers.
>
> Arnaud
>
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of
> Stefan Folkerts
> Sent: Friday, March 18, 2016 5:32 PM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: Real world deduplication rates with TSM 7.1 and container
> pools
>
> We see around 50-65% deduplication savings on the fileclass storagepools,
> most common seems to be around 55%.
> It requires what I call "deep reclaims" with very low values that need a
> lot of time.
> We are seeing 60-70% on containerpools but on average it is more like 65%
> but that is based on a much smaller install base.
> Both in heterogeneous environments.
>
>
> On Fri, Mar 18, 2016 at 5:06 PM, Ken Bury  wrote:
>
> > I have two 7.1.4 servers, one with devclass file with dedupe, and the
> other
> > is using containers. The two servers are in a node replication pair so
> the
> > data on each server is exactly the same. The workload is almost
> exclusively
> > vmware backups with datamover dedupe and compression. The data reduction
> > for both pools is 89%. I like what I am getting from container pools and
> > replication.
> > On Fri, Mar 18, 2016 at 11:35 Ryder, Michael S <
> michael_s.ry...@roche.com>
> > wrote:
> >
> > > Hi Arnaud
> > >
> > > If IBM made that commitment in black and white, then you should hold
> > their
> > > feet to the fire.  But I am willing to bet this was a salesman
> promising
> > > "similar performance."
> > >
> > > There is no technology I know where any deduplication factor can be
> > > guaranteed.  Perhaps "UP to 4" for certain kinds of data...  And
> overall
> > > reduction of storage is what you should be comparing, not simply the
> > > deduplication percentage.
> > >
> > > Here, try reading at least the introduction of this document, "
> Effective
> > > Planning and Use of TSM V6 and V7 Deduplication"
> > >
> > >
> > >
> >
> http://www.ibm.com/developerworks/community/wikis/form/anonymous/api/wiki/f731037e-c0cf-436e-88b5-862b9a6597c3/page/82e361b4-8e96-42cf-b559-0b77df9aed2c/attachment/5cf980b3-807f-464b-a1c0-b896b0cec7e6/media/TSM%20Dedup%20Best%20Practices%20-%20v2.1.pdf
> > >
> > > We haven't adopted the directory-container pools yet due to their
> lacking
> > > of support for important features like migration and copy pools, but I
> > have
> > > no doubt that IBM will be delivering those abilities soon; otherwise,
> > there
> > > are very limited use-cases for directory-containers.
> > >
> > > Best regards,
> > >
> > > Mike
> > > RMD IT Client Services
> > >
> > > On Fri, Mar 18, 2016 at 10:41 AM, PAC Brion Arnaud <
> > > arnaud.br...@panalpina.com> wrote:
> > >
> > > > Hi All,
> > > >
> > > > We are currently testing TSM 7.1 deduplication feature, in
> conjunction
> > > > with container based storage pools.
> > > > So far, my test TSM instances, installed with such a setup are
> > reporting
> > > > dedup percentage of 45 %, means dedup factor around 1.81, using a
> > sample
> > > of
> > > > clients which are representative of our production environment.
> > > > This is unfortunately pretty far from what was promised by IBM (dedup
> > > > factor of 4) ...
> > > >
> > > > I'm wondering if anybody making use of container based storage pools
> > and
> > > > deduplication would be sharing his deduplication factor, so that I
> > could
> > > > have a better appreciation of real world figures.
> > > > If you would be so kind to share your information (possibly with the
> > kind
> > > > of backed-up data  i.e. VM, DB, NAS, Exchange, and retention values
> > ...)
> > > I
> > > > would be very grateful !
> > > >
> > > > Thanks in advance for appreciated feedback.
> > > >
> > > > Cheers.
> > > >
> > > > Arnaud
> > > >
> > > >
> > > >
> > >
> >
> **
> > > > Backup and Recovery Systems Administrator
> > > > Panalpina Management Ltd.,

Re: Help on Directory Container

2016-03-21 Thread PAC Brion Arnaud
Hi Robert,

If not already done, you should have a look at following page : 
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Directory-container%20storage%20pools%20FAQs
 
There's a note there (Q8) explaining that "protect stgpool" should be used 
before "replicate node". 

In fact if you use the wizard which is now part of the 7.1.x OC to setup 
replication, you will see that it automatically creates some admin schedules 
which are run every 6 hours (can be customized), that are triggering "protect 
stgpool" commands. 
In addition to this, it also schedules a "replicate" job, which is run once a 
day, as part of TSM daily maintenance jobs.

Regarding audit and repair commands :

- Audit container : will by default use the " SCANAll l" options, which 
basically will look for damaged extents in a container. If any problem is 
found, you can then use "q damaged" to get better information on what client 
file has been damaged.

Option REMOVEDamaged means that the damaged extents will be deleted from TSM 
DB. To my mind it should be used only after an unsuccessful "repair" attempt, 
or  if there's no protection pool.
Option MARKDamaged  will force TSM to mark all of the involved container 
extents as damaged. To my mind, it should be used only if willing to forcibly 
delete a container, and should in this case be followed by an audit with option 
REMOVEDamaged.
Option SCANDamaged will force TSM to check again the extents previously marked 
as damaged. It should be used after "repair" command to reset the error 
conditions (which would be reset with SCANAll option too)

- Repair container : well, it is pretty self-explanatory : once client data 
extents have been found damaged, whether by some process, or following to an 
audit command, it will allow you to restore them, on condition that a 
protection storage pool exists, and that "protect stgpool" commands have been 
used. It is the equivalent of "restore stgpool" command , but for directory 
pools.

Hope this helped ...

Arnaud

-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Robert 
Ouzen
Sent: Monday, March 21, 2016 6:50 AM
To: ADSM-L@VM.MARIST.EDU
Subject: Help on Directory Container

Hi to all

I need a little bit more information about all the administrative process on my 
TSM environment .

I have a TSM server 7.1.5 on an O.S Windows server 2008 R2 64B.
I have mix STGPOOLS , some of type FILE (with DEDUP on server side) and some of 
type DIRECTORY

For now my order process of administrative tasks is:


1.   At late afternoon for few hours , run IDENTIFY DUPLICATES

2.   At morning 07:00 ,   run EXPIRE INVENTORY

3.   After it, run 
BACKUP DB

4.   After it, run 
RECLAIM STG

Now because I created new STGpools with type DIRECTORY (inline dedup), and  
create too an TARGET  server version 7.1.5..
Update my stgpools of type DIRECTORYwith the option:  
PROTECTstgpool=Targetstgppol  (on Target server)

Now the questions:

What will be the correct order process to add the new tasks as:

PROTECT STGPOOL
REPLICATE NODE

And what will be the correct command to repair damage container in source STG   
?

As  AUDIT CONTAINER ( with which options ??) and REPAIR CONTAINER???

T.I.A  Regards

Robert


[cid:image001.png@01D1284F.C3B0B400]

רוברט אוזן
ראש תחום שרידות וזמינות נתונים.
אוניברסיטת חיפה
משרד: בניין ראשי, חדר 5015
טלפון:  04-8240345 (פנימי: 2345)
דואר: rou...@univ.haifa.ac.il
_
אוניברסיטת חיפה | שד' אבא חושי 199 | הר הכרמל, חיפה | מיקוד: 3498838
אתר אגף מחשוב ומערכות מידע: 
http://computing.haifa.ac.il


Re: Real world deduplication rates with TSM 7.1 and container pools

2016-03-21 Thread Bas van Kampen

We have around 900 VM's in 2 container pools and we get 35% dedup
percentage.

Regards,

Bas van Kampen

On 18-3-2016 15:41, PAC Brion Arnaud wrote:

Hi All,

We are currently testing TSM 7.1 deduplication feature, in conjunction with 
container based storage pools.
So far, my test TSM instances, installed with such a setup are reporting dedup 
percentage of 45 %, means dedup factor around 1.81, using a sample of clients 
which are representative of our production environment.
This is unfortunately pretty far from what was promised by IBM (dedup factor of 
4) ...

I'm wondering if anybody making use of container based storage pools and 
deduplication would be sharing his deduplication factor, so that I could have a 
better appreciation of real world figures.
If you would be so kind to share your information (possibly with the kind of 
backed-up data  i.e. VM, DB, NAS, Exchange, and retention values ...) I would 
be very grateful !

Thanks in advance for appreciated feedback.

Cheers.

Arnaud

**
Backup and Recovery Systems Administrator
Panalpina Management Ltd., Basle, Switzerland,
CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH
Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01
Direct: +41 (61) 226 19 78
e-mail: arnaud.br...@panalpina.com
This electronic message transmission contains information from Panalpina and is 
confidential or privileged. This information is intended only for the person 
(s) named above. If you are not the intended recipient, any disclosure, 
copying, distribution or use or any other action based on the contents of this 
information is strictly prohibited.

If you receive this electronic transmission in error, please notify the sender 
by e-mail, telephone or fax at the numbers listed above. Thank you.
**



Re: Real world deduplication rates with TSM 7.1 and container pools

2016-03-21 Thread PAC Brion Arnaud
Hi Bas,

Thanks for feedback. 

On our side average dedup is 38% for VM's data , with daily incremental and 30 
days retention ...

Cheers.

Arnaud

-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Bas 
van Kampen
Sent: Monday, March 21, 2016 12:15 PM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: Real world deduplication rates with TSM 7.1 and container pools

We have around 900 VM's in 2 container pools and we get 35% dedup
percentage.

Regards,

Bas van Kampen

On 18-3-2016 15:41, PAC Brion Arnaud wrote:
> Hi All,
>
> We are currently testing TSM 7.1 deduplication feature, in conjunction with 
> container based storage pools.
> So far, my test TSM instances, installed with such a setup are reporting 
> dedup percentage of 45 %, means dedup factor around 1.81, using a sample of 
> clients which are representative of our production environment.
> This is unfortunately pretty far from what was promised by IBM (dedup factor 
> of 4) ...
>
> I'm wondering if anybody making use of container based storage pools and 
> deduplication would be sharing his deduplication factor, so that I could have 
> a better appreciation of real world figures.
> If you would be so kind to share your information (possibly with the kind of 
> backed-up data  i.e. VM, DB, NAS, Exchange, and retention values ...) I would 
> be very grateful !
>
> Thanks in advance for appreciated feedback.
>
> Cheers.
>
> Arnaud
>
> **
> Backup and Recovery Systems Administrator
> Panalpina Management Ltd., Basle, Switzerland,
> CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH
> Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01
> Direct: +41 (61) 226 19 78
> e-mail: arnaud.br...@panalpina.com
> This electronic message transmission contains information from Panalpina and 
> is confidential or privileged. This information is intended only for the 
> person (s) named above. If you are not the intended recipient, any 
> disclosure, copying, distribution or use or any other action based on the 
> contents of this information is strictly prohibited.
>
> If you receive this electronic transmission in error, please notify the 
> sender by e-mail, telephone or fax at the numbers listed above. Thank you.
> **
>


Re: Anyone using TSM Snapdiff support (linux and windows) to backup ONTAP CDOT 8.3

2016-03-21 Thread Paul Zarnowski
Thank you Sameer and Erwan.  We got it working.

At 11:36 PM 3/17/2016, Sameer Veer wrote:
>Hello Paul,
>
>Earlier this month, for one of my customers, I configured NetApp 
>snapshot-assisted progressive incremental functionality ('incremental 
>-snapdiff' command) for "cluster mode" on the ONTAP CDOT 8.3 for both Windows 
>and Linux shares successfully.
>
>We used TSM B/A client 7.1.4.2 for this configuration. Refer to this Techdoc 
>for TSM B/A client supported levels:
>http://www-01.ibm.com/support/docview.wss?uid=swg21474217
>
>In summary:
>- Spectrum Protect Client 7.1.2 (or higher) supports ONTAP 8.1.y & 8.2.y where 
>y is 0 or higher (8.2 both 7-mode and c-mode)
>- Spectrum Protect Client 7.1.3 (or higher) supports ONTAP 8.3.y c-mode in 
>addition to the above
>- Spectrum Protect Client 7.1.4 (or higher) similar to 7.1.3
>
>Regards,
>
>Sameer R. Veer
>Senior Managing Consultant, Services Delivery
>IBM Systems | Storage Software Solutions
>Storage
> Services Offerings
> 
>sv...@us.ibm.com
>Cell Phone:+1 607 343 0996
>
>
>
>
>
>
>
>From:Erwann SIMON 
>To:ADSM-L@VM.MARIST.EDU
>Date:03/17/2016 06:03 PM
>Subject:Re: [ADSM-L] Anyone using TSM Snapdiff support (linux and 
>windows) to backup ONTAP CDOT 8.3
>Sent by:"ADSM: Dist Stor Manager" 
>
>
>
>
>Hello Paul,
>
>I've done this setup for a customer last novembre (TSM 7.1.3) for both Windows 
>and Linux shares.
>We had it working as expected, thanks to the skilled people of this shop.
>
>Indeed, the TSM B/A Client documentation is (was ?) not accurate for CDOT as 
>it refers mainly to 7 Mode.
>
>-- 
>Best regards / Cordialement / مع تحياتي
>Erwann SIMON
>
>- Mail original -
>De: "Paul Zarnowski" 
>À: ADSM-L@VM.MARIST.EDU
>Envoyé: Jeudi 17 Mars 2016 21:41:51
>Objet: [ADSM-L] Anyone using TSM Snapdiff support (linux and windows) to 
>backup ONTAP CDOT 8.3
>
>Has anyone successfully used TSM on Windows (and Linux) to backup a NetApp 
>filer running ONTAP CDOT 8.3?  We're looking for confirmation that this works, 
>as we've been having some problems getting it to run against a simulator 
>running 8.3.
>
>Thanks!
>..Paul
>
>
>--
>Paul ZarnowskiPh: 607-255-4757
>Assistant Director for Storage Services   Fx: 607-255-8521
>IT at Cornell / InfrastructureEm: p...@cornell.edu
>719 Rhodes Hall, Ithaca, NY 14853-3801
>
>
>


--
Paul ZarnowskiPh: 607-255-4757
Assistant Director for Storage Services   Fx: 607-255-8521
IT at Cornell / InfrastructureEm: p...@cornell.edu
719 Rhodes Hall, Ithaca, NY 14853-3801  


Re: APAR IT12942

2016-03-21 Thread Del Hoobler
Hi Neil,

We are targeting inclusion of the new version of the VDDK in our 2Q16 
release.

Standard disclaimers apply. This is not a commitment, it is a target.


Thank you,

Del

---


"ADSM: Dist Stor Manager"  wrote on 03/18/2016 
12:52:55 PM:

> From: "Schofield, Neil (Storage & Middleware, Backup & Restore)" 
> 
> To: ADSM-L@VM.MARIST.EDU
> Date: 03/18/2016 12:53 PM
> Subject: APAR IT12942
> Sent by: "ADSM: Dist Stor Manager" 
> 
> VMware this week released VDDK 6.0.2 which opens up the possibility 
> of a fix to IT12942:
> 
> Would anyone from IBM care to speculate about how long it will take 
> before a TSM BA client based on this update will be forthcoming? 
> We've got some deployment decisions to make that depend on the answer.
> 
> Regards
> Neil Schofield
> Tivoli Storage Manager SME
> Backup & Recovery | Storage & Middleware | Central Infrastructure 
> Services | Infrastructure & Service Delivery | Group IT
> LLOYDS BANKING GROUP


Re: Real world deduplication rates with TSM 7.1 and container pools

2016-03-21 Thread Del Hoobler
I think most of you know Spectrum Protect just added in-line compression 
to the container and cloud deduplicated pools in version 7.1.5:

 
http://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.5/srv.common/r_techchg_srv_compress_715.html

Adding incremental forever, the new in-line deduplication - client or 
server-based (7.1.3), new in-line compression (7.1.5) I think you will 
find that Protect continues to drive overall data reduction. This is being 
done in the software, so you can choose what disk you want to use.

I encourage folks to try out the the new deduplication along with the new 
compression to see how it helps with the overall data reduction. 


Thank you,

Del





"ADSM: Dist Stor Manager"  wrote on 03/18/2016 
10:41:06 AM:

> From: PAC Brion Arnaud 
> To: ADSM-L@VM.MARIST.EDU
> Date: 03/18/2016 10:41 AM
> Subject: Real world deduplication rates with TSM 7.1 and container pools
> Sent by: "ADSM: Dist Stor Manager" 
> 
> Hi All,
> 
> We are currently testing TSM 7.1 deduplication feature, in 
> conjunction with container based storage pools.
> So far, my test TSM instances, installed with such a setup are 
> reporting dedup percentage of 45 %, means dedup factor around 1.81, 
> using a sample of clients which are representative of our production
> environment.
> This is unfortunately pretty far from what was promised by IBM 
> (dedup factor of 4) ...
> 
> I'm wondering if anybody making use of container based storage pools
> and deduplication would be sharing his deduplication factor, so that
> I could have a better appreciation of real world figures.
> If you would be so kind to share your information (possibly with the
> kind of backed-up data  i.e. VM, DB, NAS, Exchange, and retention 
> values ...) I would be very grateful !
> 
> Thanks in advance for appreciated feedback.
> 
> Cheers.
> 
> Arnaud
> 
> 
**
> Backup and Recovery Systems Administrator
> Panalpina Management Ltd., Basle, Switzerland,
> CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH
> Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01
> Direct: +41 (61) 226 19 78
> e-mail: arnaud.br...@panalpina.com
> This electronic message transmission contains information from 
> Panalpina and is confidential or privileged. This information is 
> intended only for the person (s) named above. If you are not the 
> intended recipient, any disclosure, copying, distribution or use or 
> any other action based on the contents of this information is 
> strictly prohibited.
> 
> If you receive this electronic transmission in error, please notify 
> the sender by e-mail, telephone or fax at the numbers listed above. 
Thank you.
> 
**
>