Wanda, one other thing I would like to point out. With the TSM Server being on Windows, one other thing to consider is RSS (Receive Side Scaling). All the processing of IP packets happens on 1 core of the server, and this can be a potential bottleneck. One way to determine this is to watch the TSM server when the Exchange Backup occurs. Is there a CPU spike on 1 core? Not only can you enable RSS to spread the packet processing out amongst more cores, but you can specify how many cores and which core numbers to start with. If a perfmon run is done against the Windows Server, it is easy to see if one particular core is doing more worth than the others, and this can be a symptom that RSS may provide a benefit.
Dave Canan TSM Performance ddca...@us.ibm.com On Mon, Feb 10, 2014 at 8:16 AM, Prather, Wanda <wanda.prat...@icfi.com>wrote: > Thank you - forgot to mention this is a Windows TSM server. > I am curious that the drive is the bottleneck - a big file of zeros should > compress, and give you > 200MB/sec on LTO5, yes? > > > -----Original Message----- > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of > Hans Christian Riksheim > Sent: Monday, February 10, 2014 9:04 AM > To: ADSM-L@VM.MARIST.EDU > Subject: Re: [ADSM-L] Exchange 2010 backup performance > > In my experience there is nothing wrong with the TCP stack in Windows. > Especially Windows2008R2 performs very well. For a single stream from a > 2008R2 client (dsm sel <big file of zeroes>) to an AIX TSM-server 500km > away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. > Bottleneck being the drive. > > After too much experimenting I have found the critical factor to be to set > TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes > in AIX(and override the tcp-settings on the NIC). Windows OS can be left > alone as its default is quite OK. YMMV of course. > > Regards, > > Hans Chr. > > > On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve <steve_sch...@bcbst.com > >wrote: > > > Wanda, > > > > I have fought with this problem myself, and here is what I concluded > > (at least in our environment, YMMV): > > > > 1. Running single-stream backups (one db at a time) you will never see > > the performance you expect, due to the Windows O/S tcpip stack. I > > haven't had a chance to stress-test Win2012-R2 yet, but at least > > through 2008-R2, there seems to be a single-thread constraint that > > prevents any backup from getting much more than about 20% of the > bandwidth. > > > > 2. The only way to get around this is to do as Del suggests and > > parallelize your backups. If you can get 4-6 concurrent jobs running, > > you can push the network card pretty close to 100%. The catch, as > > Dell also pointed out, is that you can't run concurrent backups on > > databases that live on the same disk (since the vss snap is at the disk > level). > > > > Bottom line is that you would need to divide up your Exchange > > databases so they are on different disks (or at least, create as many > > disks as you want to have concurrent backups, then create separate jobs > to backup each group). > > > > Good luck, > > > > Steve Schaub > > System Engineer II, Backup/Recovery > > Blue Cross Blue Shield of Tennessee > > > > > > -----Original Message----- > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf > > Of Prather, Wanda > > Sent: Sunday, February 09, 2014 1:08 PM > > To: ADSM-L@VM.MARIST.EDU > > Subject: Re: [ADSM-L] Exchange 2010 backup performance > > > > Del, you are a national treasure! > > You are very kind to take time to respond. > > > > My backups are already very well balanced, I have 2 servers, the DBA's > > have the DBs split between them so well that they backup almost the > > same amount of data, and finish within 30 minutes of each other. > > (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, > > with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 > > disk so should be spiffy speed there.). > > > > I tried setting resourceutilization 10 once before, was an impressive > > failure. The backup appeared to be looping doing VSS snaps (or rather > > failing to); I think it was doing as you mentioned in 2 below, trying > > to snap the same LUN multiple times. > > > > Will go through the references you included, then open a performance > > PMR if no improvement. > > > > Thank you so much! > > > > W > > > > -----Original Message----- > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf > > Of Del Hoobler > > Sent: Friday, February 07, 2014 6:48 PM > > To: ADSM-L@VM.MARIST.EDU > > Subject: Re: [ADSM-L] Exchange 2010 backup performance > > > > Hi Wanda, > > > > I have a few ideas for you... > > > > -------------------------- > > > > Are you running in a DAG environment? If so, you could do some load > > balancing between DAG Servers: > > > > Most of this in the Exchange book under "Managing Exchange Database > > Availability Group members by using a single policy": > > > > > > > > http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. > > ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html > > > > The key to "load balance" when setting up the scheduled backup script > > is to have a separate invocation of each database. For example: > > > > TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE > > TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE > > TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE > > TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE > > TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE > > > > Then, run this command from each of the Exchange servers at or about > > the same time. > > > > -------------------------- > > > > Here are a few more things to look at: > > > > To help with some performance issues, some customers have split their > > backups into multiple "threads" or "processes" in two ways: > > > > 1. Increase the value of the RESOURCEUTILIZATION parameter in the > > DSM.OPT file for the DSMAGENT. Trying setting this to "10". > > Important: This needs to the DSM.OPT file for the DSMAGENT > > not the DP/Exchange options file. > > > > 2. Split the backups into multiple parallel instances of the > > TDPEXCC backup execution. > > i.e. the create separate invocations of DP/Exchange that back > > up a different set of databases. For example: > > TDPEXCC BACKUP db1,db2,db3,db4 FULL > > TDPEXCC BACKUP db5,db6,db7,db8 FULL > > TDPEXCC BACKUP db9,db10,db11,db12 FULL > > Put these in separate command files and stagger the > > launching of them by 10 minutes or so. > > The key here is that you need to make sure that you don't > > have any LUNs that appears in more than one invocation. > > In other words, you don't want to snapshot the > > same LUN in separate invocations. > > > > Note: The integrity check is a Microsoft tool. IBM has no control over > > the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL > > program to perform the integrity check. It's a very I/O intensive > > program that must examine every page of the database file (.EDB) and all > log files. > > > > -------------------------- > > > > If none of these help, you should open a PMR to get the performance > > team to look at your environment. > > > > > > > > Thank you, > > > > Del > > > > ---------------------------------------------------- > > > > "ADSM: Dist Stor Manager" <ADSM-L@vm.marist.edu> wrote on 02/07/2014 > > 06:04:01 PM: > > > > > From: "Prather, Wanda" <wanda.prat...@icfi.com> > > > To: ADSM-L@vm.marist.edu, > > > Date: 02/07/2014 06:06 PM > > > Subject: Exchange 2010 backup performance Sent by: "ADSM: Dist Stor > > > Manager" <ADSM-L@vm.marist.edu> > > > > > > Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in > > > the baclient dsm.opt? > > > Or is there anything else I can tweak to improve TSM throughput of a > > > 2010 full backup? > > > Got a 10G network, but Exchange full backup performance not impressive. > > > > > > Thanks for any ideas - links to relevant doc also appreciated! > > > > > > Wanda > > > > > > > > > **Please note new office phone: > > > Wanda Prather | Senior Technical Specialist | > > > wanda.prat...@icfi.com > > | > > > www.icfi.com<http://www.icfi.com> | 410-868-4872 (m) ICF > > > International > > > | 7125 Thomas Edison Dr., Suite 100, Columbia, Md > > > |443-718-4900 (o) > > > > > ----------------------------------------------------- > > Please see the following link for the BlueCross BlueShield of > > Tennessee E-mail disclaimer: > > http://www.bcbst.com/email_disclaimer.shtm > > >
