Hi Richard! The log only contains all transactions (since the last database backup) when running in rollforward mode. When running in normal mode, only uncommitted transactions are written to the log. As soon as a transaction is completed, the transaction is committed to the database and thus removed from the log. I agree that a slow running client backing up very large files can pin the log for a very long time, but the log only contains transaction data. Per backup file TSM stores 600 bytes of data in the database, so a 12 Gb recovery should be able to hold quite a lot of uncommitted client transactions! When sending smaller files: as soon as the txnbytelimit is reached, the server commits the transaction, so slow backup sessions sending smaller files don't necessarily pin the log. Anyway, I'm monitoring logpinning on regular intervals and there is little logpinning except for occasional TDP for SQL nodes with very large databases. Kind regards, Eric van Loon KLM Royal Dutch Airlines
-----Original Message----- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Richard Rhodes Sent: donderdag 12 mei 2011 17:23 To: ADSM-L@VM.MARIST.EDU Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code No . . . It's completely possible for the the log in normal mode to be pinned and hit 100%. My understanding is that the log contains ALL transactions. It's the checkpoint/rollback for info for the db. An open tran (uncommitted?) pins the log. We get them every once in a while when some server sends a big file very slowly - usually a remote site, but sometimes a local with something set wrong (my favorite is the half-duplex ethernet connection which runs at 50k/s). But it's not just the log being pinned by a long running tran that is a concern, it's also the load on the TSM server for all the other backups that are running and their transactions that determine how quickly a pinned log fills up. Here is the log consumption for the past day for six of our tsm servers: tsm1 3gb tsm2 2gb tsm3 8gb tsm4 10gb tsm5 4gb tsm6 6gb They all have 12gb of log space. Back a few years ago, tsm1 was consuming over 20gb per day and we had tons of logpin problems, one time resulting in the log filling completely. This was when I put in the monitoring with the auto logpin cancel. It was one of the many indications that we needed more instances. It's also why we have a daily report of slow and long running backups so we can identify these problem servers. Rick From: "Loon, EJ van - SPLXO" <eric-van.l...@klm.com> To: ADSM-L@VM.MARIST.EDU Date: 05/12/2011 10:54 AM Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> Hi Rick! You are running in normal mode. In this mode the recovery log only contains uncommited transactions. Don't you agree that in this case, the log should NEVER reach 90%? It should not even reach something like 40%! I don't want to have to implement extra monitoring to keep TSM from crashing. It should just work. Like it did when we were running 5.3... Kind regards, Eric van Loon KLM Royal Dutch Airlines -----Original Message----- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Richard Rhodes Sent: donderdag 12 mei 2011 15:04 To: ADSM-L@VM.MARIST.EDU Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code I have a cron job that checks log utilization every 15m. If the log goes over 90% the scripts displays a bunch of stuff (q sess, q proc, q log, show logpin, etc), runs a show logpin cancel, displays the stuff again, and finally emails me. (We run in normal mode.) It's been very effective in preventing a log full condition. rick From: "Bos, Karel" <karel....@atosorigin.com> To: ADSM-L@VM.MARIST.EDU Date: 05/12/2011 06:11 AM Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> Hi, Dunno if u already looked at these options: THROUGHPUTDATATHRESHOLD and THROUGHPUTTIMETHRESHOLD. After a couple of crashes due to log filling up, a couple of years back, I standard added them to all my clients (with low values to only kill the slowest/hung ones) and untill now never had any issues with the V5.5 and below log space under normal backup loads. Log filling only happend when some sys admin made some changes to the biggest file servers we had but even this was handled like it should be done by ITSM. Regards, Karel ________________________________________ Van: ADSM: Dist Stor Manager [ADSM-L@VM.MARIST.EDU] namens Loon, EJ van - SPLXO [eric-van.l...@klm.com] Verzonden: donderdag 12 mei 2011 11:11 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code Hi Paul! We are already running 5.5.5.2 and the log is still filling up, even after switching from rollforward to normal mode. Management currently is questioning whether TSM is the right product for the future. Although I'm a big fan of TSM for 15 years, I'm really in doubt too... Kind regards, Eric van Loon KLM Royal Dutch Airlines -----Original Message----- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Paul Fielding Sent: woensdag 11 mei 2011 20:05 To: ADSM-L@VM.MARIST.EDU Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code Hi folks, if this has already been commented on, my apologies, I hvaen't been following closely but just noticed this thread. We were experiencing log pinned issues after upgrading to 5.5.0.0 code at one of my client sites. What we were finding was that, occasionally, I'd get up in the morning and look at the server to see it completely bogged down with a huge number of backups still attempting to run, but nothing was moving - the log was at 84% (over it's trigger), and it had been firing off db backups for the last 4h to no avail, the log wasn't getting low enough. A key symptom was that in the actlog we were seeing messages to the effect of "Recovery log is at 84%, transactions will be delayed by 3ms" (or something like that). In opening a ticket, it was found there was an issue in the 5.5.0.0 code where, when the recovery log gets too full, it would start delaying transactions in order to prevent the log from filling too quickly. However, the side effect was that the pinning transaction would also get delayed, causing a bit of a never-ending loop. Transactions keep getting delayed, pinned transaction never gets to finish, and everything would just grind to a halt. I would have to halt the server and restart in order to get things back to normal. IBM recognized this as an issue and recommended going to any 5.5.5.0 level of code, where the problem was supposed to be fixed. I installed 5.5.5.2, and the problem has indeed gone away. It was supposed to be fixed at 5.5.5.0, though perhaps they didn't quite get it done at that code level as they hoped? I'd try installing 5.5.5.2 and see what happens.... regards, Paul On Wed, May 11, 2011 at 9:03 AM, Loon, EJ van - SPLXO <eric-van.l...@klm.com > wrote: > Hi Robert! > Thanks you very much for your reply! Several others on this list > reported this behavior and (as far as I know) three other users opened a > PMR too. I hope they have more luck, because I'm stuck. Level 2 keeps on > saying that the log keeps on growing because of slow running client > sessions. Indeed I see slow running client sessions, but they are slowed > down by the fact that TSM is delaying all transactions because the log > is used for more that 80% during a large part of the backup window! Now > they refuse to help me, unless I buy a Passport Advantage contract!!!!!! > Kind regards, > Eric van Loon > KLM Royal Dutch Airlines > > -----Original Message----- > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of > Robert Clark > Sent: woensdag 11 mei 2011 16:05 > To: ADSM-L@VM.MARIST.EDU > Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code > > I believe we've been seeing this problem as well. > > One night in the busiest backup period, I issued a "q actlog > begint=now-00:30", and got no results back but an error. > > I started dsmadmc -console on that server, and could see that the > console > output was most of an hour behind. (And the console output was scrolling > so fast that it could barely be read.) > > In that case, I think we determined that SQL LiteSpeed was set to some > rediculously small transaction size, and this was causing way too many > actlog entries. > > I think I also noted that the session number incremented by something > like > 100,000 in an hour. > > Asking the users of SQL LiteSpeed to make some changes was enough to > rememdy this problem, although we continue to fight with the logs > getting > full. > > Thanks, > [RC] > > > > From: "Loon, EJ van - SPLXO" <eric-van.l...@klm.com> > To: ADSM-L@VM.MARIST.EDU > Date: 05/11/2011 02:05 AM > Subject: Re: [ADSM-L] TSM Recovery log is pinning since upgrade > to > 5.5.5.0 code > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > > > Hi TSM-ers! > Here is a small follow up on my PMR about the recovery log utilization. > I'm at TSM level 2, still trying to convince them that there is > something broken in the TSM server code. To convince them, I have > changed the logmode to normal on one of my servers. I created a graph > (through TSMManager) which shows the recovery log utilization during > last night's client backup window and is doesn't differ much from the > night before, with logmode rollforward. When running in normal mode, TSM > should only use the recovery log for uncommitted transactions, so > utilization should be very low. My log is 12 Gb and the backuptrigger > value (75%) was still hit twice! > This clearly shows that there is something wrong with TSM, let's hope I > can convince Level 2 too, so my case gets forwarded to the lab. > I'll keep you guys posted! > Kind regards, > Eric van Loon > KLM Royal Dutch Airlines > ******************************************************** > For information, services and offers, please visit our web site: > http://www.klm.com. This e-mail and any attachment may contain > confidential and privileged material intended for the addressee only. If > you are not the addressee, you are notified that no part of the e-mail > or > any attachment may be disclosed, copied or distributed, and that any > other > action related to this e-mail or attachment is strictly prohibited, and > may be unlawful. If you have received this e-mail by error, please > notify > the sender immediately by return e-mail, and delete this message. > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or > its > employees shall not be liable for the incorrect or incomplete > transmission > of this e-mail or any attachments, nor responsible for any delay in > receipt. > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch > Airlines) is registered in Amstelveen, The Netherlands, with registered > number 33014286 > ******************************************************** > > > > U.S. BANCORP made the following annotations > --------------------------------------------------------------------- > Electronic Privacy Notice. This e-mail, and any attachments, contains > information that is, or may be, covered by electronic communications > privacy laws, and is also confidential and proprietary in nature. If you > are not the intended recipient, please be advised that you are legally > prohibited from retaining, using, copying, distributing, or otherwise > disclosing this information in any manner. Instead, please reply to the > sender that you have received this communication in error, and then > immediately delete it. Thank you in advance for your cooperation. > > > > --------------------------------------------------------------------- > ******************************************************** > For information, services and offers, please visit our web site: > http://www.klm.com. This e-mail and any attachment may contain > confidential and privileged material intended for the addressee only. If you > are not the addressee, you are notified that no part of the e-mail or any > attachment may be disclosed, copied or distributed, and that any other > action related to this e-mail or attachment is strictly prohibited, and may > be unlawful. If you have received this e-mail by error, please notify the > sender immediately by return e-mail, and delete this message. > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its > employees shall not be liable for the incorrect or incomplete transmission > of this e-mail or any attachments, nor responsible for any delay in receipt. > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch > Airlines) is registered in Amstelveen, The Netherlands, with registered > number 33014286 > ******************************************************** > > ******************************************************** For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286 ******************************************************** ----------------------------------------- The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message. ******************************************************** For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286 ******************************************************** ******************************************************** For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286 ********************************************************
