Hi Sasa Delete batch was set to Low priority and it has helped us to address the issue. Still working with dev team for apar details.
Setopt repldeletepriority Low Regards Sarav +65 9857 8665 > On 8 Dec 2020, at 5:46 AM, Sasa Drnjevic <sasa.drnje...@srce.hr> wrote: > > Hi Sarav. > > Thanks for the info. > > Could you please just send some link or more info on the setting or a fix > version if it is a fix. > > Tnx - Regards, > > -- > Sasa Drnjevic > www.srce.unizg.hr/en/ > > > > >> On 7.12.2020. 20:02, Saravanan Palanisamy wrote: >> Team >> Just would like to update status for replication issue reported earlier. IBM >> development team managed to find the root cause and provided solution for >> intermittent replication hung. We have applied those recommended settings >> and didn’t see any hung for the past 3 weeks. >> Regards >> Sarav >> +65 9857 8665 >>>> On 24 Sep 2020, at 12:12 AM, Sasa Drnjevic <sasa.drnje...@srce.hr> wrote: >>> >>> Hi Sarav and all, >>> >>> I forgot to inform you earlier, but I have solved the problem with HUNG >>> or more precisely EXTREMELY slow replication. >>> >>> Linux nodees were ALL replicating as before the upgrade, without any >>> problems. >>> >>> The main culprit was Windows Systemstate. Ordinary NTFS filespaces >>> replicated fine. >>> >>> But when I have replicated all of the windows nodes manually one-by-one >>> (not as a nodegroup) the second and all successive runs were normal as >>> before. So, now everything runs fine as before the upgrade to >>> v8.1.10.100 (on RHEL 7.8). >>> >>> All windows would take much more to replicate when run for the first >>> time after the upgrade, some of them took 8 hours instead of less than >>> 10 minutes. >>> >>> IBM support blamed all of it on canceled replication because they have >>> found some msgs like: >>> >>> ANR1650W The server detected partially replicated data >>> from a previous replication operation. This might result >>> in extended processing time for process 2 while the >>> server is replicating node nodename... >>> >>> >>> Yes, I have canceled some, but maybe only 5-6 windows nodes, but all of >>> the 40+ windows nodes took ages to replicate for the first time after >>> upgrade. >>> >>> Most of the windows nodes used some readonly tapes (scanning only?) with >>> data more than 2 months old which was surely replicated long time ago. >>> >>> Some Linux nodes (out of 180+) were canceled too, but later replicated >>> with normal throughput. >>> >>> So, be careful :-) >>> >>> Regards, >>> >>> -- >>> Sasa Drnjevic >>> www.srce.unizg.hr/en/ >>> >>> >>> >>> >>> >>> >>> >>> >>>> On 9.9.2020. 1:08, Saravanan Palanisamy wrote: >>>> Hi Sasa >>>> >>>> Thanks for the update. We are still waiting for updates from dev team. >>>> >>>> Regards >>>> Sarav >>>> +65 9857 8665 >>>> >>>> >>>>>> On 9 Sep 2020, at 4:55 AM, Sasa Drnjevic <sasa.drnje...@srce.hr> wrote: >>>>> >>>>> On 2.9.2020. 1:30, Saravanan Palanisamy wrote: >>>>>> Hi Eric >>>>>> >>>>>> We have applied efix to address APAR IT33437 instead of patch >>>>>> v8.1.10.100 and we haven’t seen any server crash after >efix. However >>>>>> our intermittent replication issue case still in progress. >>>>> >>>>> >>>>> Hi Sarav & all. >>>>> >>>>> Last week I have upgraded my target server to v8.1.10.100, and node >>>>> replication was going without any problems from source v8.1.1.000 to >>>>> target v8.1.10.100. Both servers are RHEL 7.8. >>>>> >>>>> So, this morning I took my chances and upgraded my source server to >>>>> v8.1.10.100. The upgrade went fine. >>>>> >>>>> But, replication hangs :-( >>>>> >>>>> I have opened a case with IBM support and did a first step supplying >>>>> some logs. >>>>> >>>>> But, while trying to find out what is causing the replication to hang I >>>>> did some 30 manual single node replication jobs and ALL of the Linux >>>>> nodes replicated with SUCCESS. >>>>> >>>>> I have node groups which are replicated with schedules, and of course >>>>> some are mixed with Linux and Windows nodes. >>>>> >>>>> So, when a windows node hangs it hangs the whole node group. >>>>> >>>>> And it seems that the main culprit is the Windows Systemstate. >>>>> >>>>> Would bet on that, and I mentioned that to IBM support. >>>>> >>>>> We'll see tomorrow. >>>>> >>>>> Hope it helps. >>>>> >>>>> Rgds, >>>>> >>>>> -- >>>>> Sasa Drnjevic >>>>> www.srce.unizg.hr/en/ >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> Regards >>>>>> Sarav >>>>>> +65 9857 8665 >>>>>> >>>>>> >>>>>>>> On 1 Sep 2020, at 10:38 PM, Loon, Eric van (ITOP NS) - KLM >>>>>>>> <eric-van.l...@klm.com> wrote: >>>>>>> >>>>>>> Hi Sarav, >>>>>>> >>>>>>> On page https://www.ibm.com/support/pages/node/6212043#8110100 I do >>>>>>> find APAR IT33437 listed. >>>>>>> >>>>>>> Kind regards, >>>>>>> Eric van Loon >>>>>>> Air France/KLM Storage & Backup >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of >>>>>>> Saravanan Palanisamy >>>>>>> Sent: maandag 31 augustus 2020 23:10 >>>>>>> To: ADSM-L@VM.MARIST.EDU >>>>>>> Subject: Re: Replication Hung after upgrade to v8.1.10 >>>>>>> >>>>>>> Hi Eric >>>>>>> >>>>>>> Thanks. I have gone through v8.1.10.100 APAR details but I don’t find >>>>>>> my issue here. I am still waiting for IBM on my case. >>>>>>> >>>>>>> Regards >>>>>>> Sarav >>>>>>> +65 9857 8665 >>>>>>> >>>>>>> >>>>>>>> On 31 Aug 2020, at 3:00 PM, Loon, Eric van (ITOP NS) - KLM >>>>>>>> <eric-van.l...@klm.com> wrote: >>>>>>>> >>>>>>>> Hi Sarav, >>>>>>>> >>>>>>>> You probably noticed it already, but IBM released 8.1.10.100 >>>>>>>> containing a fix for your (and several other) issues. >>>>>>>> >>>>>>>> Kind regards, >>>>>>>> Eric van Loon >>>>>>>> Air France/KLM Storage & Backup >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of >>>>>>>> Saravanan Palanisamy >>>>>>>> Sent: donderdag 20 augustus 2020 17:42 >>>>>>>> To: ADSM-L@VM.MARIST.EDU >>>>>>>> Subject: Re: Replication Hung after upgrade to v8.1.10 >>>>>>>> >>>>>>>> IT33437 >>>>>>>> >>>>>>>> Regards >>>>>>>> Sarav >>>>>>>> +65 9857 8665 >>>>>>>> >>>>>>>> >>>>>>>>>> On 20 Aug 2020, at 11:05 PM, Loon, Eric van (ITOP NS) - KLM >>>>>>>>>> <eric-van.l...@klm.com> wrote: >>>>>>>>> >>>>>>>>> Hi Sarav, >>>>>>>>> >>>>>>>>> You stated " There is also one more issue on v8.1.10 where server >>>>>>>>> will crash abruptly if you use directory container. It appears to be >>>>>>>>> known issue and we have got efix to address it". I find this very >>>>>>>>> alarming, because I'm about to start production on a new server >>>>>>>>> running this level. Do you have an APAR number? >>>>>>>>> >>>>>>>>> Kind regards, >>>>>>>>> Eric van Loon >>>>>>>>> Air France/KLM Storage & Backup >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of >>>>>>>>> Saravanan Palanisamy >>>>>>>>> Sent: donderdag 20 augustus 2020 16:58 >>>>>>>>> To: ADSM-L@VM.MARIST.EDU >>>>>>>>> Subject: Re: Replication Hung after upgrade to v8.1.10 >>>>>>>>> >>>>>>>>> Thanks Erwann for the update. >>>>>>>>> >>>>>>>>> We are still facing intermittent hang and no progress on our case. If >>>>>>>>> it’s one server then we could say it’s environment issue but 3 >>>>>>>>> different pair and all are in different countries as well. We are >>>>>>>>> holding upgrade for rest of countries until find out root cause. >>>>>>>>> >>>>>>>>> There is also one more issue on v8.1.10 where server will crash >>>>>>>>> abruptly if you use directory container. It appears to be known issue >>>>>>>>> and we have got efix to address it. >>>>>>>>> >>>>>>>>> I will keep you posted if I hear any updates on my case. >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> Sarav >>>>>>>>> +65 9857 8665 >>>>>>>>> >>>>>>>>> >>>>>>>>>>> On 20 Aug 2020, at 10:10 PM, Erwann SIMON <erwann.si...@free.fr> >>>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I've just upgraded some servers to that level and I do not encounter >>>>>>>>>> this king of issue (right now...). >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best regards / Cordialement / مع تحياتي >>>>>>>>>> Erwann SIMON >>>>>>>>>> >>>>>>>>>> ----- Mail original ----- >>>>>>>>>> De: "Saravanan Palanisamy" <evergreen.sa...@gmail.com> >>>>>>>>>> À: ADSM-L@VM.MARIST.EDU >>>>>>>>>> Envoyé: Mardi 21 Juillet 2020 00:37:41 >>>>>>>>>> Objet: [ADSM-L] Replication Hung after upgrade to v8.1.10 >>>>>>>>>> >>>>>>>>>> Dear Team >>>>>>>>>> >>>>>>>>>> Is there anyone facing intermittent replication hung issue after >>>>>>>>>> upgrade to v8.1.10? >>>>>>>>>> >>>>>>>>>> We have upgraded TSM server from 8.1.9.300 to 8.1.10 and replication >>>>>>>>>> process is hung intermittently on all 3 servers. Just want to see >>>>>>>>>> whether anyone else impacted. >>>>>>>>>> >>>>>>>>>> Once restart TSM instance then replication works but again >>>>>>>>>> replication node process will get hung. But we don’t see any hung >>>>>>>>>> for protect stg and it works fine. >>>>>>>>>> >>>>>>>>>> We have raised case with IBM and working on it. >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> Sarav >>>>>>>>>> +65 9857 8665 >>>>>>>>> ******************************************************** >>>>>>>>> For information, services and offers, please visit our web site: >>>>>>>>> http://www.klm.com. This e-mail and any attachment may contain >>>>>>>>> confidential and privileged material intended for the addressee only. >>>>>>>>> If you are not the addressee, you are notified that no part of the >>>>>>>>> e-mail or any attachment may be disclosed, copied or distributed, and >>>>>>>>> that any other action related to this e-mail or attachment is >>>>>>>>> strictly prohibited, and may be unlawful. If you have received this >>>>>>>>> e-mail by error, please notify the sender immediately by return >>>>>>>>> e-mail, and delete this message. >>>>>>>>> >>>>>>>>> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or >>>>>>>>> its employees shall not be liable for the incorrect or incomplete >>>>>>>>> transmission of this e-mail or any attachments, nor responsible for >>>>>>>>> any delay in receipt. >>>>>>>>> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal >>>>>>>>> Dutch Airlines) is registered in Amstelveen, The Netherlands, with >>>>>>>>> registered number 33014286 >>>>>>>>> ******************************************************** >>>>>>>> ******************************************************** >>>>>>>> For information, services and offers, please visit our web site: >>>>>>>> http://www.klm.com. This e-mail and any attachment may contain >>>>>>>> confidential and privileged material intended for the addressee only. >>>>>>>> If you are not the addressee, you are notified that no part of the >>>>>>>> e-mail or any attachment may be disclosed, copied or distributed, and >>>>>>>> that any other action related to this e-mail or attachment is strictly >>>>>>>> prohibited, and may be unlawful. If you have received this e-mail by >>>>>>>> error, please notify the sender immediately by return e-mail, and >>>>>>>> delete this message. >>>>>>>> >>>>>>>> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or >>>>>>>> its employees shall not be liable for the incorrect or incomplete >>>>>>>> transmission of this e-mail or any attachments, nor responsible for >>>>>>>> any delay in receipt. >>>>>>>> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal >>>>>>>> Dutch Airlines) is registered in Amstelveen, The Netherlands, with >>>>>>>> registered number 33014286 >>>>>>>> ******************************************************** >>>>>>> ******************************************************** >>>>>>> For information, services and offers, please visit our web site: >>>>>>> http://www.klm.com. This e-mail and any attachment may contain >>>>>>> confidential and privileged material intended for the addressee only. >>>>>>> If you are not the addressee, you are notified that no part of the >>>>>>> e-mail or any attachment may be disclosed, copied or distributed, and >>>>>>> that any other action related to this e-mail or attachment is strictly >>>>>>> prohibited, and may be unlawful. If you have received this e-mail by >>>>>>> error, please notify the sender immediately by return e-mail, and >>>>>>> delete this message. >>>>>>> >>>>>>> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or >>>>>>> its employees shall not be liable for the incorrect or incomplete >>>>>>> transmission of this e-mail or any attachments, nor responsible for any >>>>>>> delay in receipt. >>>>>>> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch >>>>>>> Airlines) is registered in Amstelveen, The Netherlands, with registered >>>>>>> number 33014286 >>>>>>> ********************************************************
