SP client use NetApp API to detect file changes between 2(two) snapshots. After that client have the list of changed files and backup it from mounted volume. I believe that it is unnecessary to copy the snapshot itself because it's pointless. Efim
> 20 сент. 2018 г., в 11:29, Tommaso Bollini <t.boll...@vargroup.it> написал(а): > > @Efim > the .snapshot (or ~snapshot) folders are NetApp construct recognized by the > BA Spectrum agent api: > https://www.ibm.com/support/knowledgecenter/en/SSEQVQ_8.1.2/client/r_opt_snapdiff.html > The BA Spectrum agent will copy all files listed in the NetApp snapshot image. > The log error simply reports the point at which this backup job has in > exception echoing the full path of what BA API read from the snapshot image. > > I hope I was helpful. > Greetings. > > -----Messaggio originale----- > Da: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] Per conto di Efim > Inviato: mercoledì 19 settembre 2018 18:39 > A: ADSM-L@VM.MARIST.EDU > Oggetto: Re: [ADSM-L] Netapp snapdiff issues having TSM client on a Linux > machine, making use of NFS mounts > > Arnaud, > > Can you explain why you are copying the .snapshot folder during incremental > filesystem backup with snapdiff? > It contains snapshots created by netapp and SP for this FS. > I believe that it should be excluded from copying in your backup job. > > Efim > > >> 19 сент. 2018 г., в 18:28, PAC Brion Arnaud <arnaud.br...@panalpina.com> >> написал(а): >> >> Hi Robert, >> >> Thanks a lot for appreciated feedback, unfortunately even when using TSM own >> snapshots the problem still exists : >> >> root@xxxxxx:~ # dsmc i /Airwarder -snapdiff -diffsnapshot=create >> -snapdiffhttps -optfile=$OPTPATH >/tmp/snapshot_Airwarder.log >> >> root@ xxxxxx:~ # more /tmp/snapshot_Airwarder.log IBM Spectrum Protect >> Command Line Backup-Archive Client Interface Client Version 8, >> Release 1, Level 4.0 Client date/time: 09/19/2018 17:17:41 >> (c) Copyright by IBM Corporation and other(s) 1990, 2017. All Rights >> Reserved. >> >> Node Name: NETAPP_CH1_NFS >> Session established with server xxxxxx: Linux/ppc64le Server Version >> 8, Release 1, Level 4.000 Server date/time: 09/19/2018 17:17:42 Last >> access: 09/18/2018 12:12:17 >> >> >> Incremental by snapshot difference of volume '/Airwarder' >> >> Connected to NetApp Storage Virtual Machine >> Management Filer Host/IP Address : xxxxxx >> Filer Version Information : NetApp Release 9.3P3: Wed Apr 04 >> 13:07:06 >> UTC 2018 >> Storage VM Name : ch1ns100 >> Storage VM Host/IP Address : xxxxxx >> Storage VM Volume Style : Flex >> Login User : tsm-backup-user >> Transport : HTTPS >> >> Performing a Full Incremental Backup of volume '/Airwarder' >> Creating Base Snapshot. >> Using Base Snapshot 'TSM_CH135BA2689781857_AIRWARDER' with timestamp >> 09/19/2018 >> 17:17:43 >> ANS4007E Error processing >> '/Airwarder/.snapshot/TSM_CH135BA2689781857_AIRWARDER/ >> acrimport_ref/ch': access to the object is denied ANS4007E Error >> processing '/Airwarder/.snapshot/TSM_CH135BA2689781857_AIRWARDER/ >> acrimport_test/ch': access to the object is denied >> ANS1898I ***** Processed 500 files ***** >> ANS4007E Error processing >> '/Airwarder/.snapshot/TSM_CH135BA2689781857_AIRWARDER/ >> >> I'm lost here ... >> >> Cheers. >> >> Arnaud >> >> ********************************************************************** >> ********************************************************** >> Backup and Recovery Systems Administrator Panalpina Management Ltd., >> Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 >> Basel/CH >> Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 >> Direct: +41 (61) 226 19 78 >> e-mail: arnaud.br...@panalpina.com >> This electronic message transmission contains information from Panalpina and >> is confidential or privileged. >> This information is intended only for the person (s) named above. If >> you are not the intended recipient, any disclosure, copying, distribution or >> use or any other action based on the contents of this information is >> strictly prohibited. >> >> If you receive this electronic transmission in error, please notify the >> sender by e-mail, telephone or fax at the numbers listed above. Thank you. >> ********************************************************************** >> ********************************************************** >> www.panalpina.com >> >> >> >> -----Original Message----- >> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf >> Of Robert Talda >> Sent: Tuesday, September 18, 2018 10:48 PM >> To: ADSM-L@VM.MARIST.EDU >> Subject: Re: Netapp snapdiff issues having TSM client on a Linux >> machine, making use of NFS mounts >> >> Brion: >> Here at Cornell, we have a TSM client on a Linux system backing up NFS >> shares via NFS mounts and snapdiff differentials - and have been doing so >> for years. >> >> First, the gory details: >> - For TSM >> TSM Client Version 7, Release 1, Level 6.0 (*red faced, thought I had >> upgraded to SP 8.1.2.0 months ago*) Running on CentOS Linux release >> 7.5.1804 (kernel 3.10.0-862.2.3.el7.x86_64) Accessing TSM Server for >> Linux Version 7, Release 1, Level 7.200 >> - For NetApp >> Filer Version Information : NetApp Release 9.3P6 >> >> We aren’t seeing your issue, and I suspect I know why. We are telling the >> TSM client to create its own snapshots - and not use the snapshots being >> created hourly (or daily or weekly even) by the team managing the Netapp - >> said snapshots being equivalent to the ones you are trying to take advantage >> of in your backups. >> >> That is, the relevant part of the daily backup schedule looks like: >> Schedule Name: DAILY.INCR.NFS.SNAPDIFF >> Description: Daily SnapDiff Incremental >> Action: Incremental >> Subaction: >> Options: -snapdiff -diffsnapshot=create >> -snapdiffhttps >> >> >> During a backup, then, the following type of information is written to the >> dsmsched.log: >> 09/13/2018 00:38:08 Incremental by snapshot difference of volume >> '/nas-backup/vol/cit-pea1-test' >> 09/13/2018 00:38:10 ANS2328I Using Snapshot Differential Change Log. >> 09/13/2018 00:38:10 >> Connected to NetApp Storage Virtual Machine >> Management Filer Host/IP Address : ccc-cdot01.cit.cornell.edu >> Filer Version Information : NetApp Release 9.3P6: Thu Jun 14 >> 20:21:25 UTC 2018 >> Storage VM Name : cit-sfshared05 >> Storage VM Host/IP Address : cit-sfshared05-backup.files.cornell.edu >> Storage VM Volume Style : Flex >> Login User : tsmbackup >> Transport : HTTPS >> >> 09/13/2018 00:38:10 Performing a Snapshot Differential Backup of volume >> '/nas-backup/vol/cit-pea1-test' >> 09/13/2018 00:38:10 Creating Diff Snapshot. >> 09/13/2018 00:38:10 Using Base Snapshot >> 'TSM_SANT5B9759FC1AB6_CIT_PEA1_TEST_VOL' with timestamp 09/11/2018 >> 06:00:28 >> 09/13/2018 00:38:10 Using Diff Snapshot >> 'TSM_SANT5B99E9B24F814_CIT_PEA1_TEST_VOL' with timestamp 09/13/2018 >> 04:38:10 >> >> Then, on the next backup (48 hours later thanks to a large amount of data to >> backup on 9/13), the Diff Snapshot becomes the base: >> 09/15/2018 01:29:22 ANS2328I Using Snapshot Differential Change Log. >> 09/15/2018 01:29:22 >> Connected to NetApp Storage Virtual Machine >> Management Filer Host/IP Address : ccc-cdot01.cit.cornell.edu >> Filer Version Information : NetApp Release 9.3P6: Thu Jun 14 >> 20:21:25 UTC 2018 >> Storage VM Name : cit-sfshared05 >> Storage VM Host/IP Address : cit-sfshared05-backup.files.cornell.edu >> Storage VM Volume Style : Flex >> Login User : tsmbackup >> Transport : HTTPS >> >> 09/15/2018 01:29:22 Performing a Snapshot Differential Backup of volume >> '/nas-backup/vol/cit-pea1-test' >> 09/15/2018 01:29:22 Creating Diff Snapshot. >> 09/15/2018 01:29:22 Using Base Snapshot >> 'TSM_SANT5B99E9B24F814_CIT_PEA1_TEST_VOL' with timestamp 09/13/2018 >> 04:38:10 >> 09/15/2018 01:29:22 Using Diff Snapshot >> 'TSM_SANT5B9C98B1A15B2_CIT_PEA1_TEST_VOL' with timestamp 09/15/2018 >> 05:29:21 >> >> I think that consistency is what is missing in your approach - I suspect >> that there are intervening snapshots with information about files that have >> been deleted or changed that are not part of what the TSM client is seeing. >> >> I will add that our NetApp team is very happy with this approach. The TSM >> client has been very efficient in its management of the snapshots, and in >> those rare cases when it doesn’t, the snapshots are easy to identify and >> clean up. >> >> FWIW, >> Bob >> >> >> Robert Talda >> EZ-Backup Systems Engineer >> Cornell University >> +1 607-255-8280 >> r...@cornell.edu >> >> >>> On Sep 18, 2018, at 8:35 AM, PAC Brion Arnaud <arnaud.br...@panalpina.com> >>> wrote: >>> >>> Hi Tommaso, >>> >>> The backup command I'm using (dsmc i /Airwarder -snapdiff -snapdiffhttps >>> -diffsnapshotname="daily.*" -diffsnapshot=latest -useexistingbase >>> -optfile=dsm_netapp.opt) includes the "useexistingbase" statement. >>> Based on my understanding of the documentation, this means that the S.P. >>> client will not trigger the creation of a new snapshot on the Netapp >>> device, but instead, make use of a snapshot that had been created >>> previously by some internal mechanism of the Netapp. >>> Therefore, it seems very unlikely that any process still has a grip on any >>> of the files being part of this snapshot ... >>> My humble opinion only of course, I'm waiting on others fellow members of >>> this list feedback ! >>> >>> In addition to this, I would like to avoid the use of any exclude statement >>> during the backup : I doubt that any of my customers would be very happy to >>> hear that I'm not able to restore some of his precious files because the >>> Spectrum Protect client had limitations ... Such issues never aroused with >>> NDMP-based backups ! >>> >>> Cheers. >>> >>> Arnaud >>> >>> >>> -----Original Message----- >>> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf >>> Of Tommaso Bollini >>> Sent: Tuesday, September 18, 2018 12:29 PM >>> To: ADSM-L@VM.MARIST.EDU >>> Subject: R: [ADSM-L] Netapp snapdiff issues having TSM client on a >>> Linux machine, making use of NFS mounts >>> >>> Hello Arnaud, maybe it would be: The NetApp snapshot indicates which files >>> are modified on the LUN (for which their backup is triggered); but the >>> Backup-Archive client cannot "take" them because they are locked by the >>> application. >>> Perhaps you can solve by inserting the appropiate rules in incl.excl file >>> to skip them. >>> >>> >>> -----Messaggio originale----- >>> Da: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] Per conto >>> di PAC Brion Arnaud >>> Inviato: martedì 18 settembre 2018 12:21 >>> A: ADSM-L@VM.MARIST.EDU >>> Oggetto: [ADSM-L] Netapp snapdiff issues having TSM client on a Linux >>> machine, making use of NFS mounts >>> >>> Hi All, >>> >>> Following to IBM's decision to withdraw the web GUI in the latest versions >>> of their clients (thus making NDMP hardly usable), as well as to the >>> renewal of our NAS infrastructure (now using Netapp), I'm trying to >>> implement netapp snapdiffs in our shop. >>> >>> So far we succeeded in defining necessary user and roles on the Netapp, as >>> well as to setup the TSM client in order to interact with our filer, but >>> I'm now facing an issue where some files cannot be backed up, apparently >>> due to access rights. >>> >>> Here an extract of our backup log : >>> >>> IBM Spectrum Protect >>> Command Line Backup-Archive Client Interface Client Version 8, >>> Release 1, Level 4.0 Client date/time: 09/18/2018 10:26:28 >>> (c) Copyright by IBM Corporation and other(s) 1990, 2017. All Rights >>> Reserved. >>> >>> Node Name: NETAPP_CH1_NFS >>> Session established with server XXXXXXX: Linux/ppc64le Server Version >>> 8, Release 1, Level 4.000 Server date/time: 09/18/2018 10:26:28 Last >>> access: 09/18/2018 10:00:37 >>> >>> >>> Incremental by snapshot difference of volume '/Airwarder' >>> >>> Connected to NetApp Storage Virtual Machine >>> Management Filer Host/IP Address : XXXXXXXX >>> Filer Version Information : NetApp Release 9.3P3: Wed Apr 04 >>> 13:07:06 UTC 2018 >>> Storage VM Name : xxxxxxx >>> Storage VM Host/IP Address : XXXXXX >>> Storage VM Volume Style : Flex >>> Login User : tsm-backup-user >>> Transport : HTTPS >>> >>> Performing a Full Incremental Backup of volume '/Airwarder' >>> Using Base Snapshot >>> 'vserverdr.0.7b5a59c4-656c-11e8-bb80-00a098d614c2.2018-09-18_102500' with >>> timestamp 09/18/2018 10:25:02 ANS4007E Error processing >>> '/Airwarder/.snapshot/vserverdr.0.7b5a59c4-656c-11e8-bb80-00a098d614c2.2018-09-18_102500/acrimport_ref/ch': >>> access to the object is denied ANS4007E Error processing >>> '/Airwarder/.snapshot/vserverdr.0.7b5a59c4-656c-11e8-bb80-00a098d614c2.2018-09-18_102500/acrimport_test/ch': >>> access to the object is denied >>> ANS1898I ***** Processed 500 files ***** >>> Directory--> 4,096 /Airwarder/ [Sent] >>> Directory--> 4,096 /Airwarder/.etc [Sent] >>> >>> (skipped some lines) >>> >>> ANS1228E Sending of object '/Airwarder/acrimport_ref/config/log4j.conf' >>> failed. >>> ANS4007E Error processing '/Airwarder/acrimport_ref/config/log4j.conf': >>> access to the object is denied >>> Normal File--> 309 >>> /Airwarder/acrimport_ref/config/sapConnector.cfg ** Unsuccessful ** >>> ANS1228E Sending of object >>> '/Airwarder/acrimport_ref/config/sapConnector.cfg' failed. >>> ANS4007E Error processing >>> '/Airwarder/acrimport_ref/config/sapConnector.cfg': access to the object is >>> denied >>> Normal File--> 330 >>> /Airwarder/acrimport_ref/config/sapConnector.cfg.bak ** Unsuccessful ** >>> ANS1228E Sending of object >>> '/Airwarder/acrimport_ref/config/sapConnector.cfg.bak' failed. >>> ANS4007E Error processing >>> '/Airwarder/acrimport_ref/config/sapConnector.cfg.bak': access to the >>> object is denied >>> >>> (skipped some lines) >>> >>> ANS1802E Incremental backup of '/Airwarder' finished with 90 >>> failure(s) >>> >>> >>> Total number of objects inspected: 5,881 >>> Total number of objects backed up: 5,791 >>> Total number of objects updated: 0 >>> Total number of objects rebound: 0 >>> Total number of objects deleted: 0 >>> Total number of objects expired: 0 >>> Total number of objects failed: 90 >>> Total number of objects encrypted: 0 >>> Total number of objects grew: 0 >>> Total number of retries: 15 >>> Total number of bytes inspected: 1.27 GB >>> Total number of bytes transferred: 1.37 GB >>> Data transfer time: 18.61 sec >>> Network data transfer rate: 77,367.92 KB/sec >>> Aggregate data transfer rate: 36,807.03 KB/sec >>> Objects compressed by: 0% >>> Total data reduction ratio: 0.00% >>> Elapsed processing time: 00:00:39 >>> >>> >>> Here, a closer look at some of the files making problems : >>> >>> root@xxxxxx:~ # ls -al >>> /Airwarder/acrimport_ref/config/sapConnector.cfg.bak >>> -rw-r----- 1 root root 330 May 6 2009 >>> /Airwarder/acrimport_ref/config/sapConnector.cfg.bak >>> >>> >>> root@xxxxxx:~ # ls -la >>> /Airwarder/.snapshot/vserverdr.0.7b5a59c4-656c-11e8-bb80-00a098d614c2 >>> .2018-09-18_121000/acrimport_ref/ch >>> ls: cannot open directory >>> /Airwarder/.snapshot/vserverdr.0.7b5a59c4-656c-11e8-bb80-00a098d614c2 >>> .2018-09-18_121000/acrimport_ref/ch: Permission denied >>> >>> And who I am: >>> >>> root@xxxxxx:~ # id >>> uid=0(root) gid=0(root) groups=0(root) >>> >>> Any idea on how to solve such issues ? >>> >>> Cheers. >>> >>> Arnaud >>> >>> ********************************************************************* >>> *********************************************************** >>> Backup and Recovery Systems Administrator Panalpina Management Ltd., >>> Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 >>> Basel/CH >>> Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 >>> Direct: +41 (61) 226 19 78 >>> e-mail: arnaud.br...@panalpina.com<mailto:arnaud.br...@panalpina.com> >>> This electronic message transmission contains information from Panalpina >>> and is confidential or privileged. >>> This information is intended only for the person (s) named above. If you >>> are not the intended recipient, any disclosure, copying, distribution or >>> use or any other action based on the contents of this information is >>> strictly prohibited. >>> >>> If you receive this electronic transmission in error, please notify the >>> sender by e-mail, telephone or fax at the numbers listed above. Thank you. >>> ********************************************************************* >>> *********************************************************** >>
