Hi All,
I'm performing a restore of some users data from their production
environment into their test environment with the aim of testing their
restore processes and also refreshing their test environment with some
current data.
However, I'm coming up against a puzzling issue resulting in 'Ran out of
disk space trying to Retrieve xyz' that I'm wondering if anyone else may
have seen, or may have some pointers that the sysadmins or myself might
be able to take a look at.
Platform: Solaris (source server = Solaris 8, target server = Solaris
10)
TSM Clients: source server = 5.1.6.2, target server = 5.3.2.2
FS: VxFS
The source filesystems are *very* full indeed, and are backed up by
separate 'dsmc archive' operations for each individual Oracle database
file contained in the filesystem (which is less than ideal, I know -
unfortunately I'm not here to re-write their backup and restore scripts,
just to test them). For example, a source filesystem on the production
server looks like this:
source $ df -k /oradata/SEN/dsk27
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/oradata_ppds_dg/oradatavol25
2498560 2491768 6744 100%
/oradata/SEN/dsk27
$
$ fstyp -v /dev/vx/dsk/oradata_ppds_dg/oradatavol25
vxfs
magic a501fcf5 version 4 ctime Thu Jan 18 00:16:59 2001
logstart 0 logend 0
bsize 8192 size 312320 dsize 312320 ninode 0 nau 0
defiextsize 0 ilbsize 0 immedlen 96 ndaddr 10
<...>
$ ls -la
total 4980884
drwxrwxr-x 3 sen_dba dba 8192 Jul 2 2001 .
drwxr-xr-x 64 oracle dba 1536 Mar 25 2002 ..
-rw-r--r-- 1 oracle oinstall 134225920 Jun 26 15:11 .smo_data_01.dbf
-rw-r--r-- 1 oracle oinstall 402661376 Jun 26 15:11
.smo_data_week54_01.dbf
-rw-r--r-- 1 oracle oinstall 402661376 Jun 26 15:11
.smo_data_week55_01.dbf
-rw-r--r-- 1 oracle oinstall 402661376 Jun 26 15:11
.smo_data_week56_01.dbf
-rw-r--r-- 1 oracle oinstall 402661376 Jun 26 15:11
.smo_data_week57_01.dbf
-rw-r--r-- 1 oracle oinstall 402661376 Jun 26 15:11
.smo_data_week58_01.dbf
-rw-r--r-- 1 oracle oinstall 402661376 Jun 26 14:39
.smo_data_week59_01.dbf
drwxr-xr-x 2 root root 96 May 3 2001 lost+found
lrwxrwxrwx 1 oracle oinstall 28 Jan 18 2001 smo_data_01.dbf ->
.smo_data_01.dbf::cdev:vxfs:
lrwxrwxrwx 1 oracle oinstall 35 Jun 29 2001
smo_data_week54_01.dbf -> .smo_data_week54_01.dbf::cdev:vxfs:
lrwxrwxrwx 1 oracle oinstall 35 Jun 29 2001
smo_data_week55_01.dbf -> .smo_data_week55_01.dbf::cdev:vxfs:
lrwxrwxrwx 1 oracle oinstall 35 Jun 29 2001
smo_data_week56_01.dbf -> .smo_data_week56_01.dbf::cdev:vxfs:
lrwxrwxrwx 1 oracle oinstall 35 Jun 29 2001
smo_data_week57_01.dbf -> .smo_data_week57_01.dbf::cdev:vxfs:
lrwxrwxrwx 1 oracle oinstall 35 Jun 29 2001
smo_data_week58_01.dbf -> .smo_data_week58_01.dbf::cdev:vxfs:
lrwxrwxrwx 1 oracle oinstall 35 Jul 2 2001
smo_data_week59_01.dbf -> .smo_data_week59_01.dbf::cdev:vxfs:
$
The same filesystem on their target restore server looks like this:
target # df -k /oradata/SEN/dsk27
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/oradata_ipds_dg/oradatavol25
2499584 2499584 0 100%
/oradata/SEN/dsk27
#
# fstyp -v /dev/vx/dsk/oradata_ipds_dg/oradatavol25
vxfs
magic a501fcf5 version 6 ctime Thu Jun 21 14:16:32 2007
logstart 0 logend 0
bsize 8192 size 312448 dsize 0 ninode 312448 nau 0
defiextsize 0 ilbsize 0 immedlen 96 ndaddr 10
<...>
# ls -la
total 4956866
drwxr-xr-x 3 root root 8192 Jun 25 02:49 .
drwxr-x--- 64 root root 1024 Jan 30 09:41 ..
-rw-r--r-- 1 1219 612 134225920 Jun 16 01:26 .smo_data_01.dbf
-rw-r--r-- 1 1219 612 402661376 Jun 16 01:26
.smo_data_week54_01.dbf
-rw-r--r-- 1 1219 612 402661376 Jun 16 01:26
.smo_data_week55_01.dbf
-rw-r--r-- 1 1219 612 402661376 Jun 16 01:26
.smo_data_week56_01.dbf
-rw-r--r-- 1 1219 612 402661376 Jun 16 01:26
.smo_data_week57_01.dbf
-rw-r----- 1 root root 390365184 Jun 25 02:49
.smo_data_week58_01.dbf <<< failed whilst restoring this, note file size
-rw-r--r-- 1 1219 612 402661376 Jun 16 01:28
.smo_data_week59_01.dbf
drwxr-xr-x 2 root root 96 Jun 21 14:16 lost+found
#
(this just after the retrieve failure - hence 0 kbytes available in df
-k and the 'incomplete' file .smo_data_week58_01.dbf in the target, and
the links hadn't been created yet)
So, the target filesystem (which was completely empty prior to the
restore attempt), is actually marginally larger than the source
filesystem, yet a restore of the objects still failed with an out of
space error (on the last file to be restored) as per the below:
Processing file /oradata/SEN/dsk27/.smo_data_week58_01.dbf
dsmc retrieve -virtualnode=xxx -server=yyy_san -replace=no -password=zzz
-DESC=ORA_HOTBCV_SEN_070616.0100
/oradata/SEN/dsk27/.smo_data_week58_01.dbf
IBM Tivoli Storage Manager
Command Line Backup/Archive Client Interface
Client Version 5, Release 3, Level 2.2
Client date/time: 25-06-2007 02:49:09
(c) Copyright by IBM Corporation and other(s) 1990, 2005. All Rights
Reserved.
Retrieve function invoked.
Node Name: xxx
Session established with server yyy: Solaris 8/9
Server Version 5, Release 3, Level 2.4
Data compression forced off by the server
Server date/time: 25-06-2007 04:03:04 Last access: 25-06-2007
04:02:14
** Interrupted **
ANS1114I Waiting for mount of offline media.
ANS4009E Error processing '/oradata/SEN/dsk27/.smo_data_week58_01.dbf':
disk full condition
--- User Action is Required ---
Ran out of disk space trying to Retrieve
'/oradata/SEN/dsk27/.smo_data_week58_01.dbf'
Select an appropriate action
1. Retry this object
2. Skip this object
A. Abort this operation
Action [1,2,A] :
TSM thinks it has the following objects to retrieve:
target # dsmc query archive -virtualnode=xxx -server=yyy_san
-password=zzz -DESC="ORA_HOTBCV_SEN_070616.0100" /oradata/SEN/dsk27/
IBM Tivoli Storage Manager
Command Line Backup/Archive Client Interface
Client Version 5, Release 3, Level 2.2
Client date/time: 26-06-2007 08:36:17
(c) Copyright by IBM Corporation and other(s) 1990, 2005. All Rights
Reserved.
Node Name: xxx
Session established with server yyy: Solaris 8/9
Server Version 5, Release 3, Level 2.4
Data compression forced off by the server
Server date/time: 26-06-2007 09:50:13 Last access: 26-06-2007
09:40:19
Size Archive Date - Time File - Expires on -
Description
---- -------------------
-------------------------------
8,192 B 16-06-2007 05:51:23 /oradata/SEN/dsk27/ 16-07-2007
ORA_HOTBCV_SEN_070616.0100
134,225,920 B 16-06-2007 05:53:49
/oradata/SEN/dsk27/.smo_data_01.dbf 16-07-2007
ORA_HOTBCV_SEN_070616.0100
402,661,376 B 16-06-2007 05:51:23
/oradata/SEN/dsk27/.smo_data_week54_01.dbf 16-07-2007
ORA_HOTBCV_SEN_070616.0100
402,661,376 B 16-06-2007 05:52:07
/oradata/SEN/dsk27/.smo_data_week55_01.dbf 16-07-2007
ORA_HOTBCV_SEN_070616.0100
402,661,376 B 16-06-2007 05:54:23
/oradata/SEN/dsk27/.smo_data_week56_01.dbf 16-07-2007
ORA_HOTBCV_SEN_070616.0100
402,661,376 B 16-06-2007 05:52:56
/oradata/SEN/dsk27/.smo_data_week57_01.dbf 16-07-2007
ORA_HOTBCV_SEN_070616.0100
402,661,376 B 16-06-2007 05:53:42
/oradata/SEN/dsk27/.smo_data_week58_01.dbf 16-07-2007
ORA_HOTBCV_SEN_070616.0100
402,661,376 B 16-06-2007 13:56:14
/oradata/SEN/dsk27/.smo_data_week59_01.dbf 16-07-2007
ORA_HOTBCV_SEN_070616.0100
#
I'm out of ideas from a TSM point of view at the moment - as far I know
there should be no overhead when TSM BA Client performs a retrieve
operation - there is no compression or encryption here from a TSM point
of view either.
Does anyone have any ideas where I should be looking next?
Many thanks,
David McClelland
London, UK
This email was sent to you by Reuters, the global news and information company.
To find out more about Reuters visit www.about.reuters.com
Any views expressed in this message are those of the individual sender,
except where the sender specifically states them to be the views of Reuters
Limited.
Reuters Limited is part of the Reuters Group of companies, of which Reuters
Group PLC is the ultimate parent company.
Reuters Group PLC - Registered office address: The Reuters Building, South
Colonnade, Canary Wharf, London E14 5EP, United Kingdom
Registered No: 3296375
Registered in England and Wales
