You are right, the drive can be subdivided in that way if there
is a logical structure to the directories.
In our hardest cases, higher level directory structure changes
frequently (and then data is missed)or the directory structure so
lopsided that it is difficult to divide it evenly. In these cases, we
still don't have a way to get it backed up in a reasonable time. <sigh>.
It's Friday...
Ben
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of
David W Litten
Sent: Friday, June 17, 2005 3:42 PM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: 15,000,000 + files on one directory backup
We were using it to back up 5.4 million files but they were in fact, as
you've said, on multiple drives (actually unc connections). That is not
quite 15M. Would it still not work if the domain statements read: Domain
d:\data Domain d:\apps Domain d:\otherFiles in each of the dsm.opt
files?
I know that Tivoli's official stance on anything other than root drive
paths (d: c: e: etc) is that it is not supported. But there are
instances were certain TSM client versions still work with these deeper
path statements in the dsm.opt files. I'm using them with the TSM 5.3
client successfully.
Ben Bullock
<[EMAIL PROTECTED]
COM>
To
Sent by: "ADSM: ADSM-L@vm.marist.edu
Dist Stor
cc
Manager"
<[EMAIL PROTECTED]
Subject
.edu> Re: [ADSM-L] 15,000,000 + files
on
one directory backup
06/17/2005 05:26
PM
Please respond to
"ADSM: Dist Stor
Manager"
<[EMAIL PROTECTED]
.edu>
That methodology works when you have multiple drives, but when you have
one monster drive with millions of little files, this won't work.
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of
David W Litten
Sent: Friday, June 17, 2005 3:19 PM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: 15,000,000 + files on one directory backup
Egon,
The following is all based on Windows nodes. I'm not sure how it differs
for non Windows systems.
you can use the wizard in the TSM GUI to create the client acceptor and
scheduler service. You can also use it to create multiple scheduler
services. You need to give each instance a unique name. Each Client
Acceptor and Scheduler service pair would point to a different dsm.opt
file. (example: Acceptor1, Scheduler1, dsm1.opt; Acceptor2, Scheduler2,
dsm2.opt). Each dsm.opt file would have a DOMAIN line specifying what
that instance should back up (ex: dsm.opt might have Domain c:
systemobject; dsm2.opt might have Domain d:)
This worked for us for a while but then something happened where the
prompted backup would only start instance1 and not instance2 thru 4. Not
sure why but we found a workaround. It may work fine if you are using
schedmode=polled, which is the default. We were using schedmode=prompted
david
Egon Blouder
<[EMAIL PROTECTED]
NET>
To
Sent by: "ADSM: ADSM-L@vm.marist.edu
Dist Stor
cc
Manager"
<[EMAIL PROTECTED]
Subject
.edu> Re: [ADSM-L] 15,000,000 + files
on
one directory backup
06/17/2005 04:39
PM
Please respond to
"ADSM: Dist Stor
Manager"
<[EMAIL PROTECTED]
.edu>
Hi,
I would disagree on your point.
Having 15 million files on one nodes crash dsmsvc process due to memory
problems. Each file stored on the TSM servers creates a database entry
that needs about 300 bytes in memory during backup. That means that the
client has to handle about 300*15*10**6/10**3 ~ 4.5GB of database entry.
As on Windows 2000 each process can only have 2GB (or 3GB; TSM client
doesn't support PAE) dsmsvc stops working.
You have to use memoryefficent=yes. however that options delays the
backup process and I don't know if it ever uses
resourceutilization=10!??
I'm still looking on how to configure multiple TSM schedules which write
to a single node on the server. Any help?
Will Tivoli add PAE functionality to TSM client software?
Regards,
..--
TSM_User <[EMAIL PROTECTED]> wrote:
>Also, with my suggestion you don't need to have 10 different nodes. You
can have one node. With resource utilization of 10 I believe you will
have up to 4 producer and 4 consumer threads. It's been a while since I
saw the doc on that exact setting to thread count. If you want more you
can set it higher like 30. Just remember at some point the available
resources on the server will come into play so change that setting with
care.
>
>My main point was if you can get the data onto 10 logical drives then
>the
TSM backup will start seperate producer threads. When you have all
15,000,000 files on one drive you get one producer thread running on the
whole drive.
>
>K
>
>Egon Blouder <[EMAIL PROTECTED]> wrote:
>Hi Mark,
>
>how can I store any data as filespaces of one node(virtualnode) in that
case? I don't want to setup 10 different webagents for doing restores.
>
>
>Thanks
>..--
>
>"Stapleton, Mark" wrote:
>
>>You create 10 separate nodes, one for each mountpoint (drive). You can
>>then run 10 separate backup threads; each thread has (in theory)
>>1/10th as many files to filter through before a backup begins.
>>
>>It's done that way a lot in such cases.
>>
>>--
>>Mark Stapleton ([EMAIL PROTECTED])
>>IBM Certified Advanced Deployment Professional Tivoli Storage
>>Management Solutions 2005 IBM Certified Advanced Technical Expert
>>(CATE) AIX Office 262.521.5627
>>
>>
>>
>>>-----Original Message-----
>>>From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf
>>>Of Dearman, Richard
>>>Sent: Thursday, June 16, 2005 1:06 PM
>>>To: ADSM-L@VM.MARIST.EDU
>>>Subject: Re: 15,000,000 + files on one directory backup
>>>
>>>Yes, but the long backup time is because of the time it takes the TSM
>>>client to query the TSM database for backup file candidates and not
>>>due to the actual movement of files from the client to the TSM
>>>server. So how pulling from 10 separate drives increase the query
speed.
>>>
>>>For instance it takes hours and GB of memory on my client for the
>>>client to query the TSM server for file info for 15 million files
>>>even though the actual backup will end up being 11,000 files
>>>consisting of 300mb.
>>>And the backup take over 17+ hours to complete. This is running from
>>>a GigE connection with my client with 4 cpus and 6GB of memory
>>>although it is Win2k.
>>>
>>>-----Original Message-----
>>>From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf
>>>Of Stapleton, Mark
>>>Sent: Thursday, June 16, 2005 12:53 PM
>>>To: ADSM-L@VM.MARIST.EDU
>>>Subject: Re: 15,000,000 + files on one directory backup
>>>
>>>No, because you're pulling data off of 10 separate physical drives.
>>>It's likely to be significantly faster.
>>>
>>>--
>>>Mark Stapleton ([EMAIL PROTECTED]) IBM Certified Advanced
>>>Deployment Professional Tivoli Storage Management Solutions 2005 IBM
>>>Certified Advanced Technical Expert (CATE) AIX Office 262.521.5627
>>>
>>>
>>>
>>>>-----Original Message-----
>>>>From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On
>>>>Behalf Of Dearman, Richard
>>>>Sent: Thursday, June 16, 2005 12:46 PM
>>>>To: ADSM-L@VM.MARIST.EDU
>>>>Subject: Re: 15,000,000 + files on one directory backup
>>>>
>>>>Wouldn't the backup take close to the same amount of time has
>>>using one
>>>>mount point. Because the TSM client on that one server still
>>>had to go
>>>>through 15 million files whether its doing it in one session or 10
>>>>sessions.
>>>>
>>>>I am experiencing the same problem of an imaging system. I am trying
>>>>to go the snapshot image route of the 12 millions 300Gb of files and
>>>>sending them to disk storage pool then off to 3592 tapes nightly. It
>>>>will grow to 1Tb of the next year. I'm not sure how imaging will
work
>>>>on such a large file system.
>>>>
>>>>-----Original Message-----
>>>>From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On
>>>>Behalf Of TSM_User
>>>>Sent: Thursday, June 16, 2005 12:32 PM
>>>>To: ADSM-L@VM.MARIST.EDU
>>>>Subject: 15,000,000 + files on one directory backup
>>>>
>>>>I replied to the list about using windows mount points. A reply was
>>>>sent back about it being the opposit of what they wanted. Well I had
>>>>this thought.
>>>>
>>>>If you have an imaging system (or any other application) that can
only
>>>>use one drive letter why not use mount points for that drive. Have a
>>>>server with 10 drives with 100 GB each (D: - M:) Then use
>>>mount points
>>>>to get all 1 TB of space behind the D:\. The application
>>>will then use
>>>>the D:\ alone. Mean while you can run the backup on all 10 drives.
>>>>
>>>>For details on how to set this up you need to consult the Microsoft
>>>>doc.
>>>>There is plenty on MS's website.
>>>>
>>>>Of course if the system is already setup then you'd need to add the
>>>>drives and mount points. Then move the folders under those mount
>>>>points.
>>>>
>>>>__________________________________________________
>>>>Do You Yahoo!?
>>>>Tired of spam? Yahoo! Mail has the best spam protection around
>>>>http://mail.yahoo.com
>>>>
>>>>**************************EMAIL
DISCLAIMER***************************
>>>>
>>>>This email and any files transmitted with it may be confidential and
>>>>are intended solely for the use of the individual or entity to whom
>>>>they are addressed. If you are not the intended recipient or the
>>>>individual responsible for delivering the e-mail to the intended
>>>>recipient, any disclosure, copying, distribution or any action taken
>>>>or
>>>omitted to be
>>>>taken in reliance on it, is strictly prohibited. If you have
>>>>received this e-mail in error, please delete it and notify the
>>>>sender or contact Health Information Management 312.413.4947.
>>>>
>>>>
>>>>
>>>
>>>**************************EMAIL DISCLAIMER***************************
>>>
>>>This email and any files transmitted with it may be confidential and
>>>are intended solely for the use of the individual or entity to whom
>>>they are addressed. If you are not the intended recipient or the
>>>individual responsible for delivering the e-mail to the intended
>>>recipient, any disclosure, copying, distribution or any action taken
>>>or omitted to
be
>>>taken in reliance on it, is strictly prohibited. If you have received
>>>this e-mail in error, please delete it and notify the sender or
>>>contact Health Information Management 312.413.4947.
>>>
>>>
>>>
>>
>
>__________________________________________________________________
>Switch to Netscape Internet Service.
>As low as $9.95 a month -- Sign up today at
http://isp.netscape.com/register
>
>Netscape. Just the Net You Need.
>
>New! Netscape Toolbar for Internet Explorer
>Search from anywhere on the Web and block those annoying pop-ups.
>Download now at http://channels.netscape.com/ns/search/install.jsp
>
>
>---------------------------------
>Yahoo! Sports
> Rekindle the Rivalries. Sign up for Fantasy Football
>
__________________________________________________________________
Switch to Netscape Internet Service.
As low as $9.95 a month -- Sign up today at
http://isp.netscape.com/register
Netscape. Just the Net You Need.
New! Netscape Toolbar for Internet Explorer
Search from anywhere on the Web and block those annoying pop-ups.
Download now at http://channels.netscape.com/ns/search/install.jsp
