Daria,
I'm glad that using the STAF PROCESS service to run the xcopy command is
giving you the performance you want.
Note that in order to use the STAF local IPC interface (instead of the
TCP/IP interface which is slower), you need to specify "local" for the
TOMACHINE value (as I said in my previous note). You were specifying
"mymachine" instead of "local". For example:
STAF local FS COPY DIRECTORY \\machineName\D\QA\Test TODIRECTORY
D:/QA/Test TOMACHINE local RECURSE KEEPEMPTYDIRECTORIES IGNOREERRORS
Also, note that Windows-only commands like xcopy will have better
performance for UNC shared paths (which is what I meant previously by
mapped drives).
--------------------------------------------------------------
Sharon Lucas
IBM Austin, luc...@us.ibm.com
(512) 286-7313 or Tieline 363-7313
Daria Holden <dariahol...@gmail.com>
07/29/2009 06:28 PM
To
Sharon Lucas/Austin/i...@ibmus
cc
staf-users@lists.sourceforge.net
Subject
Re: [staf-users] fs copy directory performance
Hi Sharon,
To answer some of you questions:
1) When I copy the folder in windows explorer, I am copying from a UNC
path to the local machine. For example, I'll find the file on the
share \\<staf-machine>\d\QA\Test, copy it, and paste it to local drive
D:\QA\Test. This operation takes 15 seconds. While
\\<staf-machine>\D\QA\Test is shared, I have not mapped it to a drive
letter on the local machine. There are no mapped drives in this
scenario
2) Here again is the request that I sent from the <staf-machine>
STAF local FS COPY DIRECTORY D:/QA/Test TODIRECTORY D:/QA/Test
TOMACHINE mymachine RECURSE KEEPEMPTYDIRECTORIES IGNOREERRORS
this request takes over 16 minutes to complete.
3) I am making the request from and copying from the same machine. I
am copying to a different machine.
4) As you can see above, I did specify TOMACHINE, and I did make the
request from "local".
5) I tried your xcopy request as follows:
STAF local PROCESS START COMMAND "xcopy" PARMS "D:\QA\Test
\\mymachine\qa\Test /s/e/i/c/v/y" RETURNSTDOUT STDERRTOSTDOUT
SAMECONSOLE WAIT
and it took under 20 seconds to complete. Note D: is a local drive on
"local". \\mymachine\qa is shared.
In summary, xcopy is as fast as Explorer, and I can use it with UNC
paths to copy to shares on both our Windows and RedHat clusters.
This is EXACTLY the kind of solution I was looking for. YOU GUYS
ROCK!
I still think it's weird that the FS COPY DIRECTORY takes so much
longer, but I have my workaround, so am happy..
Thank you...
Daria
On Wed, Jul 29, 2009 at 4:17 PM, Sharon Lucas<luc...@us.ibm.com> wrote:
>
> Daria,
>
> Most operating system's copy commands (e.g. via Windows Explorer or a
copy
> or xcopy command) only let you copy a file/directory to the same
machine,
> not to a remote machine, unless you are using network drives, such as a
> mapped drive on Windows. Then it can copy to or from the network drive
that
> you have access to on the machine. Also, note that these operating
system
> copy commands are tuned specifically for that one operating system.
>
> The main purpose of the STAF FS COPY requests is to provide a common way
to
> copy files/directories from one machine to another, no matter what
operating
> system (Windows or Unix) that you're copying files to or from. The
speed of
> the copy depends on your network's TCP/IP connections when copying to a
> remote machine and so it will generally be slower than a local copy
> performed by an operating system's copy command.
>
> You said you used Windows Explorer to copy a directory to a remote
machine?
> Do you mean you copied the directory to and from a mapped drive named
D:?
> e.g. Is D: a mapped drive or a local drive?
>
> What is the exact STAF FS COPY request that you submitted?
>
> Are you copying a directory from and to the same machine and are you
> submitting the STAF FS COPY request from this same machine?
>
> Did you specify the TOMACHINE option in your STAF FS COPY request? Note
> that the TOMACHINE specifies the machine to copy the directory and its
> contents to. This defaults to the machine which originated the request.
> Specifying "local" indicates to copy the directory to the same machine
that
> the directory is being copied from. Note that specifying "local"
instead of
> the from machine's host name can significantly improve performance,
> especially if your TCP network performance is slow. This is because
"local"
> indicates to use the local IPC network interface whereas specifying a
TCP
> host name or IP address indicates to use the TCP network interface.
(This
> is documented in the STAF User's Guide.)
>
> I hope this helps explain some of the performance differences you are
> seeing.
>
> Also note that you can use the STAF PROCESS START request to run a
Windows
> operating system's copy command like xcopy if you are copying a
directory
> from one location to another on the same machine (instead of a STAF FS
COPY
> request) if its performance is faster. For example:
>
> STAF machine PROCESS START COMMAND "xcopy" PARMS "c:\mydir1 c:\mydir2
> /s/e/i/c/v/y" RETURNSTDOUT STDERRTOSTDOUT SAMECONSOLE WAIT
>
> --------------------------------------------------------------
> Sharon Lucas
> IBM Austin, luc...@us.ibm.com
> (512) 286-7313 or Tieline 363-7313
>
>
>
> Daria Holden <dariahol...@gmail.com>
>
> 07/29/2009 04:22 PM
>
> To
> staf-users@lists.sourceforge.net
> cc
> Subject
> [staf-users] fs copy directory performance
>
>
>
>
> Can you provide any insight as to why the following call in a STAX job
> to copy a 13MB directory takes over 15 minutes when copying the same
> directory between machines using Windows Explorer takes 15 seconds?
>
> FS <machine1> COPY DIRECTORY D:/QA/Test TODIRECTORY D:/QA/Test
> TOMACHINE <machine2> RECURSE KEEPEMPTYDIRECTORIES IGNOREERRORS
>
>
> Am I missing the "super speedy" option? :-)
>
> Thanks,
> Daria
>
>
------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008
30-Day
> trial. Simplify your report design, integration and deployment - and
focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now. http://p.sf.net/sfu/bobj-july
> _______________________________________________
> staf-users mailing list
> staf-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/staf-users
>
>
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
staf-users mailing list
staf-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/staf-users
