Re: [R] Error when calling (R 4.0.x on Windows) from Python

Tue, 02 Feb 2021 05:50:06 -0800 

A reproducible example:
system("\"C:\\Program Files\\R\\R-4.0.3\\bin\\R.exe\" -e commandArgs() >out")  # does not create "out" system("\"C:\\Program Files\\R\\R-3.6.3\\bin\\R.exe\" -e commandArgs() >out")  # creates "out" 

Note that this does not create "out", either:


system("\"C:\\Program Files\\R\\R-3.6.3\\bin\\x64\\Rterm.exe\" -e 
commandArgs() >out")



I think redirections should be left to the shell, so e.g. those calls 
from Python should use "os.system" if they needed redirection, as Duncan 
suggested.



I see that both R.exe and Rterm.exe give "Usage: Rterm [options] [< 
infile] [> outfile] [EnvVars]", which may be confusing to programmers 
invoking R without a shell, but then talking there about invocation via 
shell could confuse typical users, instead.



My best guess is that the redirection in R.exe in 3.6 worked rather by 
accident, as a consequence of that R.exe internally invoked (and still 
invokes) Rterm.exe via the shell, but R.exe did not protect all 
arguments from expansions from that internal shell invocation. If 
redirection was meant to work without external shell, it would have 
instead been implemented explicitly also in Rterm.exe.



My change 77926 made sure that all arguments to that internal shell 
invocation were protected, following bug reports such as PR#17709, where 
users had "&" in their file names. Therefore, this works, executing file 
code&.r


system("\"C:\\Program Files\\R\\R-4.0.3\\bin\\R.exe\" -f code&.r")


but not with 3.6.3, and I would not be surprised if even more special 
characters became popular, soon.



After all, what if someone wanted to pass an argument to R including 
">", that should work, too:


---- t.r ----
cat("Header: ", commandArgs(TRUE)[1], "\n")
-------------


system("\"C:\\Program Files\\R\\R-4.0.3\\bin\\R.exe\" -f t.r --args 
<html>") # prints Header:  <html>
system("\"C:\\Program Files\\R\\R-3.6.3\\bin\\R.exe\" -f t.r --args 
<html>") # fails with  The syntax of the command is incorrect.


So in summary, I don't agree this is a bug.

Best
Tomas

On 1/29/21 11:19 AM, Duncan Murdoch wrote:

On 29/01/2021 3:57 a.m., Marcel Baumgartner wrote:

Dear Bill, Duncan and Martin,


thanks for your investigation. Can you clarify on next steps? Is this 
now an official bug, or have you found a workaround? For your 
information: the issue showed up the first time when I called R 4.0.2 
from within a software called "IDEA" (from Caseware Analytics), using 
their scripting language (similar to Visual Basic). With my colleague 
we then simply reproduce the error calling R from Python, so that we 
could share it more easily. When we run this command directly on the 
CMD in Windows, all works fine. The issue only happens when R is 
called within another software.





I would say this is a bug, with two workarounds.  These are only 
needed (and will only work) on Windows.



1.  If you want to include redirection in the command line, then call 
cmd.exe yourself, and have it call Rterm.exe.  I think your original 
command


R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1"

could be written as

cmd.exe 1>log.txt 2>&1 /C R.exe -f code.R --args "~/file.txt"


(but I haven't tested it).  You could also use the slightly more 
efficient


cmd.exe 1>log.txt 2>&1 /C Rterm.exe -f code.R --args "~/file.txt"

if Rterm.exe is on your path; by default I don't think it will be.


2.  Change the Python code to a different function call than 
subprocess.call.  It should be one that's equivalent to the C system() 
function; I believe os.system() is what you want, so you'd use


os.system("R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1")


and the redirection would be handled by the implicit shell that is 
called by os.system.  Again, replacing R.exe by Rterm.exe would be a 
tiny bit more efficient, but it might not be on the path.


Duncan Murdoch


Best regards

Marcel



Le 2021-01-27T23:14:36.000+01:00, Bill Dunlap 
<williamwdun...@gmail.com> a écrit :


    I tried the following change, that adds quotes if the argument does
    not include ">".
    Index: front-ends/rcmdfn.c
===================================================================
    --- front-ends/rcmdfn.c (revision 79883)
    +++ front-ends/rcmdfn.c (working copy)
    @@ -173,9 +173,13 @@
    fprintf(stderr, "command line too long\n");
    return(27);
    }
    - strcat(cmd, "\"");
    + if (!strchr(argv[i], '>')) {
    + strcat(cmd, "\"");
    + }
    strcat(cmd, argv[i]);
    - strcat(cmd, "\"");
    + if (!strchr(argv[i], '>')) {
    + strcat(cmd, "\"");
    + }
    }
    /* the outermost double quotes are needed for cmd.exe */
    strcat(cmd, "\"");

    It lets the python example work. I am not sure that quoting all the
    arguments buys you much, as shQuote() is still needed for arguments
    that include spaces. E.g., with 3.6.3, 4.0.3, and my development
    build with the above patch we get

        stopifnot(dir.create(dirname <- file.path(tempfile(), "A
        SPACE"), recursive=TRUE))
        logname <- file.path(dirname, "log.txt")
        unlink(logname)
        system(paste( "C:\\R\\R-3.6.3\\bin\\R.exe --quiet --vanilla -e
        \"commandArgs()\" 1>", logname))

    ARGUMENT 'SPACE/log.txt' __ignored__

    [1] 0


        tryCatch(readLines(logname), 
error=function(e)conditionMessage(e))


    [1] "cannot open the connection"
    Warning message:
    In file(con, "r") :
    cannot open file
'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
    SPACE/log.txt': No such file or directory


        system(paste( "C:\\R\\R-4.0.3\\bin\\R.exe --quiet --vanilla -e
        \"commandArgs()\" 1>", logname))
        commandArgs()

    [1] "C:\\R\\R-40~1.3/bin/x64/Rterm.exe"
    [2] "--quiet"
    [3] "--vanilla"
    [4] "-e"
    [5] "commandArgs()"
    [6] "1>"
    [7]
"C:\\Users\\willi\\AppData\\Local\\Temp\\RtmpM5tsC7\\file1a1068734a49/A"
    [8] "SPACE/log.txt"


    [1] 0


        tryCatch(readLines(logname), 
error=function(e)conditionMessage(e))


    [1] "cannot open the connection"
    Warning message:
    In file(con, "r") :
    cannot open file
'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
    SPACE/log.txt': No such file or directory


        unlink(logname)
        system(paste(
        "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet
        --vanilla -e \"commandArgs()\" 1>", logname))

    [1] 0


        tryCatch(readLines(logname), 
error=function(e)conditionMessage(e))


    [1] "cannot open the connection"
    Warning message:
    In file(con, "r") :
    cannot open file
'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
    SPACE/log.txt': No such file or directory

        tryCatch(readLines(sub(" .*$", "", logname)),
        error=function(e)conditionMessage(e))

    [1] "> commandArgs()"
    "[1]
\"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
    [3] "[2] \"--quiet\"
    " "[3] \"--vanilla\"
    "
    [5] "[4] \"-e\"
    " "[5] \"commandArgs()\"
    "
    [7] "[6] \"SPACE/log.txt\"
    " "> "
    [9] "> "


        unlink(logname)
        system(paste(
        "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet
        --vanilla -e \"commandArgs()\" 1>", shQuote(logname)))

    [1] 0


        tryCatch(readLines(logname), 
error=function(e)conditionMessage(e))


    [1] "> commandArgs()"
    "[1]
\"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
    [3] "[2] \"--quiet\"
    " "[3] \"--vanilla\"
    "
    [5] "[4] \"-e\"
    " "[5] \"commandArgs()\"
    "
    [7] "> "
    "> "

    -Bill

    On Wed, Jan 27, 2021 at 1:25 PM Duncan Murdoch
    <murdoch.dun...@gmail.com <mailto:murdoch.dun...@gmail.com>> wrote:


        On 27/01/2021 3:40 p.m., Bill Dunlap wrote:

            I believe the problem is from svn 77925 in
            gnuwin/front-ends/rcmdfn.c,
            which was committed a few days after 3.6.3 was released.
            Rterm used

            to put double quotes around a command line argument only 
if it

            contained a space, now it double quotes all arguments. It
            sees shell
            constructs like "1>" and the following file name as
            arguments and

            double quoting them hides them from the shell, leading to 
this

            problem. I think we may have to rely on the user supplying
            quotes as
            needed instead of blindly adding them.


        Okay, now I see what you mean.

        If you invoke R using R.exe, it asks cmd.exe to run Rterm.exe,
        so it is
        possible that redirection would be handled.

        If you invoke R directly using Rterm.exe, then my description
        down below
        would be correct.

        Duncan Murdoch



            -Bill

            On Wed, Jan 27, 2021 at 12:28 PM Duncan Murdoch
            <murdoch.dun...@gmail.com <mailto:murdoch.dun...@gmail.com>>
            wrote:


                On 27/01/2021 3:17 p.m., Duncan Murdoch wrote:

                    On 27/01/2021 3:38 a.m., Martin Maechler wrote:

                                            Martin Maechler
                                            on Tue, 26 Jan 2021 12:37:58
                                            +0100 writes:


                                            Marcel Baumgartner
                                            on Tue, 26 Jan 2021 08:55:48
                                            +0100 writes:


                        Dear all, my colleague posted our issue on
                        stackoverflow:

                        Calling R script from Python does not save log
                        file in
                        version 4 - Stack Overflow
                        [stackoverflow.com/questions...
<https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4>]

                        It is about this kind of call to R:

                        R.exe -f code.R --args "~/file.txt" 1>
                        "~/log.txt" 2>&1".

                        The issue is that the log.txt file is not
                        created when
                        running R 4.x.x. The same code works perfectly
                        fine with
                        R 3.6.x.


                        Any idea what's going wrong as of version 4? 
Regards

                        Marcel


                        Dear Marcel, I think the solution is 
embarrassingly

                        simple:

                         From the SO post, where she showed a bit more
                        detail than you

                        show here, it's clear you have confused 
'R.exe' and
                        'Rscript.exe' and what you say above is not 
true:


                        'R.exe' was used for R 3.6.0 but for R 4.0.3,
                        you/she used
                        'Rscript.exe' instead.


                        ... as you've noticed now, they do behave
                        differently,
                        indeed!

                        Well, this was not the solution to their --
                        Windows-only -- problem.
                        The problem *is* indeed visible if they only use
                        R.exe (also
                        for R 4.0.3).

                        I've commented more on the SO issue (see above),
                        notably asking for a *minimal* repr.ex.
                        (reproducible example),
                        and one *not* using "<YOUR PATH>" and setwd() ..


                    Isn't this purely a Python or user problem? R
                    shouldn't process
                    redirection directives like

                    1> "~/log.txt" 2>&1

                    because it's the shell's job to process those. If
                    Python is acting as
                    the shell, it needs to handle those things. If R was
                    handling the
                    command via


                Oops, sent before finishing:

                If R was handling the command via system() or system2(),
                it would handle
                redirection itself. If it was using the Windows-only
                shell(), it would
                call cmd.exe (by default) to handle redirection. (This
                is a difference
                between R on Windows and R in Unix: in Unix a shell is
                always used.)

                Duncan Murdoch

                ______________________________________________
                R-help@r-project.org <mailto:R-help@r-project.org>
                mailing list -- To UNSUBSCRIBE and more, see
                stat.ethz.ch/mailman/listin...
<https://stat.ethz.ch/mailman/listinfo/r-help>
                PLEASE do read the posting guide
                www.R-project.org/posting-g...
<http://www.R-project.org/posting-guide.html>
                and provide commented, minimal, self-contained,
                reproducible code.





______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help

PLEASE do read the posting guide 
http://www.R-project.org/posting-guide.html

and provide commented, minimal, self-contained, reproducible code.


______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
























					Reply via email to