Hi,
Base R 4.2.0 introduced a change ([Rd] R 4.2.0 is released), "Calling if() or
while() with a condition of length greater than one gives an error rather than
a warning."
The below code is a reproducible example of the issue. If it is executed in R
>=4.2.0 then it will generate an error, else just a warning message. Sys.time()
is a multi-class object in R, and throughout the Spark R repository 'if'
statement is used as: 'if(class(x) == "Column")' - this causes the error in
latest R version. Note that, R allows an object to have multiple 'class' names
as a character vector (R: Object Classes); hence this type of check itself was
not a good idea in the first place.
t <- Sys.time() sdf <- SparkR::createDataFrame(data.frame(xx = t +
c(-1,1,-1,1,-1)))
SparkR::collect(SparkR::filter(sdf, SparkR::column("xx") > t))
The suggested change is to add 'all' function while doing the check of whether
class(.) is Column or not: 'if(all(class(x) == "Column"))'.
Creating an issue in JIRA is not very clear to me, hence mailing it to the
'user' list.
Vivek
