Hi,
At the bottom please find the session_info.
When creating north of 200-300 batches, the writing to the arrow file crashes R
– it doesn’t even show an error message. Rstudio just aborts.
I have the feeling that maybe each batch becomes a stream and R has issues with
the connections, but that’s a total guess.
Any help would be appreciated.
##
Here is the function. When running it with 3000 it crashes immediately.
Before that I ran it with 100, and then increased it slowly, and then it
randomly crashed again.
write_arrow_custom(data.frame(A=c(1:100000),B=c(1:100000)),'C:/Temp/test.arrow',3000)
write_arrow_custom <- function(df,targetarrow,nrbatches) {
ct <- nrbatches
idxs <- c(0:ct)/ct*nrow(df)
idxs <- round(idxs,0) %>% as.integer()
idxs[length(idxs)] <- nrow(df)
df_nav <- idxs %>% as.data.frame() %>% rename(colfrom=1) %>%
mutate(colto=lead(colfrom)) %>% mutate(colfrom=colfrom+1) %>%
filter(!is.na(colto)) %>% mutate(R=row_number())
stopifnot(df_nav %>% mutate(chk=colto-colfrom+1) %>% '$'('chk') %>%
sum()==nrow(df))
table_df <- Table$create(name=rownames(df[1,]),df[1,])
writer <- RecordBatchFileWriter$create(targetarrow,table_df$schema)
df_nav %>% dlply(c('R'),function(df_nav){
catl(glue('{df_nav$colfrom[1]}:{df_nav$colto[1]} / {df_nav$R[1]}...'))
tmp <- df[df_nav$colfrom[1]:df_nav$colto[1],]
writer$write_batch(record_batch(name = rownames(tmp), tmp))
NULL
}) -> batch_lst
writer$close()
rm(batch_lst)
gc()
}
[cid:image001.jpg@01D5C628.B003ACC0]
##
- Session info
-----------------------------------------------------------------------------------------------------------------------------------------------------------
setting value
version R version 3.6.1 (2019-07-05)
os Windows 10 x64
system x86_64, mingw32
ui RStudio
language (EN)
collate English_United States.1252
ctype English_United States.1252
tz America/New_York
date 2020-01-08
- Packages
---------------------------------------------------------------------------------------------------------------------------------------------------------------
! package * version date lib source
acepack 1.4.1 2016-10-29 [1] CRAN (R 3.6.1)
arrow * 0.15.1.1 2019-11-05 [1] CRAN (R 3.6.2)
askpass 1.1 2019-01-13 [1] CRAN (R 3.6.1)
assertthat 0.2.1 2019-03-21 [1] CRAN (R 3.6.1)
backports 1.1.5 2019-10-02 [1] CRAN (R 3.6.1)
base64enc 0.1-3 2015-07-28 [1] CRAN (R 3.6.0)
bit 1.1-14 2018-05-29 [1] CRAN (R 3.6.0)
bit64 0.9-7 2017-05-08 [1] CRAN (R 3.6.0)
blob 1.2.0 2019-07-09 [1] CRAN (R 3.6.1)
callr 3.3.1 2019-07-18 [1] CRAN (R 3.6.1)
cellranger 1.1.0 2016-07-27 [1] CRAN (R 3.6.1)
checkmate 1.9.4 2019-07-04 [1] CRAN (R 3.6.1)
cli 1.1.0 2019-03-19 [1] CRAN (R 3.6.1)
cluster 2.1.0 2019-06-19 [2] CRAN (R 3.6.1)
codetools 0.2-16 2018-12-24 [2] CRAN (R 3.6.1)
colorspace 1.4-1 2019-03-18 [1] CRAN (R 3.6.1)
commonmark 1.7 2018-12-01 [1] CRAN (R 3.6.1)
crayon 1.3.4 2017-09-16 [1] CRAN (R 3.6.1)
credentials 1.1 2019-03-12 [1] CRAN (R 3.6.2)
curl * 4.2 2019-09-24 [1] CRAN (R 3.6.1)
data.table 1.12.2 2019-04-07 [1] CRAN (R 3.6.1)
DBI * 1.0.0 2018-05-02 [1] CRAN (R 3.6.1)
desc 1.2.0 2018-05-01 [1] CRAN (R 3.6.1)
devtools * 2.2.0 2019-09-07 [1] CRAN (R 3.6.1)
digest 0.6.23 2019-11-23 [1] CRAN (R 3.6.1)
dplyr * 0.8.3 2019-07-04 [1] CRAN (R 3.6.1)
DT 0.9 2019-09-17 [1] CRAN (R 3.6.1)
ellipsis 0.3.0 2019-09-20 [1] CRAN (R 3.6.1)
evaluate 0.14 2019-05-28 [1] CRAN (R 3.6.1)
foreign 0.8-71 2018-07-20 [2] CRAN (R 3.6.1)
Formula * 1.2-3 2018-05-03 [1] CRAN (R 3.6.0)
fs 1.3.1 2019-05-06 [1] CRAN (R 3.6.1)
fst * 0.9.0 2019-04-09 [1] CRAN (R 3.6.1)
future * 1.15.0-9000 2019-11-19 [1] Github
(HenrikBengtsson/future@bc241c7)
ggplot2 * 3.2.1 2019-08-10 [1] CRAN (R 3.6.1)
globals 0.12.4 2018-10-11 [1] CRAN (R 3.6.0)
glue * 1.3.1 2019-03-12 [1] CRAN (R 3.6.1)
gridExtra 2.3 2017-09-09 [1] CRAN (R 3.6.1)
gt * 0.1.0 2019-11-27 [1] Github (rstudio/gt@284bbe5)
gtable 0.3.0 2019-03-25 [1] CRAN (R 3.6.1)
Hmisc * 4.3-0 2019-11-07 [1] CRAN (R 3.6.1)
htmlTable 1.13.2 2019-09-22 [1] CRAN (R 3.6.1)
D htmltools 0.3.6.9004 2019-09-20 [1] Github (rstudio/htmltools@c49b29c)
htmlwidgets 1.3 2018-09-30 [1] CRAN (R 3.6.1)
jsonlite * 1.6 2018-12-07 [1] CRAN (R 3.6.1)
knitr 1.25 2019-09-18 [1] CRAN (R 3.6.1)
lattice * 0.20-38 2018-11-04 [2] CRAN (R 3.6.1)
latticeExtra 0.6-28 2016-02-09 [1] CRAN (R 3.6.1)
lazyeval 0.2.2 2019-03-15 [1] CRAN (R 3.6.1)
lifecycle 0.1.0 2019-08-01 [1] CRAN (R 3.6.1)
listenv 0.7.0 2018-01-21 [1] CRAN (R 3.6.1)
lubridate * 1.7.4 2018-04-11 [1] CRAN (R 3.6.1)
magrittr * 1.5 2014-11-22 [1] CRAN (R 3.6.1)
Matrix 1.2-17 2019-03-22 [2] CRAN (R 3.6.1)
memoise 1.1.0 2017-04-21 [1] CRAN (R 3.6.1)
munsell 0.5.0 2018-06-12 [1] CRAN (R 3.6.1)
nnet 7.3-12 2016-02-02 [2] CRAN (R 3.6.1)
openssl 1.4.1 2019-07-18 [1] CRAN (R 3.6.1)
outliers * 0.14 2011-01-24 [1] CRAN (R 3.6.0)
pillar 1.4.2 2019-06-29 [1] CRAN (R 3.6.1)
pkgbuild 1.0.5 2019-08-26 [1] CRAN (R 3.6.1)
pkgconfig 2.0.2 2018-08-16 [1] CRAN (R 3.6.1)
pkgload 1.0.2 2018-10-29 [1] CRAN (R 3.6.1)
plyr * 1.8.4 2016-06-08 [1] CRAN (R 3.6.1)
prettyunits 1.0.2 2015-07-13 [1] CRAN (R 3.6.1)
processx 3.4.1 2019-07-18 [1] CRAN (R 3.6.1)
pryr * 0.1.4 2018-02-18 [1] CRAN (R 3.6.1)
ps 1.3.0 2018-12-21 [1] CRAN (R 3.6.1)
purrr * 0.3.2 2019-03-15 [1] CRAN (R 3.6.1)
R6 * 2.4.1 2019-11-12 [1] CRAN (R 3.6.1)
RColorBrewer 1.1-2 2014-12-07 [1] CRAN (R 3.6.0)
Rcpp 1.0.3 2019-11-08 [1] CRAN (R 3.6.1)
readxl * 1.3.1 2019-03-13 [1] CRAN (R 3.6.1)
remotes 2.1.0 2019-06-24 [1] CRAN (R 3.6.1)
rlang * 0.4.2 2019-11-23 [1] CRAN (R 3.6.1)
rmarkdown * 2.0.3 2019-12-19 [1] Github (rstudio/rmarkdown@26cc3b1)
RODBC * 1.3-16 2019-09-03 [1] CRAN (R 3.6.1)
roxygen2 * 6.1.1 2018-11-07 [1] CRAN (R 3.6.1)
rpart 4.1-15 2019-04-12 [2] CRAN (R 3.6.1)
rprojroot 1.3-2 2018-01-03 [1] CRAN (R 3.6.1)
RSQLite * 2.1.2 2019-07-24 [1] CRAN (R 3.6.1)
rstudioapi 0.10 2019-03-19 [1] CRAN (R 3.6.1)
scales 1.0.0 2018-08-09 [1] CRAN (R 3.6.1)
sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.6.1)
slide * 0.0.0.9002 2019-11-27 [1] Github (DavisVaughan/slide@92e8e02)
ssh 0.6 2019-04-09 [1] CRAN (R 3.6.2)
stringi 1.4.3 2019-03-12 [1] CRAN (R 3.6.0)
stringr * 1.4.0 2019-02-10 [1] CRAN (R 3.6.1)
survival * 2.44-1.1 2019-04-01 [2] CRAN (R 3.6.1)
testthat 2.2.1 2019-07-25 [1] CRAN (R 3.6.1)
tibble 2.1.3 2019-06-06 [1] CRAN (R 3.6.1)
tidyr * 1.0.0 2019-09-11 [1] CRAN (R 3.6.1)
tidyselect 0.2.5 2018-10-11 [1] CRAN (R 3.6.1)
usethis * 1.5.1 2019-07-04 [1] CRAN (R 3.6.1)
varhandle * 2.0.3 2018-07-04 [1] CRAN (R 3.6.0)
vctrs 0.2.0.9007 2019-11-27 [1] Github (r-lib/vctrs@945809e)
withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.1)
xfun 0.9 2019-08-21 [1] CRAN (R 3.6.1)
xml2 * 1.2.2 2019-08-09 [1] CRAN (R 3.6.1)
xts * 0.11-2 2018-11-05 [1] CRAN (R 3.6.1)
zoo * 1.8-6 2019-05-28 [1] CRAN (R 3.6.1)
[1] C:/Users/cklar/Desktop/R packages
[2] C:/Program Files/R/R-3.6.1/library
P -- Loaded and on-disk path mismatch.
D -- DLL MD5 mismatch, broken installation.
Christian Klar
TFG Asset Management
Tetragon Financial Management
399 Park Avenue, 22nd Floor | New York, NY 10022 | United States
Direct: +1 212 359 7369 | Main: +1 212 359 7300 | Mobile: +1 607 216 5045
ck...@tetragoninv.com<mailto:ck...@tetragoninv.com>
www.tetragoninv.com<http://www.tetragoninv.com>
This communication and all or some of the information contained therein may be
confidential. If you have received this communication in error, please destroy
all electronic and paper copies and notify the sender immediately. Unless
specifically indicated, this communication is not a confirmation, an offer to
sell or solicitation of any offer to buy any financial product, or an official
statement of Tetragon Financial Group or its affiliates. TFG Asset Management
L.P. and Tetragon Financial Management LP are registered as investment advisers
under the U.S. Investment Advisers Act of 1940.
