If A has more columns than in your example, you could always try to only
merge those columns of A with B that are relevant for the merging. You could
then cbind the result of the merging back together with the rest of A as
long as the merged data preserved the same order as in A.
Alternatively, you can always use chunks of A and do the merging separately,
e.g., for blocks of 10000 observations or so.
x<-sample(1:150,15000,replace=T)
y<-sample(1:150,15000,replace=T)
a<-rnorm(15000)
b<-rnorm(15000)
A<-cbind(x,a)
B<-cbind(y,b)
system.time(newdata<-merge(A,B,by.x='x',by.y='y',all.x=T,all.y=F))
On a MacBook Pro with 4 Gs of RAM and a 2.4 GHz Duo Core processor it would
take you about 40 minutes if you do chunks for 15000 observations. I am not
sure whether the loop would be slower than that.
On a different note: how are you matching if AA has multiple matches in BB?
Daniel
murilofm wrote:
>
> Thanks Daniel, that helped me. Based on your suggestions I built this
> final code:
>
> library(foreign)
> library(gdata)
>
> AA = c(4,4,4,2,2,6,8,9)
> A1 = c(3,3,11,5,5,7,11,12)
> A2 = c(3,3,7,3,5,7,11,12)
> A = cbind(AA, A1, A2)
>
> BB = c(2,2,4,6,6)
> B1 =c(5,11,7,13,NA)
> B2 =c(4,12,11,NA,NA)
> B3 =c(12,13,NA,NA,NA)
>
> A = cbind(AA, A1, A2,0)
> B=cbind(BB,B1,B2,B3)
>
> newdata<-merge(A,B,by.x='AA',by.y='BB',all.x=T,all.y=F)
> newdata$dum <- rowSums (newdata[,matchcols(newdata,
> with=c("B"))]==newdata$A1, na.rm = FALSE, dims = 1)*
> rowSums (newdata[,matchcols(newdata, with=c("B"))]==newdata$A2, na.rm =
> FALSE, dims = 1)
>
> colnames(A)[4]<-"dum"
> newdata$dum1<-newdata$dum
> A_final<-merge(A,newdata,by.x=c("AA","A1","A2","dum"),by.y=c("AA","A1","A2","dum"),all.x=T,all.y=F)
>
> Which gives me the same result of the "loop" version. Unfortunately, I
> can't replicate it on the original data since i can't make the merge work:
> i get an error message "Reached total allocation of 4090Mb". So, I'm stuck
> again.
>
> If anyone could shed some light on this problem, i would really
> appreciate.
>
--
View this message in context:
http://r.789695.n4.nabble.com/Big-data-and-column-correspondence-problem-tp3694912p3697709.html
Sent from the R help mailing list archive at Nabble.com.
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
