Hi there,
I have a data frame DF with 40 millions strings and their frequency. I
am searching for strings with a given pattern and I am trying to speed
up this part of my code. I try many options but so far I am not
satisfied. I tried:
- grepl and subset are equivalent in term of processing time
grepl(paste0("^",pattern),df$Strings)
subset(df, grepl(paste0("^",pattern), df$Strings))
- lookup(pattern,df) is not what I am looking for since it is doing an
exact matching
- I tried to convert my data frame in a data table but it didn't improve
things (probably read/write of this DT will be much faster)
- the only way I found was to remove 1/3 of the data frame with the
strings of lowest frequency which speed up the process by a factor x10 !
- didn't try yet parRapply and with a machine with multicore I can get
another factor.
I did use parLapply for some other code but I had many issue with
memory (crashing my Mac).
I had to sub-divide the dataset to have it working correctly but I
didn't manage to fully understand the issue.
I am sure their is some other smart way to do that. Any good
article/blogs or suggestion that can give me some guidance ?
Thanks a lot
Cheers
Fabien
--
Dr Fabien Tarrade
Quantitative Analyst/Developer - Data Scientist
Senior data analyst specialised in the modelling, processing and
statistical treatment of data.
PhD in Physics, 10 years of experience as researcher at the forefront of
international scientific research.
Fascinated by finance and data modelling.
Geneva, Switzerland
Email : cont...@fabien-tarrade.eu <mailto:cont...@fabien-tarrade.eu>
Phone : www.fabien-tarrade.eu <http://www.fabien-tarrade.eu>
Phone : +33 (0)6 14 78 70 90
LinkedIn <http://ch.linkedin.com/in/fabientarrade/> Twitter
<https://twitter.com/fabtar> Google
<https://plus.google.com/+FabienTarradeProfile/posts> Facebook
<https://www.facebook.com/fabien.tarrade.eu> Google
<skype:fabtarhiggs?call> Xing <https://www.xing.com/profile/Fabien_Tarrade>
______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
