[R] Reading extremly large comma separated files?

Dear List, Are there any tricks or wise suggestions on how to import huge comma separated files into R? We have used the ordinary "read syntax" but it takes forever Thank you in advance! Cheers, Patrik __ R-help@r-project.org mailing list https:/

Re: [R] Reading extremly large comma separated files?

The sqldf package may be of help to you. Regards Søren -Original Message- From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of Öhagen Patrik Sent: 19. december 2012 10:48 To: R hELP Subject: [R] Reading extremly large comma separated files? Dear List,

Re: [R] How to convert xts data into list

just try plot(zc$Close) Sent from my iPad On Dec 19, 2012, at 2:45, 박상규 wrote: > Hello, > > > How can I convert Close colume of the below xts time series data into a list > of Close values ? > I'd like to plot Close values as a list. > > > > > head(zc) > Close > (10/

Re: [R] How to convert xts data into list

it would help a lot if you posted your data using 'dput' so we know its format; all you did was show the output on the console -- you should at least do 'str' and follow the posting guide. Sent from my iPad On Dec 19, 2012, at 6:26, 박상규 wrote: > Thank you for reply. > > But still it do

Re: [R] Reading extremly large comma separated files?

> Are there any tricks or wise suggestions on how to import > huge comma separated files into R? We have used the ordinary > "read syntax" but it takes forever One potential time- and memory saver is to specify the class of each column using colClasses in read.table. For a large data set (

[R] triax.plot: control legend position and size of point labels

Given this example library(plotrix) a<-c(34,10,70) b<-c(33,10,20) c<-c(33,80,10) test<-data.frame(A=a,B=b,C=c) triax.plot(test, main ="title", at=seq(0.25,0.75,by=0.25), tick.labels=list(l=seq(0.25,0.75,by=0.25), r=seq(0.25,0.75,by=0.

Re: [R] Changing Variable Names In VCD

On 12/18/2012 1:25 PM, Simon Kiss wrote: Hello: What is the most efficient way to change the plotted variable names in mosaic plots in the vcd package? Should one do a separate contingency table first, change the dimension names there and then pass that to mosaic? Or is there a way to do it sim

Re: [R] Reading extremly large comma separated files?

On Wed, Dec 19, 2012 at 9:47 AM, Öhagen Patrik wrote: > Dear List, > > Are there any tricks or wise suggestions on how to import huge comma > separated files into R? We have used the ordinary "read syntax" but it takes > forever > How huge is "huge"? What is 'the ordinary "read syntax"'?

[R] sas2rd - write an .Rd file for a sas data set

I have quite a few legacy SAS data sets and it's a pain when I want to include one in an R package because there was no easy way to transfer variable labels into an .Rd file. Now there is: http://www.datavis.ca/sasmac/sas2rd.html -- Michael Friendly Email: friendly AT yorku DOT ca Professor

Re: [R] Reading extremly large comma separated files?

The sqldf package as suggested by a friendly soul worked excellent. ...and yes! Forever +/- epsilon ;) Cheers, Patrik -Ursprungligt meddelande- Från: b.rowling...@gmail.com [mailto:b.rowling...@gmail.com] För Barry Rowlingson Skickat: den 19 december 2012 14:47 Till: Öhagen Patrik Kopi

[R] probability of binary data

Hi, how are you? I am trying to replicate the binary data f(2) function in the attached document by starting with the simple example found below: observed <- matrix(c(0, 1, 0, 0, 1, 1, 1, 0, 0),3,3,byrow=TRUE) data <- matrix(c(1, 1, 0, 0, 1, 0, 0, 0, 1),3,3,byrow=TRUE) f2 = sum(probability of th

Re: [R] Breaking out of multiple loops

Something like this? n<-1000 pythag<-function(n){ for(i in 3:((n-3)/3)) for(j in (i+1):((n-i)/2)) if(i^2+j^2==(n-i-j)^2) return(i*j*(n-i-j)) } system.time(print(pythag(n))) Interesting -- seems to improve speed by ~12%. Not sure if that's because stop() has more overhead, or if functi

Re: [R] How to make Ordinary Kriging using gstat predict?

Hi Dimitris, The mistake is that predict.gstat doesnt have a "model" argument, as you assume. But as the function also accepts arguments through ..., it does not complain about the unused argument. Try instead to put the model argument in the gstat-object as you can see in the example in ?pred

Re: [R] probability of binary data

Hello,, Try the following. f2 <- function(obs, dat){ obs <- as.logical(obs) dat <- as.logical(dat) s1 <- sum(obs & dat) s2 <- sum(obs & !dat) s3 <- sum(!obs & dat) s1/(s1 + s2 + s3) } observed <- matrix(c(0, 1, 0, 0, 1, 1, 1, 0, 0),3,3,byrow=TRUE) data <- matrix(c(

Re: [R] probability of binary data

Hi Rui, how are you? That worked perfectly! Thank-you very much! Irucka <-Original Message-> >From: Rui Barradas [ruipbarra...@sapo.pt] >Sent: 12/19/2012 9:33:53 AM >To: iruc...@mail2world.com >Cc: r-help@r-project.org >Subject: Re: [R] probability of binary data > >Hello,, > >Try the

Re: [R] How to convert xts data into list

Thank you for your help. Now it works. SK Park -Original Message- From: "arun" To: "박상규"; Cc: "R help"; Sent: 2012-12-20 (목) 00:13:40 Subject: Re: [R] How to convert xts data into list HI, As I said earlie

Re: [R] Breaking out of multiple loops

There is. Use while loops. --- Jeff NewmillerThe . . Go Live... DCN:Basics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playi

[R] Convert negativ number to 0

Hallo together, i have a data.frame like this one. A 13 2-2 31 and now i want to convert the negative number to 0. A 13 20 31 How can i do this? Thanks. Mat -- View this message in context: http://r.789695.n4.nabble.com/Convert-negativ-number-to-0-tp46

Re: [R] How to convert xts data into list

Thank you for reply. But still it doesn't work. It seems that it's because of date time indexes. > plot(zc$Close) parse.format(format[1]) : unrecognized format %b %d %H:%M SK Park -Original Message- From: "Jim Holtman" To: "박상규"; Cc:

[R] pROC and ROCR give different values for AUC

Packages pROC and ROCR both calculate/approximate the Area Under (Receiver Operator) Curve. However the results are different. I am computing a new variable as a predictor for a label. The new variable is a (non-linear) function of a set of input values, and I'm checking how different parameter

[R] structural breaks estimation

Dear Sir I am using strucchange package to get the multiple structural breaks in the growth series. I have used the breakpoints method. For some of the countries when I calculate the confidence intervals for the breakpoints they overlap or exceed the considered time interval. Are these breaks sign

Re: [R] How to convert xts data into list

Hi, I didn't understand ">". Lines1<-textConnection("Date Time Close 10/15/12 09:00:00 252.40 10/15/12 09:01:00 253.10 10/15/12 09:02:00 253.15 10/15/12 09:03:00 253.30 10/15/12 09:04:00 253.25 10/15/12 09:05:00 253.45") library(zoo) library(xts) z<- read.zoo(Lines1,header=TRUE,index=list(1,2),FUN

Re: [R] How to convert xts data into list

HI, As I said earlier, I am able to generate the plot. pdf("Parkplot1.pdf")  plot(1:length(zc1$Close),split(zc1$Close,row(zc1)))  dev.off() #the pdf is attached. A.K. From: 박상규 To: arun Sent: Wednesday, December 19, 2012 10:04 AM Subject: Re: [R] How to c

Re: [R] Convert negativ number to 0

works perfectly. Thank you :-) -- View this message in context: http://r.789695.n4.nabble.com/Convert-negativ-number-to-0-tp4653529p4653532.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://sta

Re: [R] Convert negativ number to 0

mydf[mydf < 0] <- 0 or mydf$A[mydf$A < 0] <- 0 depending on what exactly you want. Note: this is the kind of operation for which there are many possible equivalent solutions, and many tweaks possible depending on how you want to deal with NA value, etc. Sarah On Wed, Dec 19, 2012 at 10:03 AM,

Re: [R] How to convert xts data into list

chron indexes aren't well-supported in xts. Convert the index class to POSIXct and it will plot. indexClass(zc) <- "POSIXct" plot(zc) Best, -- Joshua Ulrich | about.me/joshuaulrich FOSS Trading | www.fosstrading.com On Wed, Dec 19, 2012 at 9:47 AM, 박상규 wrote: > > > Thank you for your help

Re: [R] pROC and ROCR give different values for AUC

A reproducible example sent to the package maintainer(s) might yield results. Max On Wed, Dec 19, 2012 at 7:47 AM, Ivana Cace wrote: > Packages pROC and ROCR both calculate/approximate the Area Under (Receiver > Operator) Curve. However the results are different. > > I am computing a new varia

[R] JRI and XTS library

Hi all, I am using JRI with the XTS library. Rengine rEngine = new Rengine(new String [] {/*"--vanilla"*/}, false, null); rEngine.eval("library(xts)"); When I create an xts object, using previously assigned double/date arrays with roughly 5000 elements, the eval method needs very long (up to a

Re: [R] pROC and ROCR give different values for AUC

On Dec 19, 2012, at 4:47 AM, Ivana Cace wrote: > Packages pROC and ROCR both calculate/approximate the Area Under (Receiver > Operator) Curve. However the results are different. > > I am computing a new variable as a predictor for a label. The new variable is > a (non-linear) function of a set

Re: [R] pROC and ROCR give different values for AUC

I considered that but on the list more people are likely to see it. So if they ran into the same thing they may already have figured out what is going on and have answers. Or if not, I'll make a reproducible example, ask the maintainers and post back here where others may find it. (but not til

Re: [R] Breaking out of multiple loops

There is a break() function. Does it not do the job? -Don -- Don MacQueen Lawrence Livermore National Laboratory 7000 East Ave., L-627 Livermore, CA 94550 925-423-1062 On 12/19/12 7:27 AM, "McCloskey, Bryan" wrote: >Something like this? > >n<-1000 >pythag<-function(n){ > for(i in 3:((n-

[R] Copy data from Excel

Hello again, I my day to day calculation, I need to take lot of data from Excel and forth and generally I use 'clipboard' option with read.delim() function. However many time, the data in Excel are like '(111,000)' instead of '-111000'. Generally I convert data in the 2nd form in Excel itself

Re: [R] pROC and ROCR give different values for AUC

On Wed, Dec 19, 2012 at 12:36 PM, Ivana Cace wrote: > I considered that but on the list more people are likely to see it. So if > they ran into the same thing they may already have figured out what is going > on and have answers. > Or if not, I'll make a reproducible example, ask the maintainers

Re: [R] How to convert xts data into list

On Dec 19, 2012, at 1:07 PM, arun wrote: > Hi, > > I didn't understand ">". I think it's the result of HTMangLing the R prompt. M __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide h

[R] Non-linear regression analysis in R‏

Hey all, I'm trying to fit a non-linear model y ~ a * constant ^ b * x ^ c and estimates the paramaters a, b and c. Using the nls function, I'm getting following error message: Error in nlsModel(formula, mf, start, wts) : singular gradient matrix at initial parameter estimates If I logarit

[R] problem with opening more than one SOCK cluster with package snow

Dear list, i have some problems using the snow package to create a SOCK cluster. The errors just occour irregularly but it seems to me that they occour when I try to create more than one cluster on the same machine via different R instances started via submitting LSF jobs to a cluster. Does a

Re: [R] Copy data from Excel

On Dec 19, 2012, at 10:22 AM, Christofer Bogaso wrote: > Hello again, > > I my day to day calculation, I need to take lot of data from Excel and forth > and generally I use 'clipboard' option with read.delim() function. However > many time, the data in Excel are like '(111,000)' instead of '-1

Re: [R] Copy data from Excel

If you are going to be reading a lot of data from EXCEL, use the XLConnect package since it will give you back the values as numeric instead of the "formatted" values with commas. Also it means you can automate the activity since you don't have to use the "clipboard". On Wed, Dec 19, 2012 at 1:22

Re: [R] Non-linear regression analysis in R

Could you provide the code that you're running, so we can see what you're trying to do? Even better would be a repeatable example. Jeremy On 19 December 2012 09:42, Yann Labou wrote: > Hey all, > > I'm trying to fit a non-linear model y ~ a * constant ^ b * x ^ c and > estimates the paramaters

[R] Display of data points in the Scatterplot

Respected R forum I am learning R and relatively quite new to R. I am generating a scatter-plot as given below. (My actual table is much larger). # Sample data frame y = c(20, 23, 17, 31, 68) x = c(200, 300, 400, 500, 600) plot(x, y, type = 'l') If I plot this scatter-plot in excel, the da

Re: [R] Non-linear regression analysis in R

Jeremy: Don't be silly. The model is overdetermined -- a and b "tradeoff" with each other. e.g. for any solution (a,b), (a/k^m,b+m) for any m is also a solution, where k = const (assuming I have correctly interpreted the model, of course). -- Bert On Wed, Dec 19, 2012 at 11:59 AM, Jeremy Miles w

[R] create stratified splits

Hi, I have a vector like: r <- runif(100) Now I would like to split r into 10 pieces (each with 10 elements) – but the 'pieces' should be roughly similar with regard to mean and sd. what is an efficient way to do this in R? thanks! __ R-help@r-pro

Re: [R] Display of data points in the Scatterplot

On Dec 19, 2012, at 11:28 AM, Katherine Gobin wrote: > Respected R forum > > I am learning R and relatively quite new to R. I am generating a scatter-plot > as given below. (My actual table is much larger). > > > > # Sample data frame > > > y = c(20, 23, 17, 31, 68) > x = c(200, 300, 400,

[R] "For" loop and "if" question

All - I have a large data frame that looks like ID p1 p2 p3...p20 Lat1 Lat2 Lat3...Lat20 Long1 Long2 Long3...Long20 1 0 0 1 0NA NA 29.xx NA NA NA -89.xx NA 2 1 0 0 127.xx NA

Re: [R] "For" loop and "if" question

A friendly r-helper asked me to use dput() to output some of my data. The results are below: structure(list(ID = 2004:2008, p1 = c(1L, 1L, 1L, 1L, 0L), p2 = c(0L, 0L, 0L, 0L, 1L), p3 = c(0L, 0L, 0L, 0L, 0L), p4 = c(0L, 0L, 1L, 1L, 0L), p5 = c(0L, 0L, 0L, 0L, 0L), p6 = c(1L, 1L, 0L, 1L, 1L ), p7 =

[R] Theoretical confidence regions for any non-symmetric bivariate statistical distributions

Respected R Users, I looking for help with generating theoretical confidence regions for any of non-symmetric bivariate statistical distributions (bivariate Chi-squared distribution, bivariate F-distribution, or any of the others). I want to to used it as a benchmark to compare a few strategies co

[R] confirming a formula for use with lmer

Hello, I recently began using R and the lme4 package to carry out linear mixed effects analyses. I am interested in the effects of variables 'prime','time', and 'mood' on 'reaction_time' while taking into account the random effect 'subjects.' I've read through documentation on lme4 and came

Re: [R] Theoretical confidence regions for any non-symmetric bivariate statistical distributions

What does this have to do with R? Post on a statistical list (like stats.stackexchange.com). I would urge respondents here to reply privately. -- Bert On Wed, Dec 19, 2012 at 1:08 PM, Zhiqiu Hu wrote: > Respected R Users, > > I looking for help with generating theoretical confidence regions f

Re: [R] scope, lme, ns, nlme, splines

On 12-12-19 5:02 PM, Jacob Wegelin wrote: Your solution works beautifully for predict.lme in the same data that were used to compute the model. But what if I want to compute the population fitted values in newdata? I didn't offer a solution, I offered a workaround for a bug. The solution is

Re: [R] scope, lme, ns, nlme, splines

Your solution works beautifully for predict.lme in the same data that were used to compute the model. But what if I want to compute the population fitted values in newdata? In the expanded example below, predict.lm is able to find the object called "fixed". But predict.lme is unable to find it

Re: [R] create stratified splits

Hi Martin, Interesting question. This is not efficient, but I thought I would post a brute force method that might be good enough. Surely someone will have a better approach... Well we'll see. Here is a dumb, inefficient (but workable) way: # create the vector to be split r <- runif(100) # write

Re: [R] pROC and ROCR give different values for AUC

On Thu, Dec 20, 2012 at 6:36 AM, Ivana Cace wrote: > I considered that but on the list more people are likely to see it. So if > they ran into the same thing they may already have figured out what is > going on and have answers. > Or if not, I'll make a reproducible example, ask the maintainers a

[R] Fitting a predefined classification tree

Hi, I've searched R-help and haven't found an answer. I have a set of data from which I can create a classification tree using rpart. However, what I'd like to do is predefine the blank structure of the binary tree (i.e., which nodes to include) and then use a package like rpart to fit for the

Re: [R] "For" loop and "if" question

Hello, Thanks for the data example. Try the following. fun <- function(DF){ f <- function(x, y){ if(ncol(y) == 3) rbind(x, y) else x } p <- grep("^p[[:digit:]]+$", names(DF)) lat <- grep("^lat[[:digit:]]+$", names(DF)) long <- grep("^long[[:digit:]]+$", names(DF))

Re: [R] create stratified splits

On Dec 19, 2012, at 12:23 PM, Martin Batholdy wrote: > Hi, > > > I have a vector like: > > r <- runif(100) > > Now I would like to split r into 10 pieces (each with 10 elements) – > but the 'pieces' should be roughly similar with regard to mean and sd. > > what is an efficient way to do this

Re: [R] confirming a formula for use with lmer

Matthew Panichello nmr.mgh.harvard.edu> writes: > [snip] > > I am interested in the effects of variables 'prime','time', and 'mood' > on 'reaction_time' while taking into account the random effect > 'subjects.' I've read through documentation on lme4 and came up with the > following formula f

[R] [mgcv][gam] Problem defining axis labels for non-smooth terms (via termplot)

Dear List, plot.gam appears to be having trouble communicating its xlab and ylab information to termplot. A simple example: library(mgcv) x = 1:1000 y = runif(1000)*x^.5 z = rnorm(1000)*y other = sin(z) fit = gam(y~s(x)+s(z)+other) plot(fit,all.terms=TRUE) plot(fit,select=3,xlab="???") de

[R] how to read different files into different objects in one time?

Dear All I have a lot of files in a directory as follows: "02-03.txt" "03-04.txt" "04-05.txt" "05-06.txt" "06-07.txt" "07-08.txt" "08-09.txt" "09-10.txt" "G0.txt" "G1.txt" "raw_ped.txt" .. I want to read them into different objects according to their

Re: [R] how to read different files into different objects in one time?

Dear Yao, You can use a list() as follows (untested): setwd("~/path/to/your/files") f <- list.files(pattern = ".txt") # list the files info <- lapply(f, read.table, header = TRUE) names(info) <- f info To see the information in the first data set just do info[[1]] See ?list.files and ?lapply

Re: [R] how to read different files into different objects in one time?

The short answer is: don't try. You really don't want dozens of different objects in memory that you didn't name yourself. What you want instead is a list of objects. You can start with a vector of filenames and use lapply to create another list containing the data frames. For convenience you c

Re: [R] "For" loop and "if" question

Hi, You could also try this: #dat1<-structure BTW, p20 was labelled incorrectly as p29 library(reshape)  dat2<-melt(dat1) res1<-na.omit(cbind(ID=rep(dat2[,2][grep("ID",dat2$variable)],times=((nrow(melt(dat1))-nrow(dat1))/3)/nrow(dat1) ),dat2[grep("p",dat2$variable),],dat2[grep("lat",dat2$vari