So,
some feedback.
Have installed MariaDB 10.0.10 on the Linux box. That speeded things up.
Changed from InnoDB/XtraDb to Aria. That speeded loading of the data up.
Have installed MariaDB on the iMac. That speeded things up more.
Tried to tune MariadDB's config. Didn't speed things up much, but f
On Wed, May 7, 2014 at 2:21 AM, David R Forrest wrote:
> It sounds as if your underlying MySQL database is too slow for your purposes.
> Whatever you layer on top of it will be constrained by the underlying
> database. To speed up the process significantly, you may need to do work on
> the da
I believe this discussion should be taken offlist as it no longer
seems to be concerned with R.
-- Bert Gunter
Bert Gunter
Genentech Nonclinical Biostatistics
(650) 467-7374
"Data is not information. Information is not knowledge. And knowledge
is certainly not wisdom."
H. Gilbert Welch
On Tu
The dataset is not large by database standards. Even in mySQL - not known
for its speed at multi-row querying - the queries you describe should
complete within a few seconds on even moderately recent hardware if your
indexes are reasonable.
What are your performance criteria for processing these
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Exactly,
which is why I am looking for something faster :-)-O
el
on 2014-05-06, 15:21 David R Forrest said the following:
> It sounds as if your underlying MySQL database is too slow for your
> purposes. Whatever you layer on top of it will be cons
It sounds as if your underlying MySQL database is too slow for your purposes.
Whatever you layer on top of it will be constrained by the underlying database.
To speed up the process significantly, you may need to do work on the database
backend part of the process.
Dave
On May 6, 2014, at 7
Thanks,
tried all of that, too slow.
el
on 2014-05-06, 12:00 Gabor Grothendieck said the following:
> On Tue, May 6, 2014 at 5:12 AM, Dr Eberhard Lisse wrote:
>> Jeff
>>
>> It's in MySQL, at the moment roughly 1.8 GB, if I pull it into a
>> dataframe it saves to 180MB. I work from the dataframe
On Tue, May 6, 2014 at 5:12 AM, Dr Eberhard Lisse wrote:
> Jeff
>
> It's in MySQL, at the moment roughly 1.8 GB, if I pull it into a
> dataframe it saves to 180MB. I work from the dataframe.
>
> But, it's not only a size issue it's also a speed issue and hence I
> don't care what I am going to use
David,
this is quite slow :-)-O
el
on 2014-05-06, 10:55 David McPearson said the following:
[...]
> It seems like you are trying to extract a (relatively) small data set from a
> much larger SQL databaseWhy not do the SQL stiff in the database and the
> analysis *statsm graphics...) in R? Maybe
On Tue, 6 May 2014 10:12:50 +0100 Dr Eberhard Lisse wrote
> Jeff
>
> It's in MySQL, at the moment roughly 1.8 GB, if I pull it into a
> dataframe it saves to 180MB. I work from the dataframe.
>
> But, it's not only a size issue it's also a speed issue and hence I
> don't care what I am going to
Hi,
Yes dplyr syntax is quite equivalent to SQL, although it is faster.
Another alternative you could consider is to use *data.table* which has a
syntax very similar to the way you select subset within a data.frame and in
terms of performance is faster (a bit) than sqldf.
You can get some idea of
Jeff
It's in MySQL, at the moment roughly 1.8 GB, if I pull it into a
dataframe it saves to 180MB. I work from the dataframe.
But, it's not only a size issue it's also a speed issue and hence I
don't care what I am going to use, as long as it is fast.
sqldf is easy to understand for me but it ta
In what format is this "growing" data stored? CSV? SQL? Log textfile? You say
you don't want to use sqldf, but you haven't said what you do want to use.
---
Jeff NewmillerThe . . Go L
Thank you.
My requirements are that simple. One table, 11 fields, of which 3 are
interesting, 30 Million records, growing daily by between 30.
And, yes I have spent an enormous amount of time reading these things,
but for someone not dealing with this professionally and/or on a daily
basis, t
On Fri, May 2, 2014 at 5:23 PM, Dr Eberhard Lisse wrote:
> Hi,
>
> How do I do something like this without using sqldf?
>
> a <- sqldf("SELECT COUNT(*) FROM b WHERE c = 'd'")
>
> or
>
> e <- sqldf("SELECT f, COUNT(*) FROM b GROUP BY f ORDER BY f")
>
In the examples section at the bottom of ?sqldf
On May 5, 2014, at 11:44 AM, Dr Eberhard Lisse wrote:
> I do not wish to prolong this metadiscussion but I remain confused
> by your advice:
>
> 1) You don't understand what I asked (ie would have to parse two
> simple SQL statements)
Correct ... at least for me. I could have guessed at what th
intent didn't
seem justified by the level of courtesy offered by the questioner.
--
David.
>
> -Original Message-
> From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On
> Behalf Of Bert Gunter
> Sent: Saturday, May 03, 2014 1:12 AM
> To: D
?table
?aggregate
Also, packages plyr, data.table, and dplyr. You might consider reading [1], but
if your interests are really as simple as your examples then the table function
should be sufficient. That function is discussed in the Introduction to R
document that you really should have read b
1:12 AM
To: Dr Eberhard Lisse
Cc: r
Subject: Re: [R] SQL vs R
By making the effort to learn R?
See e.g. the "Introduction to R" tutorial that ships with R.
-- Bert
Bert Gunter
Genentech Nonclinical Biostatistics
(650) 467-7374
"Data is not information. Information is not kn
On 04/05/14 09:58, Sarah Goslee wrote:
Personally, I'm not sure how much work someone who appears to have not
read the posting guide should really expect the list to do on his
behalf. But snarky replies to reasonable requests to read the
documentation are easier than doing one's own work.
We
On 04/05/14 10:16, Dr Eberhard W Lisse wrote:
Google Pressdram :-)-O
el
On 2014-05-03, 23:42 , Rolf Turner wrote:>
On 04/05/14 00:05, Dr Eberhard W Lisse wrote:
Thank you very much, Mr Arkell.
I don't get it. Can anyone explain the (joke? allusion?) ?
Thank you.
cheers,
Rolf Turner
_
Google Pressdram :-)-O
el
On 2014-05-03, 23:42 , Rolf Turner wrote:>
> On 04/05/14 00:05, Dr Eberhard W Lisse wrote:
>
>> Thank you very much, Mr Arkell.
>
> I don't get it. Can anyone explain the (joke? allusion?) ?
>
> cheers,
>
> Rolf Turner
>
__
R
On Sat, May 3, 2014 at 5:42 PM, Rolf Turner wrote:
>
> On 04/05/14 00:05, Dr Eberhard W Lisse wrote:
>
>> Thank you very much, Mr Arkell.
>
>
> I don't get it. Can anyone explain the (joke? allusion?) ?
I believe it's a moderately offensive reply from someone who feels
unfairly dismissed, derive
On 04/05/14 00:05, Dr Eberhard W Lisse wrote:
Thank you very much, Mr Arkell.
I don't get it. Can anyone explain the (joke? allusion?) ?
cheers,
Rolf Turner
On 2014-05-03, 07:11 , Bert Gunter wrote:
By making the effort to learn R?
See e.g. the "Introduction to R" tutorial that ships w
Thanks,
will try to figure this out :-)-O
el
On 2014-05-03, 06:40 , Carlos Ortega wrote:
> Hi,
>
> With the new package "dplyr" you can create equivalent SQL sintaxt
> queries like the one you need.
> You can find examples of how to apply it here:
>
> http://martinsbioblogg.wordpress.com/2014/
Thank you very much, Mr Arkell.
el
On 2014-05-03, 07:11 , Bert Gunter wrote:
> By making the effort to learn R?
>
> See e.g. the "Introduction to R" tutorial that ships with R.
>
> -- Bert
>
> Bert Gunter
> Genentech Nonclinical Biostatistics
> (650) 467-7374
>
> "Data is not information. Inf
By making the effort to learn R?
See e.g. the "Introduction to R" tutorial that ships with R.
-- Bert
Bert Gunter
Genentech Nonclinical Biostatistics
(650) 467-7374
"Data is not information. Information is not knowledge. And knowledge
is certainly not wisdom."
H. Gilbert Welch
On Fri, May 2
Hi,
With the new package "dplyr" you can create equivalent SQL sintaxt queries
like the one you need.
You can find examples of how to apply it here:
http://martinsbioblogg.wordpress.com/2014/03/26/using-r-quickly-calculating-summary-statistics-with-dplyr/
http://martinsbioblogg.wordpress.com/201
Hi,
How do I do something like this without using sqldf?
a <- sqldf("SELECT COUNT(*) FROM b WHERE c = 'd'")
or
e <- sqldf("SELECT f, COUNT(*) FROM b GROUP BY f ORDER BY f")
greetings, el
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mail
29 matches
Mail list logo
