Re: efficient way to process data

2014-01-14 Thread Chris Angelico
On Wed, Jan 15, 2014 at 3:23 AM, Larry Martell wrote: >> As far as I'm concerned, you won geek cred the moment you said >> "electron microscope", and "semiconductor wafers as they are being >> manufactured" is just gravy. > > Thanks! You made my day. I showed this to my wife and she asked "Is that

Re: efficient way to process data

2014-01-14 Thread Larry Martell
On Tue, Jan 14, 2014 at 7:26 AM, Chris Angelico wrote: > On Wed, Jan 15, 2014 at 1:18 AM, Larry Martell > wrote: >> if you're interested in what the application is, this is data >> collected with an electron microscope from semiconductor wafers as >> they are being manufactured. The x and y are

Re: efficient way to process data

2014-01-14 Thread Chris Angelico
On Wed, Jan 15, 2014 at 1:18 AM, Larry Martell wrote: > if you're interested in what the application is, this is data > collected with an electron microscope from semiconductor wafers as > they are being manufactured. The x and y are the position on the wafer > that the data was collected, in micr

Re: efficient way to process data

2014-01-14 Thread Larry Martell
On Mon, Jan 13, 2014 at 11:32 AM, Chris Angelico wrote: > On Tue, Jan 14, 2014 at 5:27 AM, Larry Martell > wrote: >> Thanks. Unfortunately this has been made a low priority task and I've >> been put on to something else (I hate when they do that). > > Ugh, I know that feeling all too well! Life'

Re: efficient way to process data

2014-01-13 Thread Petite Abeille
On Jan 13, 2014, at 7:42 PM, Mark Lawrence wrote: > I've not followed this thread closely but would this help > http://pandas.pydata.org/ ? When and if you get back to it, that is!!! I doubt it. The mean overhead by far would be to shuffle pointless data between the server & client. Best to

Re: efficient way to process data

2014-01-13 Thread Larry Martell
On Mon, Jan 13, 2014 at 1:32 PM, Chris Angelico wrote: > On Tue, Jan 14, 2014 at 5:27 AM, Larry Martell > wrote: >> Thanks. Unfortunately this has been made a low priority task and I've >> been put on to something else (I hate when they do that). > > Ugh, I know that feeling all too well! Right

Re: efficient way to process data

2014-01-13 Thread Mark Lawrence
On 13/01/2014 18:27, Larry Martell wrote: On Mon, Jan 13, 2014 at 1:09 AM, Chris Angelico wrote: On Mon, Jan 13, 2014 at 2:35 PM, Larry Martell wrote: Thanks for the reply. I'm going to take a stab at removing the group by and doing it all in python. It doesn't look too hard, but I don't know

Re: efficient way to process data

2014-01-13 Thread Mark Lawrence
On 13/01/2014 18:32, Chris Angelico wrote: On Tue, Jan 14, 2014 at 5:27 AM, Larry Martell wrote: Thanks. Unfortunately this has been made a low priority task and I've been put on to something else (I hate when they do that). Ugh, I know that feeling all too well! Life's better when you're une

Re: efficient way to process data

2014-01-13 Thread Chris Angelico
On Tue, Jan 14, 2014 at 5:27 AM, Larry Martell wrote: > Thanks. Unfortunately this has been made a low priority task and I've > been put on to something else (I hate when they do that). Ugh, I know that feeling all too well! Life's better when you're unemployed, and you can choose the interesting

Re: efficient way to process data

2014-01-13 Thread Larry Martell
On Mon, Jan 13, 2014 at 1:09 AM, Chris Angelico wrote: > On Mon, Jan 13, 2014 at 2:35 PM, Larry Martell > wrote: >> Thanks for the reply. I'm going to take a stab at removing the group >> by and doing it all in python. It doesn't look too hard, but I don't >> know how it will perform. > > Well,

Re: efficient way to process data

2014-01-12 Thread Chris Angelico
On Mon, Jan 13, 2014 at 2:35 PM, Larry Martell wrote: > Thanks for the reply. I'm going to take a stab at removing the group > by and doing it all in python. It doesn't look too hard, but I don't > know how it will perform. Well, if you can't switch to PostgreSQL or such, then doing it in Python

Re: efficient way to process data

2014-01-12 Thread Larry Martell
On Sun, Jan 12, 2014 at 6:27 PM, Chris Angelico wrote: > On Mon, Jan 13, 2014 at 6:23 AM, Larry Martell > wrote: >> I have an python app that queries a MySQL DB. The query has this form: >> >> SELECT a, b, c, d, AVG(e), STD(e), CONCAT(x, ',', y) as f >> FROM t >> GROUP BY a, b, c, d, f >> >> x a

Re: efficient way to process data

2014-01-12 Thread Larry Martell
On Sun, Jan 12, 2014 at 5:43 PM, Dennis Lee Bieber wrote: > On Sun, 12 Jan 2014 14:23:17 -0500, Larry Martell > declaimed the following: > >>I have an python app that queries a MySQL DB. The query has this form: >> >>SELECT a, b, c, d, AVG(e), STD(e), CONCAT(x, ',', y) as f >>FROM t >>GROUP BY a,

Re: efficient way to process data

2014-01-12 Thread Larry Martell
On Sun, Jan 12, 2014 at 5:18 PM, Chris Angelico wrote: > On Mon, Jan 13, 2014 at 6:53 AM, Petite Abeille > wrote: >> On Jan 12, 2014, at 8:23 PM, Larry Martell wrote: >> >>> AFAIK, there is no way to do this in SQL. >> >> Sounds like a job for window functions (aka analytic functions) [1][2]. >

Re: efficient way to process data

2014-01-12 Thread Larry Martell
On Sun, Jan 12, 2014 at 2:53 PM, Petite Abeille wrote: > > On Jan 12, 2014, at 8:23 PM, Larry Martell wrote: > >> AFAIK, there is no way to do this in SQL. > > Sounds like a job for window functions (aka analytic functions) [1][2]. > > [1] http://www.postgresql.org/docs/9.3/static/tutorial-window

Re: efficient way to process data

2014-01-12 Thread Chris Angelico
On Mon, Jan 13, 2014 at 6:23 AM, Larry Martell wrote: > I have an python app that queries a MySQL DB. The query has this form: > > SELECT a, b, c, d, AVG(e), STD(e), CONCAT(x, ',', y) as f > FROM t > GROUP BY a, b, c, d, f > > x and y are numbers (378.18, 2213.797 or 378.218, 2213.949 or > 10053.4

Re: efficient way to process data

2014-01-12 Thread Chris Angelico
On Mon, Jan 13, 2014 at 6:53 AM, Petite Abeille wrote: > On Jan 12, 2014, at 8:23 PM, Larry Martell wrote: > >> AFAIK, there is no way to do this in SQL. > > Sounds like a job for window functions (aka analytic functions) [1][2]. That's my thought too. I don't think MySQL has them, though, so it

Re: efficient way to process data

2014-01-12 Thread Petite Abeille
On Jan 12, 2014, at 8:23 PM, Larry Martell wrote: > AFAIK, there is no way to do this in SQL. Sounds like a job for window functions (aka analytic functions) [1][2]. [1] http://www.postgresql.org/docs/9.3/static/tutorial-window.html [2] http://docs.oracle.com/cd/E11882_01/server.112/e26088/fu

efficient way to process data

2014-01-12 Thread Larry Martell
I have an python app that queries a MySQL DB. The query has this form: SELECT a, b, c, d, AVG(e), STD(e), CONCAT(x, ',', y) as f FROM t GROUP BY a, b, c, d, f x and y are numbers (378.18, 2213.797 or 378.218, 2213.949 or 10053.490, 2542.094). The business issue is that if either x or y in 2 rows