Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-29 Thread Jeff Janes
On Sat, Jul 26, 2014 at 3:46 AM, Jiří Nádvorník wrote: > The reason why I solve the performance issues here is that the table of > observations has atm cca 3e8 rows after 1.5 year of gathering the data. The > number growth is linear. > So about 500,000 new records a day.

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-29 Thread Jiří Nádvorník
July 28, 2014 5:03 PM To: Oleg Bartunov Cc: Jiří Nádvorník; pgsql-performance@postgresql.org Subject: Re: [PERFORM] Cursor + upsert (astronomical data) Hi Jiri, I understand your problem (and I actually have exactly the same in my sky monitoring experiment). Unfortunately, I have no comple

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-29 Thread Craig James
Hi Jiri, I’m really interested in those [clustering] algorithms and study them. But > I would need somebody to point me directly at a specific algorithm to look > at. The main problem with choosing the right one (which couldn’t get over > even my university teacher) is that you don’t know the numb

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-29 Thread Jiří Nádvorník
: Jiří Nádvorník Cc: pgsql-performance@postgresql.org; Sergey Karpov Subject: Re: [PERFORM] Cursor + upsert (astronomical data) Jiri, as I understand your problem is called crossmatch ? I attach pdf of our work in progress, where we compared several spatial indexing techniques, including postgis

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-29 Thread Jiří Nádvorník
mance@postgresql.org Subject: Re: [PERFORM] Cursor + upsert (astronomical data) I am not sure I understand the problem fully, e.g. what to do if there are observations A,B and C with A to B and B to C less then treshold and A to C over treshold, but anyway. Could you first apply a kind of g

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Vitalii Tymchyshyn
s on one image with another > 999 images overlapping it means 1 *1000^2). > > > > Best regards, > > > > Jiri Nadvornik > > > > *From:* tiv...@gmail.com [mailto:tiv...@gmail.com] *On Behalf Of *Vitalii > Tymchyshyn > *Sent:* Sunday, July 27, 2014 8:06 A

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Marc Mamin
[Craig] >>If you haven't looked at clustering algorithms yet, you might want to do so. >>Your problem is a special case of clustering, where you have a large number >>of small clusters. A good place to start is the overview on Wikipedia: >>http://en.wikipedia.org/wiki/Cluster_analysis According t

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Craig James
r > 999 images overlapping it means 1 *1000^2). > > > > Best regards, > > > > Jiri Nadvornik > > > > *From:* tiv...@gmail.com [mailto:tiv...@gmail.com] *On Behalf Of *Vitalii > Tymchyshyn > *Sent:* Sunday, July 27, 2014 8:06 AM > *To:* Jiří Nádvorník > *

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Jiří Nádvorník
: Jiří Nádvorník Cc: pgsql-performance@postgresql.org Subject: Re: [PERFORM] Cursor + upsert (astronomical data) I am not sure I understand the problem fully, e.g. what to do if there are observations A,B and C with A to B and B to C less then treshold and A to C over treshold, but anyway. Could

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-26 Thread Vitalii Tymchyshyn
I am not sure I understand the problem fully, e.g. what to do if there are observations A,B and C with A to B and B to C less then treshold and A to C over treshold, but anyway. Could you first apply a kind of grid to your observations? What I mean is to round your coords to, say, 1/2 arcsec on ea

[PERFORM] Cursor + upsert (astronomical data)

2014-07-26 Thread Jiří Nádvorník
Hello guys. My issue kind of hits multiple topics, but the main question is about performance. I think you need to understand the background a little bit to be able to help me. So I will firstly define the problem and my solutions to it and place the questions for you to the end of this message