Re: [PERFORM] Performance issues with large amounts of time-series data

2009-09-01 Thread हृषीकेश मेहेंदळ े
Hi Tom, Greg, Thanks for your helpful suggestions - switching the BIGINT to FLOAT and fixing the postgresql.conf to better match my server configuration gave me about 30% speedup on the queries. Because of the fact that my data insert order was almost never the data retrieval order, I also got a

Re: [PERFORM] Performance issues with large amounts of time-series data

2009-08-27 Thread Greg Smith
On Wed, 26 Aug 2009, Hrishikesh (??? ) wrote: key = {device_id (uint64), identifier (uint32), sub_identifier (uint32), unix_time} (these four taken together are unique) You should probably tag these fields as NOT NULL to eliminate needing to consider that possibility during query

Re: [PERFORM] Performance issues with large amounts of time-series data

2009-08-26 Thread Greg Stark
2009/8/26 Tom Lane : >> How does a float ("REAL") compare in terms of SUM()s ? > > Casting to float or float8 is certainly a useful alternative if you > don't mind the potential for roundoff error.  On any non-ancient > platform those will be considerably faster than numeric.  BTW, > I think that 8

Re: [PERFORM] Performance issues with large amounts of time-series data

2009-08-26 Thread Tom Lane
=?UTF-8?B?SHJpc2hpa2VzaCAo4KS54KWD4KS34KWA4KSV4KWH4KS2IOCkruClh+CkueClh+CkguCkpuCksw==?= =?UTF-8?B?4KWHKQ==?= writes: > 2009/8/26 Tom Lane >> Do the data columns have to be bigint, or would int be enough to hold >> the expected range? > For the 300-sec tables I probably can drop it to an intege

Re: [PERFORM] Performance issues with large amounts of time-series data

2009-08-26 Thread हृषीकेश मेहेंदळ े
Hi Tom, Thanks for your quick response. 2009/8/26 Tom Lane > writes: > > In my timing tests, the performance of PG is quite a lot worse than the > > equivalent BerkeleyDB implementation. > > Are you actually comparing apples to apples?  I don't recall that BDB > has any built-in aggregation fun

Re: [PERFORM] Performance issues with large amounts of time-series data

2009-08-26 Thread Tom Lane
=?UTF-8?B?SHJpc2hpa2VzaCAo4KS54KWD4KS34KWA4KSV4KWH4KS2IOCkruClh+CkueClh+CkguCkpuCksw==?= =?UTF-8?B?4KWHKQ==?= writes: > In my timing tests, the performance of PG is quite a lot worse than the > equivalent BerkeleyDB implementation. Are you actually comparing apples to apples? I don't recall tha

[PERFORM] Performance issues with large amounts of time-series data

2009-08-26 Thread हृषीकेश मेहेंदळ े
Hi All, We are improving our network appliance monitoring system, and are evaluating using PostgreSQL as the back-end traffic statistics database (we're currently running a home-grown Berkeley-DB based statistics database). We log data from various network elements (it's mainly in/out bytes and p