Re: [PERFORM] index structure for 114-dimension vector

2007-04-26 Thread Arjen van der Meijden
On 21-4-2007 1:42 Mark Kirkwood wrote: I don't think that will work for the vector norm i.e: |x - y| = sqrt(sum over j ((x[j] - y[j])^2)) I don't know if this is usefull here, but I was able to rewrite that algorithm for a set of very sparse vectors (i.e. they had very little overlapping fac

Re: [PERFORM] Fragmentation of WAL files

2007-04-26 Thread Greg Smith
On Thu, 26 Apr 2007, Bill Moran wrote: I've seen marketing material that claims that modern NTFS doesn't suffer performance problems from fragmentation. You're only reading half of the marketing material then. For a balanced picture, read the stuff generated by the companies that sell defrag

Re: [PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread Oleg Bartunov
Folks, we in astronomy permanently work with billiards objects with spherical atributes and have several sky-indexing schemes. See my page for links http://www.sai.msu.su/~megera/wiki/SkyPixelization We have q3c package for PostgreSQL available from q3c.sf.net, which we use in production with t

Re: [PERFORM] index structure for 114-dimension vector

2007-04-26 Thread Oleg Bartunov
On Fri, 27 Apr 2007, Alexander Staubo wrote: On 4/20/07, Andrew Lazarus <[EMAIL PROTECTED]> wrote: I have a table with 2.5 million real[] arrays. (They are points in a time series.) Given a new array X, I'd like to find, say, the 25 closest to X in some sense--for simplification, let's just say

[PERFORM] Usage up to 50% CPU

2007-04-26 Thread Andres Retzlaff
Hi, I have pg 8.1.4 running in Windows XP Pro wirh a Pentium D and I notice that I can not use more than 50% of the cpus (Pentium D has 2 cpus), how can I change the settings to use the 100% of it. Regards, Andrew Retzlaff _ Adv

Re: [PERFORM] Feature Request --- was: PostgreSQL Performance Tuning

2007-04-26 Thread Tom Lane
Carlos Moreno <[EMAIL PROTECTED]> writes: > ... But, wouldn't it make sense that the configure script > determines the amount of physical memory and perhaps even do a HD > speed estimate to set up defaults that are closer to a > performance-optimized > configuration? No. Most copies of Postgres

[PERFORM] Feature Request --- was: PostgreSQL Performance Tuning

2007-04-26 Thread Carlos Moreno
Steve Crawford wrote: Have you changed _anything_ from the defaults? The defaults are set so PG will run on as many installations as practical. They are not set for performance - that is specific to your equipment, your data, and how you need to handle the data. Is this really the sensible thin

Re: [PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread joe
Is there a reason you are not using postgis. The R tree indexes are designed for exactly this type of query and should be able to do it very quickly. Hope that helps, Joe > I have this table: > > CREATE TABLE test_zip_assoc ( > id serial NOT NULL, > f_id integer DEFAULT 0 NOT NULL, >

Re: [PERFORM] postgres: 100% CPU utilization

2007-04-26 Thread Mark Kirkwood
Sergey Tsukinovsky wrote: Just for the record - the hardware that was used for the test has the following parameters: AMD Opteron 2GHZ 2GB RAM LSI Logic SCSI And you ran FreeBSD 4.4 on it right? This may be a source of high cpu utilization in itself if the box is SMP or dual core, as multi-c

Re: [PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread Alexander Staubo
On 4/27/07, Alexander Staubo <[EMAIL PROTECTED]> wrote: [snip] PostGIS implements the whole GIS stack, and it's so good at this that it's practically the de facto tool among GIS analysts. Installing PostGIS into a database is simple, and once you have done this, you can augment your table with a

Re: [PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread Alexander Staubo
On 24 Apr 2007 14:26:46 -0700, zardozrocks <[EMAIL PROTECTED]> wrote: I have this table: CREATE TABLE test_zip_assoc ( id serial NOT NULL, f_id integer DEFAULT 0 NOT NULL, lat_radians numeric(6,5) DEFAULT 0.0 NOT NULL, long_radians numeric(6,5) DEFAULT 0.0 NOT NULL ); CRE

Re: [PERFORM] postgres: 100% CPU utilization

2007-04-26 Thread Scott Marlowe
On Tue, 2007-04-24 at 10:30, Sergey Tsukinovsky wrote: > Thanks for this reply, Ron. > This is almost what I was looking for. > > While the upgrade to the latest version is out of the question (which > unfortunately for me became the subject of this discussion) still, I was > looking for the ways

Re: [PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread Jeff Hoffmann
zardozrocks wrote: I have this table: CREATE TABLE test_zip_assoc ( id serial NOT NULL, f_id integer DEFAULT 0 NOT NULL, lat_radians numeric(6,5) DEFAULT 0.0 NOT NULL, long_radians numeric(6,5) DEFAULT 0.0 NOT NULL ); CREATE INDEX lat_radians ON test_zip_assoc USING btree

Re: [PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread Scott Marlowe
On Tue, 2007-04-24 at 16:26, zardozrocks wrote: > I have this table: > > CREATE TABLE test_zip_assoc ( > id serial NOT NULL, > f_id integer DEFAULT 0 NOT NULL, > lat_radians numeric(6,5) DEFAULT 0.0 NOT NULL, > long_radians numeric(6,5) DEFAULT 0.0 NOT NULL > ); Like someo

Re: [PERFORM] index structure for 114-dimension vector

2007-04-26 Thread Alexander Staubo
On 4/20/07, Andrew Lazarus <[EMAIL PROTECTED]> wrote: I have a table with 2.5 million real[] arrays. (They are points in a time series.) Given a new array X, I'd like to find, say, the 25 closest to X in some sense--for simplification, let's just say in the usual vector norm. Speed is critical he

Re: [PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread Luke Lonergan
NUMERIC operations are very slow in pgsql. Equality comparisons are somewhat faster, but other operations are very slow compared to other vendor's NUMERIC. We've sped it up a lot here internally, but you may want to consider using FLOAT for what you are doing. - Luke Msg is shrt cuz m on ma t

Re: [PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread Benjamin Minshall
zardozrocks wrote: lat_radians numeric(6,5) DEFAULT 0.0 NOT NULL, long_radians numeric(6,5) DEFAULT 0.0 NOT NULL Native data types such as integer or real are much faster than numeric. If you need 6 digits, it's better to multiply your coordinates by 10^6 and store as INTEGER

Re: [PERFORM] [GENERAL] PostgreSQL Performance Tuning

2007-04-26 Thread Steve Crawford
Shohab Abdullah wrote: > > Dear, > We are facing performance tuning problem while using PostgreSQL Database > over the network on a linux OS. > Our Database consists of more than 500 tables with an average of 10K > records per table with an average of 20 users accessing the database > simultaneous

Re: [PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread Bill Moran
In response to zardozrocks <[EMAIL PROTECTED]>: > I have this table: > > CREATE TABLE test_zip_assoc ( > id serial NOT NULL, > f_id integer DEFAULT 0 NOT NULL, > lat_radians numeric(6,5) DEFAULT 0.0 NOT NULL, > long_radians numeric(6,5) DEFAULT 0.0 NOT NULL > ); > CREATE I

Re: [PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread Merlin Moncure
On 24 Apr 2007 14:26:46 -0700, zardozrocks <[EMAIL PROTECTED]> wrote: I have this table: CREATE TABLE test_zip_assoc ( id serial NOT NULL, f_id integer DEFAULT 0 NOT NULL, lat_radians numeric(6,5) DEFAULT 0.0 NOT NULL, long_radians numeric(6,5) DEFAULT 0.0 NOT NULL ); CRE

[PERFORM] Simple query, 10 million records...MySQL ten times faster

2007-04-26 Thread zardozrocks
I have this table: CREATE TABLE test_zip_assoc ( id serial NOT NULL, f_id integer DEFAULT 0 NOT NULL, lat_radians numeric(6,5) DEFAULT 0.0 NOT NULL, long_radians numeric(6,5) DEFAULT 0.0 NOT NULL ); CREATE INDEX lat_radians ON test_zip_assoc USING btree (lat_radians); CREAT

Re: [PERFORM] postgres: 100% CPU utilization

2007-04-26 Thread Sergey Tsukinovsky
Thanks for this reply, Ron. This is almost what I was looking for. While the upgrade to the latest version is out of the question (which unfortunately for me became the subject of this discussion) still, I was looking for the ways to improve the performance of the 7.0.2 version. Extensive use of

Re: [PERFORM] index structure for 114-dimension vector

2007-04-26 Thread C Storm
On Apr 20, 12:07 pm, [EMAIL PROTECTED] (Andrew Lazarus) wrote: > I have a table with 2.5 million real[] arrays. (They are points in a > time series.) Given a new array X, I'd like to find, say, the 25 > closest to X in some sense--for simplification, let's just say in the > usualvectornorm. Speed i

Re: [PERFORM] not using indexes on large table

2007-04-26 Thread Scott Marlowe
On Sat, 2007-04-21 at 15:17, Jeroen Kleijer wrote: > Hi all, > > I'm a bit new to PostgreSQL and database design in general so forgive me > for asking stupid questions. ;-) > > I've setup a PostgreSQL database on a Linux machine (2 processor, 1GB > mem) and while the database itself resides on a

Re: Filesystem fragmentation (Re: [PERFORM] Fragmentation of WAL files)

2007-04-26 Thread Tom Lane
Gregory Stark <[EMAIL PROTECTED]> writes: > In the case of a performance-critical file like the WAL that's always read > sequentially it may be to our advantage to defeat this technique and force it > to be allocated sequentially. I'm not sure whether any filesystems provide any > option to do so.

Re: [PERFORM] [GENERAL] Fw: PostgreSQL Performance Tuning

2007-04-26 Thread Gerhard Wiesinger
Hello! I would do the following (in that order): 1.) Check for a performant application logic and application design (e.g. degree of granularity of the Java Hibernate Mapping, are there some object iterators with hundreds of objects, etc.) 2.) Check the hibernate generated queries and whether t

Re: Filesystem fragmentation (Re: [PERFORM] Fragmentation of WAL files)

2007-04-26 Thread Gregory Stark
"Craig A. James" <[EMAIL PROTECTED]> writes: > More specifically, this problem was solved on UNIX file systems way back in > the > 1970's and 1980's. No UNIX file system (including Linux) since then has had > significant fragmentation problems, unless the file system gets close to 100% > full. If

Re: Filesystem fragmentation (Re: [PERFORM] Fragmentation of WAL files)

2007-04-26 Thread Craig A. James
Bill Moran wrote: In response to Heikki Linnakangas <[EMAIL PROTECTED]>: Can anyone else confirm this? I don't know if this is a windows-only issue, but I don't know of a way to check fragmentation in unix. I can confirm that it's only a Windows problem. No UNIX filesystem that I'm aware of s

Re: [PERFORM] Fragmentation of WAL files

2007-04-26 Thread Tom Lane
> In response to Jim Nasby <[EMAIL PROTECTED]>: >> I was recently running defrag on my windows/parallels VM and noticed >> a bunch of WAL files that defrag couldn't take care of, presumably >> because the database was running. What's disturbing to me is that >> these files all had ~2000 fragm

Filesystem fragmentation (Re: [PERFORM] Fragmentation of WAL files)

2007-04-26 Thread Bill Moran
In response to Heikki Linnakangas <[EMAIL PROTECTED]>: [snip] > >> Can anyone else confirm this? I don't know if this is a windows-only > >> issue, but I don't know of a way to check fragmentation in unix. > > > > I can confirm that it's only a Windows problem. No UNIX filesystem > > that I'm

Re: [PERFORM] Fragmentation of WAL files

2007-04-26 Thread Heikki Linnakangas
Bill Moran wrote: In response to Jim Nasby <[EMAIL PROTECTED]>: I was recently running defrag on my windows/parallels VM and noticed a bunch of WAL files that defrag couldn't take care of, presumably because the database was running. What's disturbing to me is that these files all had ~200

Re: [PERFORM] Fragmentation of WAL files

2007-04-26 Thread Bill Moran
In response to Jim Nasby <[EMAIL PROTECTED]>: > I was recently running defrag on my windows/parallels VM and noticed > a bunch of WAL files that defrag couldn't take care of, presumably > because the database was running. What's disturbing to me is that > these files all had ~2000 fragments.

Re: [PERFORM] [GENERAL] Fw: PostgreSQL Performance Tuning

2007-04-26 Thread Richard Huxton
Please try to post to one list at a time. I've replied to this on the -performance list. -- Richard Huxton Archonet Ltd ---(end of broadcast)--- TIP 6: explain analyze is your friend

Re: [PERFORM] Fw: PostgreSQL Performance Tuning

2007-04-26 Thread Richard Huxton
Please try to keep postings to one mailing list - I've replied to the performance list here. Shohab Abdullah wrote: Dear, We are facing performance tuning problem while using PostgreSQL Database over the network on a linux OS. Our Database consists of more than 500 tables with an average of 10

[PERFORM] Fw: PostgreSQL Performance Tuning

2007-04-26 Thread Shohab Abdullah
Dear, We are facing performance tuning problem while using PostgreSQL Database over the network on a linux OS. Our Database consists of more than 500 tables with an average of 10K records per table with an average of 20 users accessing the database simultaneously over the network. Each table has

Re: [PERFORM] What`s wrong with JFS configuration?

2007-04-26 Thread Cosimo Streppone
Jim Nasby wrote: On Apr 25, 2007, at 8:51 AM, Paweł Gruszczyński wrote: where u6 stores Fedora Core 6 operating system, and u0 stores 3 partitions with ext2, ext3 and jfs filesystem. Keep in mind that drives have a faster data transfer rate at the outer-edge than they do at the inner edge [.

Re: [PERFORM] seeking advise on char vs text or varchar in search table

2007-04-26 Thread Gregory Stark
"Jim Nasby" <[EMAIL PROTECTED]> writes: > AIUI, char, varchar and text all store their data in *exactly* the same way in > the database; char only pads data on output, and in the actual tables it > still > contains the regular varlena header. The only reason I've ever used char in > other datab

Re: [PERFORM] What`s wrong with JFS configuration?

2007-04-26 Thread Luke Lonergan
The outer track / inner track performance ratio is more like 40 percent. Recent example is 78MB/s outer and 44MB/s inner for the new Seagate 750MB drive (see http://www.storagereview.com for benchmark results) - Luke Msg is shrt cuz m on ma treo -Original Message- From: Jim Nasby [

Re: [PERFORM] What`s wrong with JFS configuration?

2007-04-26 Thread Jim Nasby
On Apr 25, 2007, at 8:51 AM, Paweł Gruszczyński wrote: where u6 stores Fedora Core 6 operating system, and u0 stores 3 partitions with ext2, ext3 and jfs filesystem. Keep in mind that drives have a faster data transfer rate at the outer-edge than they do at the inner edge, so if you've got a

Re: [PERFORM] seeking advise on char vs text or varchar in search table

2007-04-26 Thread Jim Nasby
On Apr 23, 2007, at 7:16 AM, Merlin Moncure wrote: On 4/20/07, chrisj <[EMAIL PROTECTED]> wrote: I have a table that contains a column for keywords that I expect to become quite large and will be used for web searches. I will either index the column or come up with a simple hashing algorit

[PERFORM] Fragmentation of WAL files

2007-04-26 Thread Jim Nasby
I was recently running defrag on my windows/parallels VM and noticed a bunch of WAL files that defrag couldn't take care of, presumably because the database was running. What's disturbing to me is that these files all had ~2000 fragments. Now, this was an EnterpriseDB database which means t