RE: [GENERAL] LARGE db dump/restore for upgrade question
> Any suggestion on how to prepare for the next upgrade would be > appreciated. I think it has to be said that if you want decent performance on excessively large (50GB+) databases, you're going to need excessively good hardware to operate it on. Buy a 3ware IDE RAID controller (www.hypermicro.com) and a stripe a couple of seagate 80GB IDE drives (the new ones have an awesome transfer rate). Turn of fsync, and use softupdates or even async I/O. I am guessing though you've already done all of the above. You might have to go beefier hardware, for PCs that means DDR SDRAM/Rambus, and a 64 bit PCI 3ware card with 4 or 8 hard drives. Hope that helps Andrew ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
RE: [GENERAL] Re: Storing images in PG?
> Yes, but good is encode binary data (image) to same ascii safe > encoding like base64, else you will have problem in INSERT/UPDATE > queries. Correct me if I'm wrong, but the only thing "lztext" can't store is NULLs, which could escaped somehow with other (rare) characters. Still overhead, but you shouldn't be storing large pieces of binary data this way, I think. - Andrew ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[GENERAL] RE: Re: Storing images in PG?
> > Correct me if I'm wrong, but the only thing "lztext" can't store is > > NULLs, which could escaped somehow with other (rare) characters. > > Still overhead, but you shouldn't be storing large pieces of binary > > data this way, I think. > > Here is not a problem with storing binary data to some > datetype (like bytea), but problem is with other parts of PG. > For example libpq > functions read queries as string, other thing is query > parsing where is needful set correct chars (like quote), etc. > IMHO if you want to work with queries with no problems and limits is Right, and I'm saying you can save that space by simply escaping all your characters in the normal way, e.g. SELECT '\377' is the ASCII character 0xFF. This would work for all characters except NULL, which you could represent by \377\377 (and you could represent a real \377 by \377\376 or something). Then the space wastage would be minimal compared to using base 64. I wrote a perl module which wraps around Pg, in which you can use parameters in queries, and it automatically escapes extended ascii characters as per above, however I have never tried storing binary data using the whole 0x01 -> 0xFF range as I just described, but I think it would work. - Andrew ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
RE: [GENERAL] Storing images in PG?
> I have found (and confirmed by studying the PostgreSQL > source) that to reliably insert arbitrary binary data into a > bytea column there are only 3 characters which need to be > escaped: \000, \047 ( ' ), \134 ( \ ). Here's the PHP > function that I've been using: Postgresl, in treating things as strings, handles \000 as NULL as an end of string. select 'abc\000def' as hehehe; hehehe abc (1 row) How do you get the data back from a query, with all the nulls in it? - Andrew ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
RE: [GENERAL] Finding database names for overnight vacuum
Out of curiosity, whats wrong with vacuumdb -a? ("Vacuum all databases.") > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]] On Behalf Of > Stephen Davies > Sent: Monday, 20 August 2001 7:33 PM > To: [EMAIL PROTECTED] > Subject: [GENERAL] Finding database names for overnight vacuum > > > G'day all. > > I have a cron job which vacuums etc all of my databases but > it does not > work with 7.1.2 because of the change in structure of .../data/base. > > It currently says: > > for q in /usr/local/pgsql/data/base/*; do > blaa > blaa > done > > How does one get an equivalent list of all databases from > 7.1.2? (and where is it documented?) ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
RE: [GENERAL] Is it possible to work with big databases (up to 10Gb) under Postgres?
I believe if you throw decent hardware at it (the sort of hardware you need to throw at 10GB database to make it perform well) then it is definitely worth giving it a go. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]] On Behalf Of Ivan Babikov > Sent: Monday, 20 August 2001 9:06 PM > To: [EMAIL PROTECTED] > Subject: [GENERAL] Is it possible to work with big databases > (up to 10Gb) under Postgres? > > > Hello All! > > Maybe this is one of those typical questions, but I have > heard people complaining that Postgres is just for quite > small bases, less than 2Gb. > > Now, we have to choose a free database engine for an > inexpensive branch of our project and Interbase looks better > in sense of capability to work with quite big bases (up to > 10-20Gb). I am not sure that Postgres will work fine with > databases greater than 10Gb. What does All think of it? > > Thanks in advance, Ivan Babikoff. > > > > > > ---(end of > broadcast)--- > TIP 4: Don't 'kill -9' the postmaster > ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
RE: [GENERAL] Re: SELECT FOR UPDATE
I prefer the way Notes (for example) handles it. All records/documents/views are in read-only mode until the user indicates they actually want to edit. They then exclusively lock that record for editing, with optional timeouts (in case their workstation crashes or whatever). This turns out to work well in many situations where you realise the number of times you want to edit compared to the number of times you want to view, is quite small. Stops users having to worry whether anyone else is editing the record at the same time - the system simply won't let them - no loss of data. Andrew ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
RE: MySQL's (false?) claims... (was: Re: [GENERAL] PL/java?)
> We can always ask them to change things. The thing which > strike me as wrong the most is the stability issue with > PostgreSQL. I've only very rarely heard reports by anyone > saying MySQL was more stable than PostgreSQL for them. Yeah, saying mysql is more stable than postgres is a complete joke from my own experiences and those around me. Also, I think people move from mysql to postgres, rarely the other way round.. - Andrew ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
RE: [GENERAL] raw partition
> It wouldn't be a very bad idea for systems where mmap is > noticeably faster than read/write using syscalls. > Unfortunately on some of those systems mmap is broken for > multiple processes mapping the same file...:) Yeah, fair enough. But mmap works well on the more popular platforms used for PostgreSQL. And it can't *hurt* performance, and its probably worth doing simply so that PostgreSQL "plays nicely" with other applications using the VM resources on a particular system, instead of the "fixed size buffer cache" approach. > But if someone wants to work on it, this would be fairly > modest-sized project that only affects bufmgr... Interesting.. might be worth taking a look at.. - Andrew ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
RE: [GENERAL] MySQL treads belong else where.
> So far I haven't seen any real flames or insults thrown about > yet - no MYsql will kill your puppy threads at all. There > can be rigourous discussion without insults being thrown and > so far that's all I've seen. No, but it WILL kill your data, if the DB is big enough and has been in use long enough.. speaking from personal experience *weep* ;-) I used to think mysql was great while I was learning SQL, because it was free, unbloated, and easy to use. But I think one grows out of it for larger and more complex projects... Postgres' larger set of features and sophisticated locking become invaluable. I know (from reading this list) that people sometimes have trouble with PGSQL, but quite honestly I found that after I had read the Postgresql docs in their entirety, everything "fit" and seemed intuitive. (I found it not unlike the feeling of becoming a FreeBSD user after having used linux...) For the project as a whole I guess its a matter of killing off the genuine FUD with one hand, and fixing the areas in Postgres that MySQL does have genuinely have an advantage, with the other hand. - Andrew ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [GENERAL] Indexes and Timestamp Comparisons
Sequential scans are always going to be faster than using the index if a large number of rows are going to be returned. Consider this: if you are doing regular DELETEs, will there be that many rows selected for deletion each time? Probably not, so Postgres will most likely use the index for those regular DELETEs. Take this into account, and pick some new, more realistic "WHERE" clauses for testing, and hopefully that will be the case. - Andrew > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]] On Behalf Of Josh Voils > Sent: Tuesday, 28 August 2001 4:41 AM > To: [EMAIL PROTECTED] > Subject: [GENERAL] Indexes and Timestamp Comparisons > > > I have a table called session: > Table "session" > Attribute | Type | Modifier > +--+-- > sessionid | character(32)| not null > usernumber | integer | not null > timestamp | timestamp with time zone | not null > > I have two indices on this table: moo, a btree on timestamp, > and session_pkey, the implicit unique btree on sessionid. > > I need to periodically eliminate timed-out entries from this > table. This is where my problem is. I created a test table > that has a million entries in it. I try to select the valid > session ids and it uses an index scan with moo; I try to > select the invalid sessionids and it uses a sequential scan > which is considerably slower. I was also playing around with > some other timestamps to compare values and found that just > by changing the > to a < I would change whether it used a seq > scan or an index scan (it wasn't consistent though as to > which scan it used for which boolean. Any suggestions on > what I can do? I vacuum analyzed the table right before I > performed these explains: > > This is a where for invalid sessionids. This is the one I > actually need to be able to run. main=# explain delete from > session where timestamp < '2001-08-27 12:26:03-05'::timestamp; > NOTICE: QUERY PLAN: > > Seq Scan on session (cost=0.00..22815.21 rows=999257 width=6) > > EXPLAIN > This is a where for valid sessionids. > main=# explain delete from session where timestamp > > '2001-08-27 12:26:03-05'::timestamp; > NOTICE: QUERY PLAN: > > Index Scan using moo on session (cost=0.00..3653.36 > rows=1000 width=6) > > EXPLAIN > > These are a couple of random wheres. If it's of any > consequence, this timestamp is before all of the timestamps > in the table. main=# explain delete from session where > timestamp < '2000-08-27 12:26:03-05'::timestamp; > NOTICE: QUERY PLAN: > > Index Scan using moo on session (cost=0.00..3653.36 > rows=1000 width=6) > > EXPLAIN > main=# explain delete from session where timestamp > > '2000-08-27 12:26:03-05'::timestamp; > NOTICE: QUERY PLAN: > > Seq Scan on session (cost=0.00..22815.21 rows=999257 width=6) > > EXPLAIN > > > Any help would be appreciated, > > Josh Voils > > > ---(end of > broadcast)--- > TIP 1: subscribe and unsubscribe commands go to > [EMAIL PROTECTED] > ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [GENERAL] Perl won't eval PgSQL boolean value
When comparing strings in perl you usually use the eq operator, not == e.g. if($cust_data->{hold} eq 't') - Andrew > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]] On Behalf Of Randall Perry > Sent: Monday, 3 September 2001 10:01 AM > To: [EMAIL PROTECTED] > Subject: [GENERAL] Perl won't eval PgSQL boolean value > > > I've got an if statement that checks if a boolean value is true: > > if ($cust_data->{'hold'} eq 't') > > But perl will not evaluate the value. $cust_data->{'hold'} is > taken from a PgSQL boolean field (either t or f). If I use > the construct above it tell me that 'eq' is not defined > thinking it's a string. > > If I use: > if ($cust_data->{'hold'} == 't') > > I get an error saying $cust_data->{'hold'} is a non-numeric value. > > If I do this: > if ($cust_data->{'hold'}) > > it will always evaluate to true whether the value is 't' or 'f'. > > > HELP. How do you evaluate boolean values from PgSQL in Perl? > > > -- > Randy Perry > sysTame > Mac Consulting/Sales > > phn 561.589.6449 > mobile email[EMAIL PROTECTED] > > > > > ---(end of > broadcast)--- > TIP 4: Don't 'kill -9' the postmaster > ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [GENERAL] PL/java?
> | Its not a perception. Java is still a dog. > > Back it up or back out please. > > The most scalable and stable enterprise solutions out there are today > running Java. In Java you actually get more time to concentrate on > removing the real performance bottlenecks of your application. Hehe OK, imagine if the whole of PostgreSQL was written in Java. Yeah, we'd be able to really remove its performance bottlenecks then. Really, I think we're all convinced on that one ;-) Having said that if people want PL/Java then let them write it. Its just another option and that can't hurt... - Andrew ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
RE: [GENERAL] count & distinct
> Hi people. I would like to make a query that > tells me how many distinct values there are > in a column. Standard count doesn't seems > to support a count distinct option. > select distint count(*) of course doens't > work (distinti clause is applied after the > result are calculated). I've tried to define > a view, but it didn't worked ( > create view distValues as select distinct ... > but views doesn't support distinct clause)... SELECT COUNT(DISTINCT field_name) FROM table_name; Works for me in v7. Regards, Andrew Snow [EMAIL PROTECTED]
RE: [GENERAL] Lock rekord
> I usually prefer the following trick for preventing long locking > times. On every > table I define a timestamp field which is updated every time the record is > written to the database. If a user edits a record (without > locking) and commit his changes > the timestamp is returned from the client program unchanged. The program > reads the record again for update and compares the timestamp from > the database > and the timestamp from the user program. If the timestamp has changed > in the meantime the record was updated from someone else and the > transaction will be rejected. What happens if someone else updates the record *just* after the record is reread for update and timestamp compared? - Andrew
RE: [GENERAL] Two questions in a row
> > 1. How do you remove one of two identical rows > > from a pgsql table ? > > DELETE FROM t1 WHERE wil do the stuff for you. If you don't know > the value into the duplicate field just export the database with pg_dump > create > a unique index onto this field and reimport all your data. Duplicate > data will > not be inserted. Another way could be to SELECT oid FROM table WHERE ...; and then delete based on that. However I believe it is possible to have two rows have the same oid but usually that isn't the case.
RE: [GENERAL] 4 billion record limit?
> Simply waiting for 64bit numbers is rather inelegant and also > presumes usage > parameters for the database... remember Bill Gates saying that he couldn't > foresee any usage for more than 64MB of RAM? I've heard this before and I just don't agree. 64MB ram, perhaps, but who is going to need 64 * 2^32? The magnitude of increase is fairly vast! I probably need not mention that a 32 bit value can store up to 4,294,967,296 where a 64 bit number can store a number that is 4,294,967,296 times as big. If 32 bit wasn't enough to keep you happy for more than a few years, a 64 bit oid really should be enough for anyone for long enough that you won't be using the same software/hardware any more. Similar to how a 32 bit unix time in seconds is only good for another ~40 years, but a 64 bit one will keep us going for billions of years. I guess the real issue is rewriting the system so that the type of oid can be chosen at compile time, so you can use whatever damn one you feel like. I would also guess that the majority of systems out there using the latest versions of postgres, already have compiler support for 64 bit integers. So when someone gets around to doing the necessary work, everything will be nice. - Andrew P.S. IMHO if you can't afford to do a drop and reload once in a while, Postgres isn't a good choice at the moment for your application.
RE: [GENERAL] 4 billion record limit?
> My mistake! ;-) > I remember wondering who would ever need more that the 16K that > the Sinclair Spectrum could give you! To go back to my original point about putting things in perspective - increasing this by 2^32 would give you 68 terabytes of RAM. But if we can get rid of oid's altogether then I think that would be best. (At least we can make our own 64 bit serial types until they are officially implemented). - Andrew
RE: [GENERAL] Re: 4 billion record limit?
> > That's an excellent point, especially considering that *sequences* use > > an integer to hold their max_value, which is by default 2,147,483,647. > > You cannot go larger than that, either. I guess it's constrained to be > > positive. So OIDs give you more potential unique values than sequences, > > far as I can tell. What about postgres' int8 type which holds up to +/- 9 billion billion or something. - Andrew
RE: [GENERAL] Large selects handled inefficiently?
> I believe I can work around this problem using cursors (although I > don't know how well DBD::Pg copes with cursors). However, that > doesn't seem right -- cursors should be needed to fetch a large query > without having it all in memory at once... Actually, I think thats why cursors were invented in the first place ;-) A cursor is what you are using if you're not fetching all the results of a query. - Andrew
RE: [GENERAL] Inserts using plpgsql - Further
> Puzzling I am pretty sure there are some bugs in v7.1.x PL/PGSQL, they are difficult to produce and nail down though. A few of my functions that were working fine in v7.0.x broke completely. I am doing more testing and will submit some bug reports... - Andrew ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
RE: [GENERAL] XML question
If you're asking how to get XML data into PostgreSQL, for perl users: Matt Sergeant's DBIx::XML_RDB module looks handy. (I think it just got added to the FreeBSD ports collection too). http://www.xml.com/pub/a/2000/12/13/perlxmldb.html Or use any XML parser module (XML::Simple!) and pump the data into PostgreSQL using Pg or DBD. Or roll your own parser, this doesn't take long in perl! (I have successfully used the latter two methods) - Andrew > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]]On Behalf Of Tony Grant > Sent: Wednesday, 6 June 2001 12:49 AM > To: [EMAIL PROTECTED] > Subject: [GENERAL] XML question > > > hello, > > Just a couple of questions: > > - what is the _most popular_ way of storing XML data in posgresql? > > - what is the _best_ way of stocking XML data in postgresql? > > Quick and nasty opinion poll... ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[GENERAL] Problem report (dropping views that are called by other views)
I'm using PostgreSQL 6.5.3 on a FreeBSD 3.4-STABLE box (all you linux postgresql users out there, switch to freebsd already!). I had created a View A, based on an existing View B. When I dropped View B and re-created it one day, the View A did not get deleted, but as soon I went to use A again, the PGSQL backend crashed and burned. I was forced to drop View A and re-create it, to restore working order. Is this a known bug or feature or issue? Should I be worried? Thanks, Andrew
Re: [GENERAL] Postgresqlism & Vacuum?
On Fri, 14 Apr 2000, Thomas wrote: > For large 24x7 installations, it's impossible to vacuum nightly because when > postgresql is vacuuming the table is locked up, to the end-user the database > has already hung. That's right. I complained about this in a discussion with a Postgresql developer, who assured me they were working towards a fix. I really don't care whether the vacuuming is fixed so that it does not lock the table completely, or that vacuuming becomes say, a once-a-month or less frequent operation. For some reason everyone who is used to working with PostgreSQL accepts the fact that you have to vacuum nightly - to outsiders it seems like a major flaw with the system. > There has been effort to speed up the vacuuming process, but this isn't the > cure. I believe the fault lies on the optimizer. Sure, the vacuum process speed is fine for small tables, but what about the big ones where the table gets locked for 5 minutes? What a joke! > Why save on micro-seconds to use sequential scan when the table is small and > later 'forgets' that the table is now big because you didn't vacuum analyze? > Why can't the optimizer just use indexes when they are there and not > 'optimize' for special cases when the table is small to save micro-seconds? Well its more than microseconds I presume, as opening indexes involves opening files, which takes milliseconds rather than microseconds. Andrew.
Re: [GENERAL] Postgresqlism & Vacuum?
On Fri, 14 Apr 2000, Thomas wrote: > For large 24x7 installations, it's impossible to vacuum nightly because when > postgresql is vacuuming the table is locked up, to the end-user the database > has already hung. That's right. I complained about this in a discussion with a Postgresql developer, who assured me they were working towards a fix. I really don't care whether the vacuuming is fixed so that it does not lock the table completely, or that vacuuming becomes say, a once-a-month or less frequent operation. For some reason everyone who is used to working with PostgreSQL accepts the fact that you have to vacuum nightly - to outsiders it seems like a major flaw with the system. > There has been effort to speed up the vacuuming process, but this isn't the > cure. I believe the fault lies on the optimizer. Sure, the vacuum process speed is fine for small tables, but what about the big ones where the table gets locked for 5 minutes? What a joke! > Why save on micro-seconds to use sequential scan when the table is small and > later 'forgets' that the table is now big because you didn't vacuum analyze? > Why can't the optimizer just use indexes when they are there and not > 'optimize' for special cases when the table is small to save micro-seconds? Well its more than microseconds I presume, as opening indexes involves opening files, which takes milliseconds rather than microseconds. Andrew.
RE: [GENERAL] unique row identifier data type exhausted . . .
> It feels like there should be some *really* obvious answer to this > question, and I'll find myself whacking my forehead in self-abasement > and out of sheer relief to have found the answer to a problem that > should not have bothered me in the first place since the answer is too > self-evident . . . however, it is bothering me: what happens if the data > type that you've chosen to uniquely identify a row is exhausted? If, for > instance you use int4 and you've had your couple billion deletes and > inserts on the table and the next nextval('seq') . . . well, what > exactly happens and how do they do it? Admittedly, 10^9 is a big number > but it is far from out of the question that you'd reach it on a really > busy database (can't think of a real-world example but that ought to be > a moot point), not to mention oids since they are unique across an > entire database. I am curious to know how difficult it would be (if at all) to change the type that oid represents, to a 64 bit number. C'mon guys, this isn't the 90s any more! - Andrew
RE: [GENERAL] unique row identifier data type exhausted . . .
> When we are sure all platforms support 64-bit int's, we will move in > that direction. Sorry if this is a stupid question, but couldn't you fairly easily make it an option at compile time? To use either 32 or 64 bit OID's. (And, less importantly, for sequences) - Andrew
RE: [GENERAL] unique row identifier data type exhausted . . .
> Is this necessarily a good solution? If you use 64-bit OIDs, some joker > will just hook up a several-terra-byte disk array to his machine, try to > store the location of every molecule in the universe and break it. If you have to have OIDs at all, its a lot better than a 32 bit number. I think it would be easier to switch to 64 bit OIDs than ditch them completely. The "serial" type should definitely be 64 bit. To make matters worse I believe its really only a 31 bit number as the plus/minus symbol is discarded. But I think moving to 64 bit will take place soon enough, when it needs to, and it should shut everyone up. > "Christ died for sin, the righteous for the unrighteous, to bring > you to God." > - 1 Peter 3:18 Is it just me, or are there a lot of Christian PostgreSQL users? ;-) (I'm not knocking it, I'm one too..) - Andrew