Re: [GENERAL] Combine Date and Time Columns to Timestamp

2013-01-18 Thread Steven Schlansker
On Jan 18, 2013, at 4:26 PM, Rich Shepard wrote: > On Fri, 18 Jan 2013, Adrian Klaver wrote: > >> How are they stored, as date and time type, strings, other? > > Adrian, > > ISO date and time. > >> A sample of the data would help also. > > Example: 2012-10-29 | 10:19 | 2012-10-30 |

[GENERAL] Using partial index in combination with prepared statement parameters

2013-02-03 Thread Steven Schlansker
or otherwise). I ask largely because we use prepared statements everywhere and this makes it very hard to use partial indices, which would offer us significant performance gains. Does anyone know of any acceptable workaround? Is there continued interest in maybe improving the PostgreSQL behavior

Re: [GENERAL] best config

2013-02-06 Thread Steven Schlansker
On Feb 6, 2013, at 8:14 AM, Roberto Scattini wrote: > hi list, > > we have two new dell poweredge r720. based on recommendations from this list > we have configued the five disks in raid10 + 1 hot spare. You might mention a bit more about how your drives are configured. 5 drives in a RAID1+

Re: [GENERAL] best config

2013-02-06 Thread Steven Schlansker
On Feb 6, 2013, at 9:55 AM, Roberto Scattini wrote: > > hi steven, > > > we have two new dell poweredge r720. based on recommendations from this > > list we have configued the five disks in raid10 + 1 hot spare. > > You might mention a bit more about how your drives are configured. 5 drives

Re: [GENERAL] Testing Technique when using a DB

2013-03-12 Thread Steven Schlansker
On Mar 12, 2013, at 8:09 PM, Joe Van Dyk wrote: > On Mar 12, 2013, at 8:42 AM, Perry Smith wrote: > >> >> The other thought is perhaps there is a "snap shot" type concept. I don't >> see it in the list of SQL commands. A "snap shot" would do exactly what it >> sounds like. It would take

Re: [GENERAL] Queries seldomly take >4s while normally take <1ms?

2013-04-09 Thread Steven Schlansker
On Apr 9, 2013, at 11:25 AM, Scott Marlowe wrote: > One of the most common causes I've seen for this is linux's vm.*dirty* > settings to get in the way. Like so many linux kernel "optimizations" this > one looks good on paper but gives at best middling improvements with > occasional io storms

[GENERAL] Using an index for IS DISTINCT FROM queries

2013-04-22 Thread Steven Schlansker
Hi everyone, I have a large table (~150M rows) that keeps a version field. At any given time, it is expected that the vast majority of the rows are on the "current" version, but some may be missing. To figure out all the missing our outdated values, I run a query along the lines of SELECT id

Re: [GENERAL] pg_restore from split files

2013-04-22 Thread Steven Schlansker
On Apr 22, 2013, at 12:47 PM, akp geek wrote: > pg_dump dbname -n schemaname -t table_name -Fc | split -b 500m -t table.dump Since you split the files outside of the Postgres world, you have to combine them again. Roughly, cat table.dump.* > table.dump.combined pg_restore --usual-arguments t

Re: [GENERAL] Simple SQL INSERT to avoid duplication failed: why?

2013-04-30 Thread Steven Schlansker
On Apr 30, 2013, at 4:00 PM, "Carlo Stonebanks" wrote: > Hi Tom, > >>> There's nothing obviously wrong with that, which means the issue is in > something you didn't show us. Care to assemble a self-contained example? > << > > Unfortunately, it happens erratically and very, very rarely so I c

Re: [GENERAL] Simple SQL INSERT to avoid duplication failed: why?

2013-05-01 Thread Steven Schlansker
On May 1, 2013, at 9:36 AM, "Carlo Stonebanks" wrote: > I have to ask myself, is it more likely that I have discovered some PG > anomaly in 9.0 that no one has ever noticed, or that the client has > accidentally launched the process twice and doesn't know it? Given my (admittedly limited) exper

Re: [GENERAL] Deploying PostgreSQL on CentOS with SSD and Hardware RAID

2013-05-10 Thread Steven Schlansker
On May 10, 2013, at 7:14 AM, Matt Brock wrote: > Hello. > > We're intending to deploy PostgreSQL on Linux with SSD drives which would be > in a RAID 1 configuration with Hardware RAID. > > My first question is essentially: are there any issues we need to be aware of > when running PostgreSQL

Re: [GENERAL] Deploying PostgreSQL on CentOS with SSD and Hardware RAID

2013-05-10 Thread Steven Schlansker
On May 10, 2013, at 11:38 AM, Merlin Moncure wrote: >> >> PostgreSQL configuration changes: >> synchronous_commit = off >> > > that's good info, but it should be noted that synchronous_commit > trades a risk of some data loss (but not nearly as much risk as > volatile storage) for a big increa

Re: [GENERAL] Deploying PostgreSQL on CentOS with SSD and Hardware RAID

2013-05-10 Thread Steven Schlansker
On May 10, 2013, at 11:35 AM, Lonni J Friedman wrote: >> >> I am not sure that these numbers will end up being anywhere near what works >> for you, but these are my notes from tuning a 4xMLC SSD RAID-10. I haven't >> proven that this is optimal, but it was way better than the defaults. We >

Re: [GENERAL] upsert functionality

2013-05-16 Thread Steven Schlansker
On May 15, 2013, at 11:52 PM, Thomas Kellerer wrote: > Sajeev Mayandi, 16.05.2013 07:01: >> Hi, >> >> Our company is planning to move to postreSQL. We were initially using >> sybase where upsert functionality was available using "insert on >> existing update" clause. I know there multiple ways

[GENERAL] Index over only uncommon values in table

2013-06-18 Thread Steven Schlansker
Hi everyone, I assume this is not easy with standard PG but I wanted to double check. I have a column that has a very uneven distribution of values. ~95% of the values will be the same, with some long tail of another few dozens of values. I want to have an index over this value. Queries that

Re: [GENERAL] Index over only uncommon values in table

2013-06-18 Thread Steven Schlansker
On Jun 18, 2013, at 12:23 PM, John R Pierce wrote: > On 6/18/2013 12:17 PM, Steven Schlansker wrote: >> 1) The common value is not known at schema definition time, and may change >> (very slowly) over time. > > how could a value thats constant in 95% of the rows change, u

Re: [GENERAL] Index over only uncommon values in table

2013-06-18 Thread Steven Schlansker
On Jun 18, 2013, at 1:49 PM, David Johnston wrote: > Steven Schlansker-3 wrote >> At some point, the code changes, and CURRENT_VERSION gets incremented. >> Rows then slowly (over a period of days / weeks) get "upgraded" to the new >> current version, in batch

Re: [GENERAL] Index over only uncommon values in table

2013-06-18 Thread Steven Schlansker
On Jun 18, 2013, at 2:29 PM, Jeff Janes wrote: > On Tue, Jun 18, 2013 at 12:17 PM, Steven Schlansker > wrote: > Hi everyone, > > I assume this is not easy with standard PG but I wanted to double check. > > I have a column that has a very uneven distribution of values.

Re: [GENERAL] unique constraint violations

2013-06-26 Thread Steven Schlansker
On Jun 26, 2013, at 11:04 AM, pg noob wrote: > > Hi all, > > There are some places in our application where unique constraint violations > are difficult to avoid due to multithreading. > What we've done in most places to handle this is to retry in a loop. > > Generally it starts by checking

Re: [GENERAL] backend hangs at sendto() and can't be terminated

2013-07-11 Thread Steven Schlansker
On Jul 8, 2013, at 6:48 AM, Jov wrote: > netstat show nothing about the socket of the process,so I think the TCP > timeout took effect.so it is really wired. > > Jov > blog: http:amutu.com/blog > > > 2013/7/8 Tom Lane > Merlin Moncure writes: >> On Mon, Jul 8, 2013 at 4:56 AM, Jov wrote: >

Re: [GENERAL] backend hangs at sendto() and can't be terminated

2013-07-11 Thread Steven Schlansker
On Jul 8, 2013, at 6:48 AM, Jov wrote: > netstat show nothing about the socket of the process,so I think the TCP > timeout took effect.so it is really wired. > > Jov > blog: http:amutu.com/blog > > > 2013/7/8 Tom Lane > Merlin Moncure writes: > > On Mon, Jul 8, 2013 at 4:56 AM, Jov wrote:

Re: [GENERAL] Best Postgresql books

2013-07-18 Thread Steven Schlansker
PostgreSQL 9.0 High Performance: http://www.amazon.com/books/dp/184951030X On Jul 18, 2013, at 3:11 PM, Pedro Costa wrote: > Hi guys, > > Can anyone tell me the best books about postgresql? Specialy about tunning > and querys performances > Thanks > > > > Enviado do meu tablet Samsung

Re: [GENERAL] ZFS snapshots - to use pg_start_backup() and pg_stop_backup() or no?

2013-09-11 Thread Steven Schlansker
On Sep 11, 2013, at 4:29 PM, Gregory Haase wrote: > I was trying to figure out how to get the following syntax to work: > > echo "select pg_start_backup('zfs_snapshot'); \\! zfs snapshot > zroot/zpgsql@test; \\ select pg_stop_backup();" | psql postgres I do: psql -c "select pg_start_backup('

Re: [GENERAL] Deduplication and transaction isolation level

2013-09-25 Thread Steven Schlansker
On Sep 25, 2013, at 6:04 AM, Merlin Moncure wrote: > On Tue, Sep 24, 2013 at 10:19 PM, François Beausoleil > wrote: >> Hi all! >> >> I import many, many rows of data into a table, from three or more computers, >> 4 times per hour. I have a primary key, and the query I use to import the >> da

Re: [GENERAL] Blowfish Encrypted String

2013-09-26 Thread Steven Schlansker
On Sep 26, 2013, at 10:55 AM, Craig Boyd wrote: > Dmitriy, > > Thank you very much for the reply! > > Right...got the the data type...but how do I actually insert a binary string > into the column? What does the SQL look like? > For the moment assume I have the following bit of binary: 4PO

Re: [GENERAL] JDBC driver for Postgres 9.3

2013-09-26 Thread Steven Schlansker
On Sep 26, 2013, at 6:35 AM, "Kohler Manuel" wrote: > Hi, > we are developing a Java based software with Postgres as a DB. > Could someone tell me if there will be a JDBC driver for 9.3 out soon or > is it safe and recommended to use the latest JDBC driver available? > Currently we are using: >

Re: [GENERAL] Trying to create DB / user to import some data

2013-09-26 Thread Steven Schlansker
On Sep 26, 2013, at 10:28 PM, mdr wrote: > > create user import_dbms_user with password 'import_dbms'; > create database import_dbms_db; > grant all privileg > However when I try to run psql from the command line: > psql -h localhost -U import_dbms_user -WI enter password when prompted > Passwo

[GENERAL] Timing for release with fix for "Nov2013ReplicationIssue"

2013-12-03 Thread Steven Schlansker
Hi everyone, I’ve seen murmuring on the list regarding https://wiki.postgresql.org/wiki/Nov2013ReplicationIssue Is there an ETA on a release with the bug fix for this? I’m putting off building from source because I prefer to use the pgdg RPM packages, but if we don’t get a release soon it mig

Re: [GENERAL] Timing for release with fix for "Nov2013ReplicationIssue"

2013-12-03 Thread Steven Schlansker
On Dec 3, 2013, at 2:15 PM, Tom Lane wrote: > Steven Schlansker writes: >> I’ve seen murmuring on the list regarding >> https://wiki.postgresql.org/wiki/Nov2013ReplicationIssue > >> Is there an ETA on a release with the bug fix for this? I’m putting off >>

Re: [GENERAL] [JDBC] Cannot insert to 'path' field using EclipseLink

2014-03-12 Thread Steven Schlansker
> On Mar 12, 2014, at 10:12 AM, Daryl Foster wrote: > > java.lang.ClassCastException: org.postgresql.geometric.PGpath cannot be cast > to org.postgresql.geometric.PGpath > That's a sure sign of ClassLoader confusion. Make sure there is only one copy of the driver jar in your application or

[GENERAL] Trimming transaction logs after extended WAL archive failures

2014-03-25 Thread Steven Schlansker
Hi everyone, I have a Postgres 9.3.3 database machine. Due to some intelligent work on the part of someone who shall remain nameless, the WAL archive command included a ‘> /dev/null 2>&1’ which masked archive failures until the disk entirely filled with 400GB of pg_xlog entries. I have fixed

Re: [GENERAL] Trimming transaction logs after extended WAL archive failures

2014-03-25 Thread Steven Schlansker
On Mar 25, 2014, at 3:52 PM, Adrian Klaver wrote: > On 03/25/2014 01:56 PM, Steven Schlansker wrote: >> Hi everyone, >> >> I have a Postgres 9.3.3 database machine. Due to some intelligent work on >> the part of someone who shall remain nameless, the WAL arc

Re: [GENERAL] Trimming transaction logs after extended WAL archive failures

2014-03-25 Thread Steven Schlansker
On Mar 25, 2014, at 4:02 PM, Adrian Klaver wrote: > On 03/25/2014 03:54 PM, Steven Schlansker wrote: >> >> On Mar 25, 2014, at 3:52 PM, Adrian Klaver wrote: >> >>> On 03/25/2014 01:56 PM, Steven Schlansker wrote: >>>> Hi everyone, >>>>

Re: [GENERAL] Trimming transaction logs after extended WAL archive failures

2014-03-25 Thread Steven Schlansker
On Mar 25, 2014, at 4:45 PM, Adrian Klaver wrote: > On 03/25/2014 04:17 PM, Steven Schlansker wrote: >> >> On Mar 25, 2014, at 4:02 PM, Adrian Klaver wrote: >> >>> On 03/25/2014 03:54 PM, Steven Schlansker wrote: >>>> >>>> O

Re: [GENERAL] Trimming transaction logs after extended WAL archive failures

2014-03-26 Thread Steven Schlansker
On Mar 25, 2014, at 7:58 PM, Adrian Klaver wrote: > On 03/25/2014 04:52 PM, Steven Schlansker wrote: >> > >>> Some more questions, what happens when things begin to dawn on me:) >>> >>> You said the disk filled up entirely with log files yet currently t

Re: [GENERAL] Trimming transaction logs after extended WAL archive failures

2014-03-26 Thread Steven Schlansker
On Mar 26, 2014, at 9:04 AM, Jeff Janes wrote: > On Tue, Mar 25, 2014 at 6:33 PM, Jeff Janes wrote: > On Tuesday, March 25, 2014, Steven Schlansker wrote: > Hi everyone, > > I have a Postgres 9.3.3 database machine. Due to some intelligent work on > the part of someon

Re: [GENERAL] Trimming transaction logs after extended WAL archive failures

2014-03-27 Thread Steven Schlansker
On Mar 27, 2014, at 5:29 AM, Michael Paquier wrote: > On Thu, Mar 27, 2014 at 1:42 AM, Steven Schlansker > wrote: >> >> On Mar 25, 2014, at 7:58 PM, Adrian Klaver wrote: >>>> >>> Yea, vacuum just marks space as available for reuse it does not actua

Re: [GENERAL] Is it safe to stop postgres in between pg_start_backup and pg_stop_backup?

2014-04-03 Thread Steven Schlansker
On Apr 2, 2014, at 3:08 PM, Jacob Scott wrote: > Hi, Hello there ;) > > > Does upgrading a a disk being used by postgres (9.1, on Ubuntu) with the > following process sound safe? > • pg_start_backup > • Take a filesystem snapshot (of a volume containing postgres data but > not

Re: [GENERAL] any way for a transaction to "see" inserts done earlier in the transaction?

2014-04-16 Thread Steven Schlansker
On Apr 16, 2014, at 4:27 PM, Susan Cassidy wrote: > Is there any way to let a transaction "see" the inserts that were done > earlier in the transaction? I want to insert a row, then later use it within > the same transaction. > > If not, I will have to commit after each insert, and I don't

Re: [GENERAL] any way for a transaction to "see" inserts done earlier in the transaction?

2014-04-16 Thread Steven Schlansker
>> On Wed, Apr 16, 2014 at 4:31 PM, Tom Lane wrote: >> Susan Cassidy writes: >> > Is there any way to let a transaction "see" the inserts that were done >> > earlier in the transaction? >> >> It works that way automatically, as long as you're talking about separate >> statements within one tra

[GENERAL] Suboptimal query plan fixed by replacing OR with UNION

2012-07-05 Thread Steven Schlansker
27;{-02f6-379d-c000-00026810,-0320-b467-c000-00026810,-000d-cefb-c000-00026810}'::uuid[])) (16 rows) Wow! Changing the query from using an OR clause to a UNION DISTINCT with two SELECTs reduced the cost from 1379485.60 to 3366.35! And the gains are rea

Re: [GENERAL] Suboptimal query plan fixed by replacing OR with UNION

2012-07-05 Thread Steven Schlansker
On Jul 5, 2012, at 3:51 PM, Tom Lane wrote: > Steven Schlansker writes: >> Why is using an OR so awful here? > > Because the OR stops it from being a join (it possibly needs to return > some rows that are not in the semijoin of the two tables). > >> Why does it

Re: [GENERAL] Suboptimal query plan fixed by replacing OR with UNION

2012-07-06 Thread Steven Schlansker
On Jul 5, 2012, at 6:35 PM, Jasen Betts wrote: > I note you've decided to rewrite this query as a union > >> SELECT * FROM account >> WHERE user_id in >>(SELECT user_id FROM account >> WHERE id = >> ANY('{-02f6-379d-c000-00026810,-0320-b467-c000-00026810,000

Re: [GENERAL] Suboptimal query plan fixed by replacing OR with UNION

2012-07-09 Thread Steven Schlansker
On Jul 6, 2012, at 9:24 PM, Gurjeet Singh wrote: > On Thu, Jul 5, 2012 at 7:16 PM, Steven Schlansker wrote: > > On Jul 5, 2012, at 3:51 PM, Tom Lane wrote: > > > Steven Schlansker writes: > >> Why is using an OR so awful here? > > > > Because the OR

[GENERAL] Ignore hash indices on replicas

2012-07-10 Thread Steven Schlansker
I'm using Postgres hash indices on a streaming replica master. As is documented, hash indices are not logged, so the replica does not have access to them. I understand that the current wisdom is "don't use hash indices", but (unfortunately?) I have benchmarks that show that our particular applic

Re: [GENERAL] Replication/cloning: rsync vs modification dates?

2012-07-16 Thread Steven Schlansker
I think it's pretty easy to show that timestamp+size isn't good enough to do this 100% reliably. Imagine that your timestamps have a millisecond resolution. I assume this will vary based on OS / filesystem, but the point remains the same no matter what size it is. You can have multiple writes

[GENERAL] Postgres will not compile on Mac 10.8 with contrib/uuid-ossp

2012-07-30 Thread Steven Schlansker
It is not possible to compile Postgres contrib/uuid-ossp on the newest release of Mac OS X, 10.8 The specific compile error: make -C uuid-ossp install /bin/sh ../../config/install-sh -c -d '/usr/local/Cellar/postgresql/9.1.3/lib' /usr/bin/clang -Os -w -pipe -march=native -Qunused-arguments -I

Re: [GENERAL] Postgres will not compile on Mac 10.8 with contrib/uuid-ossp

2012-07-30 Thread Steven Schlansker
On Jul 30, 2012, at 7:35 PM, Tom Lane wrote: > Steven Schlansker writes: >> It is not possible to compile Postgres contrib/uuid-ossp on the newest >> release of Mac OS X, 10.8 > > This looks like some variant of the same issue that OSSP's uuid > package has had

Re: [GENERAL] Ignore hash indices on replicas

2012-08-20 Thread Steven Schlansker
On Aug 19, 2012, at 8:01 PM, Scott Marlowe wrote: > On Tue, Jul 10, 2012 at 1:09 AM, Steven Schlansker > wrote: >> I'm using Postgres hash indices on a streaming replica master. >> As is documented, hash indices are not logged, so the replica does not have >

Re: [GENERAL] Ignore hash indices on replicas

2012-08-20 Thread Steven Schlansker
On Aug 19, 2012, at 2:37 PM, Jeff Davis wrote: > On Tue, 2012-07-10 at 00:09 -0700, Steven Schlansker wrote: >> I understand that the current wisdom is "don't use hash indices", but >> (unfortunately?) I have benchmarks that >> show that our particular appli

Re: [GENERAL] Confirming \timing output

2012-08-23 Thread Steven Schlansker
On Aug 23, 2012, at 11:13 AM, "Gauthier, Dave" wrote: > With \timing set on, I run an update statement and it reports > > Time: 0.524 ms > > Is that really 0.524 ms? As in 524 nanoseconds? 0.524ms = 524000ns Perhaps you meant microseconds? 0.524ms = 524us If all your data happens to

Re: [GENERAL] Noticed something odd with pgbench

2012-11-16 Thread Steven Schlansker
On Nov 16, 2012, at 11:59 AM, Richard Huxton wrote: > On 16/11/12 19:35, Shaun Thomas wrote: >> Hey guys, >> >> So, we have a pretty beefy system that runs dual X5675's with 72GB of RAM. >> After our recent upgrade to 9.1, things have been... odd. I managed to track >> it down to one setting: