> -Original Message-
> From: Greg Smith [mailto:g...@2ndquadrant.com]
> Sent: Monday, August 08, 2011 9:42 PM
> To: mark
> Cc: pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] benchmark woes and XFS options
>
> I think your notion that you have an HP CC
I think your notion that you have an HP CCISS driver in this older
kernel that just doesn't drive your card very fast is worth exploring.
What I sometimes do in the situation you're in is boot a Linux
distribution that comes with a decent live CD, such as Debian or
Ubuntu. Just mount the susp
Hello PG perf junkies,
Sorry this may get a little long winded. Apologies if the formatting gets
trashed. Also apologies if this double posts. (I originally set it yesterday
with the wrong account and the message is stalled - so my bad there) if
someone is a mod and it's still in the wait queue
Hi.
Not strictly connected to your tests, but:
As of ZFS, we've had experience that it degrades over time after random
updates because of files becoming non-linear and sequential reads becomes
random.
Also there are Q about ZFS block size - setting it to 8K makes first problem
worse, setting it to
Hi,
The tests were made without the -s parameter, (so 1 is assumed). I'm running
the numbers
again on CentOS, with the optimized config and I'll test also different
scale values. I also will be able
to repeat the test again in FreeBSD with ZFS with the new options and
different scale,
but probably
Joao,
Wow, thanks for doing this!
In general, your tests seem to show that there isn't a substantial
penalty for using ZFS as of version 8.0.
If you have time for more tests, I'd like to ask you for a few more tweaks:
(1) change the following settings according to conventional wisdom:
This is not a rigorous test and should not be used as a direct comparison
between
operating systems. My objective was to estimate the ZFS toll in Postgres,
and
what to expect in performance, comparing to the old server this machine will
be replacing. Not all possible configurations were tested, an
James Mansion wrote:
> Ivan Voras wrote:
>> I wish that, when people got the idea to run a simplistic benchmark
>> like this, they would at least have the common sense to put the
>> database on a RAM drive to avoid problems with different cylinder
>> speeds of rotational media and fragmentation fro
Kevin Grittner wrote:
It is if you don't have an index on the table which has a foreign
key defined which references the table in which you're doing
deletes. The author of the benchmark apparently didn't realize that
MySQL automatically adds such an index to the dependent table, while
PostgreSQL
Ivan Voras wrote:
I wish that, when people got the idea to run a simplistic benchmark
like this, they would at least have the common sense to put the
database on a RAM drive to avoid problems with different cylinder
speeds of rotational media and fragmentation from multiple runs.
Huh?
It's to
On Wednesday 27 January 2010 15:49:06 Matthew Wakeling wrote:
> On Wed, 27 Jan 2010, Thom Brown wrote:
> > Had a quick look at a benchmark someone put together of MySQL vs
> > PostgreSQL, and while PostgreSQL is generally faster, I noticed the bulk
> > delete was very slow:
> > http://www.randombug
On Wed, Jan 27, 2010 at 9:54 AM, Kevin Grittner wrote:
> It is if you don't have an index on the table which has a foreign
> key defined which references the table in which you're doing
> deletes. The author of the benchmark apparently didn't realize that
> MySQL automatically adds such an index
Thom Brown wrote:
> Had a quick look at a benchmark someone put together of MySQL vs
> PostgreSQL, and while PostgreSQL is generally faster, I noticed
> the bulk delete was very slow:
> http://www.randombugs.com/linux/mysql-postgresql-benchmarks.html
>
> Is this normal?
It is if you don't hav
On Wed, 27 Jan 2010, Thom Brown wrote:
Had a quick look at a benchmark someone put together of MySQL vs PostgreSQL,
and while PostgreSQL is generally faster, I noticed the bulk delete was very
slow: http://www.randombugs.com/linux/mysql-postgresql-benchmarks.html
Is this normal?
On the contrar
On 01/27/10 14:28, Thom Brown wrote:
Had a quick look at a benchmark someone put together of MySQL vs
PostgreSQL, and while PostgreSQL is generally faster, I noticed the bulk
delete was very slow:
http://www.randombugs.com/linux/mysql-postgresql-benchmarks.html
I wish that, when people got the
Had a quick look at a benchmark someone put together of MySQL vs PostgreSQL,
and while PostgreSQL is generally faster, I noticed the bulk delete was very
slow: http://www.randombugs.com/linux/mysql-postgresql-benchmarks.html
Is this normal?
Thom
On Mon, Feb 23, 2009 at 1:29 PM, Sergio Lopez wrote:
> El Sat, 21 Feb 2009 21:04:49 -0500
> I've taken down the article and I'll bring up it again when I've
> collected new numbers.
Please do, this subject is very interesting.
Regards.
--
Sent via pgsql-performance mailing list (pgsql-performa
El Sat, 21 Feb 2009 21:04:49 -0500
"Jonah H. Harris" escribió:
> On Fri, Feb 20, 2009 at 8:40 PM, Denis Lussier <
> denis.luss...@enterprisedb.com> wrote:
>
> > Hi all,
> >
> > As the author of BenchmarkSQL and the founder of EnterpriseDB I
> > can assure you that BenchmarkSQL was NOT writt
On Fri, Feb 20, 2009 at 8:40 PM, Denis Lussier <
denis.luss...@enterprisedb.com> wrote:
> Hi all,
>
> As the author of BenchmarkSQL and the founder of EnterpriseDB I
> can assure you that BenchmarkSQL was NOT written specifically for
> PostgreSQL.It is intended to be a completely database
Hi all,
As the author of BenchmarkSQL and the founder of EnterpriseDB I
can assure you that BenchmarkSQL was NOT written specifically for
PostgreSQL.It is intended to be a completely database agnostic
tpc-c like java based benchmark.
However; as Jonah correctly points out in painstaking
El Fri, 20 Feb 2009 16:54:58 -0500
Robert Haas escribió:
> On Fri, Feb 20, 2009 at 4:34 PM, Jonah H. Harris
> wrote:
> > On Fri, Feb 20, 2009 at 3:40 PM, Merlin Moncure
> > wrote:
> >>
> >> ISTM you are the one throwing out unsubstantiated assertions
> >> without data to back it up. OP ran ben
El Fri, 20 Feb 2009 14:48:06 -0500
"Jonah H. Harris" escribió:
> On Fri, Feb 20, 2009 at 1:15 PM, Sergio Lopez
> wrote:
>
> Having this said, the benchmark is not as unfair as you thought. I've
> > taken care to prepare all databases to meet similar values for their
> > cache, buffers and I/O co
Robert Haas wrote:
The biggest flaw in the benchmark by far has got to be that it was
done with a ramdisk, so it's really only measuring CPU consumption.
Measuring CPU consumption is interesting, but it doesn't have a lot to
do with throughput in real-life situations.
... and memory access
On Fri, Feb 20, 2009 at 2:54 PM, Robert Haas wrote:
> On Fri, Feb 20, 2009 at 4:34 PM, Jonah H. Harris
> wrote:
>> On Fri, Feb 20, 2009 at 3:40 PM, Merlin Moncure wrote:
>>>
>>> ISTM you are the one throwing out unsubstantiated assertions without
>>> data to back it up. OP ran benchmark. showe
On Fri, Feb 20, 2009 at 4:34 PM, Jonah H. Harris wrote:
> On Fri, Feb 20, 2009 at 3:40 PM, Merlin Moncure wrote:
>>
>> ISTM you are the one throwing out unsubstantiated assertions without
>> data to back it up. OP ran benchmark. showed hardware/configs, and
>> demonstrated result. He was carefu
On Fri, Feb 20, 2009 at 3:40 PM, Merlin Moncure wrote:
> ISTM you are the one throwing out unsubstantiated assertions without
> data to back it up. OP ran benchmark. showed hardware/configs, and
> demonstrated result. He was careful to hedge expectations and gave
> rationale for his analysis me
On Fri, Feb 20, 2009 at 2:48 PM, Jonah H. Harris wrote:
> On Fri, Feb 20, 2009 at 1:15 PM, Sergio Lopez
> wrote:
>>
>> On the other hand, I've neved said that what I've done is the
>> Perfect-Marvelous-Definitive Benchmark, it's just a personal project,
>> and I don't have an infinite amount of t
On Fri, Feb 20, 2009 at 2:48 PM, Jonah H. Harris wrote:
> Having this said, the benchmark is not as unfair as you thought. I've
>> taken care to prepare all databases to meet similar values for their
>> cache, buffers and I/O configuration (to what's possible given their
>> differences), and the I
On Fri, Feb 20, 2009 at 2:35 PM, Robert Haas wrote:
> > First of all, you need to do some research on the benchmark kit itself,
> > rather than blindly downloading and using one. BenchmarkSQL has
> significant
> > bugs in it which affect the result. I can say that authoritatively as I
> > worke
On Fri, Feb 20, 2009 at 1:15 PM, Sergio Lopez wrote:
> On the other hand, I've neved said that what I've done is the
> Perfect-Marvelous-Definitive Benchmark, it's just a personal project,
> and I don't have an infinite amount of time to invest on it.
When you make comments such as "As for datab
> First of all, you need to do some research on the benchmark kit itself,
> rather than blindly downloading and using one. BenchmarkSQL has significant
> bugs in it which affect the result. I can say that authoritatively as I
> worked on/with it for quite awhile. Don't trust any result that come
El Fri, 20 Feb 2009 12:39:41 -0500
"Jonah H. Harris" escribió:
> On Fri, Feb 20, 2009 at 6:28 AM, Sergio Lopez
> wrote:
>
> > Hi,
> >
> > I've made a benchmark comparing PostgreSQL, MySQL and Oracle under
> > three environments: GNU/Linux-x86, Solaris-x86 (same machine as
> > GNU/Linux) and Sola
On Fri, Feb 20, 2009 at 6:28 AM, Sergio Lopez wrote:
> Hi,
>
> I've made a benchmark comparing PostgreSQL, MySQL and Oracle under three
> environments: GNU/Linux-x86, Solaris-x86 (same machine as GNU/Linux) and
> Solaris-SPARC. I think you might find it interesting:
>
>
> http://blogs.nologin.es/s
El Fri, 20 Feb 2009 08:36:44 -0800
Alan Hodgson escribió:
> On Friday 20 February 2009, Sergio Lopez
> wrote:
> > Hi,
> >
> > I've made a benchmark comparing PostgreSQL, MySQL and Oracle under
> > three environments: GNU/Linux-x86, Solaris-x86 (same machine as
> > GNU/Linux) and Solaris-SPARC. I
On Friday 20 February 2009, Sergio Lopez wrote:
> Hi,
>
> I've made a benchmark comparing PostgreSQL, MySQL and Oracle under three
> environments: GNU/Linux-x86, Solaris-x86 (same machine as GNU/Linux) and
> Solaris-SPARC. I think you might find it interesting:
>
> http://blogs.nologin.es/slopez/a
Hi,
I've made a benchmark comparing PostgreSQL, MySQL and Oracle under three
environments: GNU/Linux-x86, Solaris-x86 (same machine as GNU/Linux) and
Solaris-SPARC. I think you might find it interesting:
http://blogs.nologin.es/slopez/archives/17-Benchmarking-Databases-I.-Volatile-Storage..html
On Sun, Mar 16, 2008 at 12:04:44PM -0700, Craig James wrote:
Just out of curiosity: Last time I did research, the word seemed to be that
xfs was better than ext2 or ext3. Is that not true? Why use ext2/3 at all
if xfs is faster for Postgres?
For the WAL, the filesystem is largely irrelevant.
[EMAIL PROTECTED] wrote:
WAL is on a RAID 0 drive along with the OS
Isn't that just as unsafe as having the whole lot on RAID0?
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-perform
On Mon, Mar 17, 2008 at 2:58 PM, Enrico Sirola <[EMAIL PROTECTED]> wrote:
> Hi Justin,
>
> Il giorno 17/mar/08, alle ore 20:38, Justin ha scritto:
>
> > it is a RAID 10 controller with 6 SAS 10K 73 gig drives.The
> > server is 3 weeks old now.
> >
> > it has 16 gigs of RAM
> > 2 quad core
Hi Justin,
Il giorno 17/mar/08, alle ore 20:38, Justin ha scritto:
it is a RAID 10 controller with 6 SAS 10K 73 gig drives.The
server is 3 weeks old now.
it has 16 gigs of RAM
2 quad core Xenon 1.88 Ghz processors
2 gig Ethernet cards. RAID controller perc 6/i with battery backup
512meg
Craig James wrote:
Justin wrote:
2000 tps ??? do you have fsync turned off ?
Dave
No its turned on.
Unless I'm seriously confused, something is wrong with these numbers.
That's the sort of performance you expect from a good-sized RAID 10
six-disk array. With a single 7200 rpm SATA di
Justin wrote:
2000 tps ??? do you have fsync turned off ?
Dave
No its turned on.
Unless I'm seriously confused, something is wrong with these numbers. That's
the sort of performance you expect from a good-sized RAID 10 six-disk array.
With a single 7200 rpm SATA disk and XFS, I get 640
2000 tps ??? do you have fsync turned off ?
Dave
No its turned on.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
On 17-Mar-08, at 2:50 PM, Justin wrote:
Just out of curiosity: Last time I did research, the word seemed to
be that xfs was better than ext2 or ext3. Is that not true? Why
use ext2/3 at all if xfs is faster for Postgres?
Criag
Ext2 vs XFS on my setup there is difference in the perf
Just out of curiosity: Last time I did research, the word seemed to be
that xfs was better than ext2 or ext3. Is that not true? Why use
ext2/3 at all if xfs is faster for Postgres?
Criag
Ext2 vs XFS on my setup there is difference in the performance between
the two file systems but its
Well every thing worked right up to the point where i tried to mount the
file system
Warning: xfs_db: /dev/sdb1 contains a mounted file system
fatal error -- couldn't initialize XFS library.
think i'm missing something???
Craig Ringer wrote:
Justin wrote:
OK i'm showing my ignorance of
Justin wrote:
> OK i'm showing my ignorance of linux. On Ubuntu i can't seem to figure
> out if XFS file system is installed, if not installed getting it
> installed.
There are two parts to the file system, really. One is the kernel driver
for the file system. This is almost certainly available
On 17/03/2008, Justin <[EMAIL PROTECTED]> wrote:
> OK i'm showing my ignorance of linux. On Ubuntu i can't seem to figure
> out if XFS file system is installed, if not installed getting it
> installed.
...
> any pointers would be nice. I 'm not going to reinstall the OS. Nor do
> i want to
Justin wrote:
OK i'm showing my ignorance of linux. On Ubuntu i can't seem to figure
out if XFS file system is installed, if not installed getting it
installed.
Hm? Installed/not installed? You can select that when you are preparing
your partitions. If you run the automated partitioner ther
OK i'm showing my ignorance of linux. On Ubuntu i can't seem to figure
out if XFS file system is installed, if not installed getting it
installed.
I would like to see the difference between XFS and ext2 performance
numbers.
any pointers would be nice. I 'm not going to reinstall the OS.
On Sun, Mar 16, 2008 at 1:36 PM, Dave Cramer <[EMAIL PROTECTED]> wrote:
>
> On 16-Mar-08, at 3:04 PM, Craig James wrote:
> > Just out of curiosity: Last time I did research, the word seemed to
> > be that xfs was better than ext2 or ext3. Is that not true? Why
> > use ext2/3 at all if xfs is
On 16-Mar-08, at 3:04 PM, Craig James wrote:
Dave Cramer wrote:
On 16-Mar-08, at 2:19 AM, Justin wrote:
I decided to reformat the raid 10 into ext2 to see if there was
any real big difference in performance as some people have noted
here is the test results
please note the WAL files
Craig James wrote:
Dave Cramer wrote:
On 16-Mar-08, at 2:19 AM, Justin wrote:
I decided to reformat the raid 10 into ext2 to see if there was any
real big difference in performance as some people have noted here
is the test results
please note the WAL files are still on the raid 0 set
Dave Cramer wrote:
On 16-Mar-08, at 2:19 AM, Justin wrote:
I decided to reformat the raid 10 into ext2 to see if there was any
real big difference in performance as some people have noted here is
the test results
please note the WAL files are still on the raid 0 set which is still
in e
On 16-Mar-08, at 2:19 AM, Justin wrote:
I decided to reformat the raid 10 into ext2 to see if there was any
real big difference in performance as some people have noted here
is the test results
please note the WAL files are still on the raid 0 set which is still
in ext3 file system f
I decided to reformat the raid 10 into ext2 to see if there was any real
big difference in performance as some people have noted here is the
test results
please note the WAL files are still on the raid 0 set which is still in
ext3 file system format. these test where run with the fsync as
On Thu, Mar 13, 2008 at 05:27:09PM -0400, Greg Smith wrote:
I haven't found fdatasync to be significantly better in my tests on Linux
but I never went out of my way to try and quantify it. My understanding
is that some of the write barrier implementation details on ext3
filesystems make any sy
Greg Smith wrote:
On Fri, 14 Mar 2008, Justin wrote:
I played with shared_buffer and never saw much of an improvement from
100 all the way up to 800 megs moved the checkpoints from 3 to 30 and
still never saw no movement in the numbers.
Increasing shared_buffers normally improves performan
On Fri, 14 Mar 2008, Justin wrote:
I played with shared_buffer and never saw much of an improvement from
100 all the way up to 800 megs moved the checkpoints from 3 to 30 and
still never saw no movement in the numbers.
Increasing shared_buffers normally improves performance as the size of the
Is this on a 64 bit or 32 bit machine? We had the problem with a 32
bit linux box (not sure what flavor) just a few months ago. I would
not create a filesystem on a partition of 2+TB
Yes this machine is 64bit
You do know that effective_cache_size is the size of the OS level
cache. i.e. i
Scott Marlowe wrote:
On Thu, Mar 13, 2008 at 3:09 PM, justin <[EMAIL PROTECTED]> wrote:
I chose to use ext3 on these partition
You should really consider another file system. ext3 has two flaws
that mean I can't really use it properly. A 2TB file system size
limit (at least on the servers
Scott Marlowe wrote:
On Fri, Mar 14, 2008 at 12:17 AM, Jesper Krogh <[EMAIL PROTECTED]> wrote:
Scott Marlowe wrote:
> On Thu, Mar 13, 2008 at 3:09 PM, justin <[EMAIL PROTECTED]> wrote:
>
>> I chose to use ext3 on these partition
>
> You should really consider another file system. ext3 has
On Fri, Mar 14, 2008 at 12:19 AM, Scott Marlowe <[EMAIL PROTECTED]> wrote:
>
> On Fri, Mar 14, 2008 at 12:17 AM, Jesper Krogh <[EMAIL PROTECTED]> wrote:
> >
> > Scott Marlowe wrote:
> > > On Thu, Mar 13, 2008 at 3:09 PM, justin <[EMAIL PROTECTED]> wrote:
> > >
> > >> I chose to use ext3 on
On Fri, Mar 14, 2008 at 12:17 AM, Jesper Krogh <[EMAIL PROTECTED]> wrote:
>
> Scott Marlowe wrote:
> > On Thu, Mar 13, 2008 at 3:09 PM, justin <[EMAIL PROTECTED]> wrote:
> >
> >> I chose to use ext3 on these partition
> >
> > You should really consider another file system. ext3 has two flaws
On Thu, Mar 13, 2008 at 3:09 PM, justin <[EMAIL PROTECTED]> wrote:
> I chose to use ext3 on these partition
You should really consider another file system. ext3 has two flaws
that mean I can't really use it properly. A 2TB file system size
limit (at least on the servers I've tested) and it loc
On Thu, Mar 13, 2008 at 4:53 PM, justin <[EMAIL PROTECTED]> wrote:
>
> I'm ran pgbench from my laptop to the new server
>
> My laptop is dual core with 2 gigs of ram and 1 gig enthernet connection to
> server. so i don't think the network is going to be a problem in the test.
>
> When i look
- Original Message -
From: "Greg Smith" <[EMAIL PROTECTED]>
To:
Sent: Thursday, March 13, 2008 4:27 PM
Subject: Re: [PERFORM] Benchmark: Dell/Perc 6, 8 disk RAID 10
On Thu, 13 Mar 2008, Joshua D. Drake wrote:
Greg Smith <[EMAIL PROTECTED]> wrote:
wal_s
On Thu, 13 Mar 2008, Joshua D. Drake wrote:
Greg Smith <[EMAIL PROTECTED]> wrote:
wal_sync_method = open_sync
There was a bug report I haven't had a chance to investigate yet that
suggested some recent Linux versions have issues when using
open_sync. I'd suggest popping that back to the defau
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Thu, 13 Mar 2008 12:01:50 -0400 (EDT)
Greg Smith <[EMAIL PROTECTED]> wrote:
> On Thu, 13 Mar 2008, Craig James wrote:
>
> >> wal_sync_method = open_sync
>
> There was a bug report I haven't had a chance to investigate yet that
> suggested some r
Absolutely on the battery backup.
I did not load the linux drivers from dell, it works so i figured i was not
going to worry about it. This server is so oversized for its load its
unreal. I have always gone way overboard on server specs and making sure
its redundant.
The difference in our
On Thu, 13 Mar 2008, Craig James wrote:
wal_sync_method = open_sync
There was a bug report I haven't had a chance to investigate yet that
suggested some recent Linux versions have issues when using open_sync.
I'd suggest popping that back to the default for now unless you have time
to reall
Joshua D. Drake wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Wed, 12 Mar 2008 21:55:18 -0700
Craig James <[EMAIL PROTECTED]> wrote:
Diffs from original configuration:
max_connections = 1000
shared_buffers = 400MB
work_mem = 256MB
max_fsm_pages = 100
max_fsm_relations = 5000
wa
Doug Knight wrote:
All,
I am in the process of specing out a purchase for our production
systems, and am looking at the Dell 2950s as well. I am very interested
to see where this thread goes, and what combinations work with different
application loading types. Our systems will have one pair of
Justin Graf wrote:
I recent just got a new server also from dell 2 weeks ago
went with more memory slower CPU, and smaller harddrives
have not run pgbench
Dell PE 2950 III
2 Quad Core 1.866 Ghz
16 gigs of ram.
8 hard drives 73Gig 10k RPM SAS
2 drives in Mirrored for OS, Binaries, and WAL
6 in a
I did not run into one install problem, I read a thread about people having
problems but the thread is over a year old now.
I used the 7.1 gutsy amd64 server version
I then installed gnome desktop because its not installed by default. "i'm a
windows admin i have to have my gui"
then instal
Justin,
This may be a bit out of context, but did you run into any troubles
getting your Perc6i RAID controller to work under Ubuntu 7.1? I've
heard there were issues with that.
Thanks,
Will
On Mar 13, 2008, at 3:11 AM, Justin Graf wrote:
I recent just got a new server also from dell 2
All,
I am in the process of specing out a purchase for our production
systems, and am looking at the Dell 2950s as well. I am very interested
to see where this thread goes, and what combinations work with different
application loading types. Our systems will have one pair of
heartbeat-controlled, d
I recent just got a new server also from dell 2 weeks ago
went with more memory slower CPU, and smaller harddrives
have not run pgbench
Dell PE 2950 III
2 Quad Core 1.866 Ghz
16 gigs of ram.
8 hard drives 73Gig 10k RPM SAS
2 drives in Mirrored for OS, Binaries, and WAL
6 in a
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Wed, 12 Mar 2008 21:55:18 -0700
Craig James <[EMAIL PROTECTED]> wrote:
> Diffs from original configuration:
>
> max_connections = 1000
> shared_buffers = 400MB
> work_mem = 256MB
> max_fsm_pages = 100
> max_fsm_relations = 5000
> wal_buffers
On Wed, Mar 12, 2008 at 9:55 PM, Craig James <[EMAIL PROTECTED]> wrote:
> I just received a new server and thought benchmarks would be interesting. I
> think this looks pretty good, but maybe there are some suggestions about the
> configuration file. This is a web app, a mix of read/write, wher
I just received a new server and thought benchmarks would be interesting. I think this
looks pretty good, but maybe there are some suggestions about the configuration file.
This is a web app, a mix of read/write, where writes tend to be "insert into ...
(select ...)" where the resulting inser
On Mon, 4 Feb 2008 15:09:58 -0500 (EST)
Greg Smith <[EMAIL PROTECTED]> wrote:
> On Mon, 4 Feb 2008, Simon Riggs wrote:
>
> > Would anybody like to repeat these tests with the latest production
> > versions of these databases (i.e. with PGSQL 8.3)
>
> Do you have any suggestions on how people sho
On Mon, 04 Feb 2008 17:33:34 -0500
"Jignesh K. Shah" <[EMAIL PROTECTED]> wrote:
> Hi Simon,
>
> I have some insight into TPC-H on how it works.
>
> First of all I think it is a violation of TPC rules to publish numbers
> without auditing them first. So even if I do the test to show the
> bett
> > I was thinking of not even reading the file content from the controller
> > thread, just decide splitting points in bytes (0..ST_SIZE/4 -
> > ST_SIZE/4+1..2*ST_SIZE/4 etc) and let the reading thread fine-tune by
> > beginning to process input after having read first newline, etc.
>
> The probl
On Thu, Feb 07, 2008 at 12:06:42PM -0500, Greg Smith wrote:
> On Thu, 7 Feb 2008, Dimitri Fontaine wrote:
>
>> I was thinking of not even reading the file content from the controller
>> thread, just decide splitting points in bytes (0..ST_SIZE/4 -
>> ST_SIZE/4+1..2*ST_SIZE/4 etc) and let the readin
On Thu, 7 Feb 2008, Greg Smith wrote:
The problem I was pointing out is that if chunk#2 moved foward a few bytes
before it started reading in search of a newline, how will chunk#1 know that
it's supposed to read up to that further point? You have to stop #1 from
reading further when it catches
On Thu, 7 Feb 2008, Dimitri Fontaine wrote:
I was thinking of not even reading the file content from the controller
thread, just decide splitting points in bytes (0..ST_SIZE/4 -
ST_SIZE/4+1..2*ST_SIZE/4 etc) and let the reading thread fine-tune by
beginning to process input after having read fir
Le jeudi 07 février 2008, Greg Smith a écrit :
>Le mercredi 06 février 2008, Dimitri Fontaine a écrit :
>> In other cases, a logical line is a physical line, so we start after first
>> newline met from given lseek start position, and continue reading after the
>> last lseek position until a newline
On Wed, 6 Feb 2008, Dimitri Fontaine wrote:
In fact, the -F option works by having pgloader read the given number of lines
but skip processing them, which is not at all what Greg is talking about here
I think.
Yeah, that's not useful.
Greg, what would you think of a pgloader which will separ
Le Wednesday 06 February 2008 18:49:56 Luke Lonergan, vous avez écrit :
> Improvements are welcome, but to compete in the industry, loading will need
> to speed up by a factor of 100.
Oh, I meant to compete with internal COPY command instead of \copy one, not
with the competition. AIUI competing
Le Wednesday 06 February 2008 18:37:41 Dimitri Fontaine, vous avez écrit :
> Le mercredi 06 février 2008, Greg Smith a écrit :
> > If I'm loading a TB file, odds are good I can split that into 4 or more
> > vertical pieces (say rows 1-25%, 25-50%, 50-75%, 75-100%), start 4
> > loaders at once, and
[mailto:[EMAIL PROTECTED]
Sent: Wednesday, February 06, 2008 12:41 PM Eastern Standard Time
To: pgsql-performance@postgresql.org
Cc: Greg Smith
Subject:Re: [PERFORM] Benchmark Data requested --- pgloader CE design
ideas
Le mercredi 06 février 2008, Greg Smith a écrit :
> If
Le mercredi 06 février 2008, Greg Smith a écrit :
> If I'm loading a TB file, odds are good I can split that into 4 or more
> vertical pieces (say rows 1-25%, 25-50%, 50-75%, 75-100%), start 4 loaders
> at once, and get way more than 1 disk worth of throughput reading.
pgloader already supports st
Le mercredi 06 février 2008, Greg Smith a écrit :
> COPY. If you're loading a TB, if you're smart it's going onto the server
> itself if it all possible and loading directly from there. Would probably
> get a closer comparision against psql \copy, but recognize you're always
> going to be compare
Greg Smith wrote:
On Wed, 6 Feb 2008, Simon Riggs wrote:
For me, it would be good to see a --parallel=n parameter that would
allow pg_loader to distribute rows in "round-robin" manner to "n"
different concurrent COPY statements. i.e. a non-routing version.
Let me expand on this. In many of
Hi Greg,
On 2/6/08 7:56 AM, "Greg Smith" <[EMAIL PROTECTED]> wrote:
> If I'm loading a TB file, odds are good I can split that into 4 or more
> vertical pieces (say rows 1-25%, 25-50%, 50-75%, 75-100%), start 4 loaders
> at once, and get way more than 1 disk worth of throughput reading. You
> ha
On Wed, 6 Feb 2008, Simon Riggs wrote:
For me, it would be good to see a --parallel=n parameter that would
allow pg_loader to distribute rows in "round-robin" manner to "n"
different concurrent COPY statements. i.e. a non-routing version.
Let me expand on this. In many of these giant COPY sit
On Wed, 6 Feb 2008, Dimitri Fontaine wrote:
Did you compare to COPY or \copy?
COPY. If you're loading a TB, if you're smart it's going onto the server
itself if it all possible and loading directly from there. Would probably
get a closer comparision against psql \copy, but recognize you're
Le mercredi 06 février 2008, Simon Riggs a écrit :
> For me, it would be good to see a --parallel=n parameter that would
> allow pg_loader to distribute rows in "round-robin" manner to "n"
> different concurrent COPY statements. i.e. a non-routing version.
What happen when you want at most N paral
On Wed, 2008-02-06 at 12:27 +0100, Dimitri Fontaine wrote:
> Multi-Threading behavior and CE support
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>
> Now, pgloader will be able to run N threads, each one loading some
> data to a
> partitionned child-table target. N will certainly be configured
> depending
1 - 100 of 177 matches
Mail list logo
