Re: [GENERAL] Scaling Database for heavy load

2016-05-11 Thread Melvin Davidson
On Wed, May 11, 2016 at 8:52 PM, Scott Marlowe wrote: > On Wed, May 11, 2016 at 4:09 AM, Digit Penguin > wrote: > > Hello, > > > > > > we use PostgreSql 9.x in conjunction with BIND/DNS for some Companies > with > > about 1.000 queries per second. > > Now we have to scale the system up to 100.00

Re: [GENERAL] Scaling Database for heavy load

2016-05-11 Thread Scott Marlowe
On Wed, May 11, 2016 at 4:09 AM, Digit Penguin wrote: > Hello, > > > we use PostgreSql 9.x in conjunction with BIND/DNS for some Companies with > about 1.000 queries per second. > Now we have to scale the system up to 100.000 queries per second (about). > > Bind/DNS is very light and i think can n

Re: [GENERAL] Scaling Database for heavy load

2016-05-11 Thread Chris Travers
On Wed, May 11, 2016 at 12:09 PM, Digit Penguin wrote: > Hello, > > > we use PostgreSql 9.x in conjunction with BIND/DNS for some Companies with > about 1.000 queries per second. > Now we have to scale the system up to 100.000 queries per second (about). > > Bind/DNS is very light and i think can

[GENERAL] Scaling Database for heavy load

2016-05-11 Thread Digit Penguin
Hello, we use PostgreSql 9.x in conjunction with BIND/DNS for some Companies with about 1.000 queries per second. Now we have to scale the system up to 100.000 queries per second (about). Bind/DNS is very light and i think can not give us bottleneck. The question is how to dimension the backend

Re: [GENERAL] scaling postgres - can child tables be in a different tablespace?

2015-08-04 Thread John R Pierce
On 8/4/2015 6:14 AM, Melvin Davidson wrote: As additional advice, to get the best performance, you will want all your tablespaces to be on separate spindles/disks. EG: disk1/tblspc1 disk2/tblspc2 disk3/tblspc3 ... disk99/tblspc99 actually, I find to get best pe

Re: [GENERAL] scaling postgres - can child tables be in a different tablespace?

2015-08-04 Thread Melvin Davidson
As additional advice, to get the best performance, you will want all your tablespaces to be on separate spindles/disks. EG: disk1/tblspc1 disk2/tblspc2 disk3/tblspc3 ... disk99/tblspc99 On Tue, Aug 4, 2015 at 5:31 AM, Albe Laurenz wrote: > Chris Withers wrote: > >

Re: [GENERAL] scaling postgres - can child tables be in a different tablespace?

2015-08-04 Thread Albe Laurenz
Chris Withers wrote: > This raises an interesting question: can a child table be in a different > tablespace to its parent and other children of that parent? Yes. Inheritance is a logical concept and is independent of physical placement. Yours, Laurenz Albe -- Sent via pgsql-general mailing li

Re: [GENERAL] scaling postgres

2015-08-04 Thread Chris Withers
On 04/08/2015 09:11, Seref Arikan wrote: I work in healthcare and patient centric records let me consider multiple servers for lots and lots of patients. The engineering team from instagram has been sharing their experience with postgres, which is possib

Re: [GENERAL] scaling postgres

2015-08-04 Thread Seref Arikan
On Tue, Aug 4, 2015 at 9:06 AM, Chris Withers wrote: > On 03/08/2015 08:34, Seref Arikan wrote: > >> >> >> At what point does postgres stop scaling? >> What happens when the computational load no longer fits on one >> machine? What are the options then? >> >> >> I think it is hard to

Re: [GENERAL] scaling postgres

2015-08-04 Thread Chris Withers
On 03/08/2015 08:34, Seref Arikan wrote: At what point does postgres stop scaling? What happens when the computational load no longer fits on one machine? What are the options then? I think it is hard to come up with blanket responses to generic questions such as "What happens wh

Re: [GENERAL] scaling postgres - can child tables be in a different tablespace?

2015-08-04 Thread Chris Withers
On 03/08/2015 08:40, Jony Cohen wrote: Servers now days reach very impressive write speeds and at rather low prices - it's simpler to split the write to 2 tables on different tablespaces/devices than 2 servers. This raises an interesting question: can a child table be in a different tablespace t

Re: [GENERAL] scaling postgres

2015-08-03 Thread Seref Arikan
On Mon, Aug 3, 2015 at 8:22 AM, Chris Withers wrote: > On 03/08/2015 08:15, Jony Cohen wrote: > >> SSD disks are cheep these days but they don't like repeated >> writes/deletes so it might cause problems down the line (hence my first RAM >> recommendation) >> >> as for keeping the raw data - you

Re: [GENERAL] scaling postgres

2015-08-03 Thread Chris Withers
On 03/08/2015 08:15, Jony Cohen wrote: SSD disks are cheep these days but they don't like repeated writes/deletes so it might cause problems down the line (hence my first RAM recommendation) as for keeping the raw data - you could easily do it if you use partitions, if you have daily partitio

Re: [GENERAL] scaling postgres

2015-08-03 Thread Jony Cohen
Hi, I'd recommend looking into 2 solutions here, depending if you want to keep the data or not and how "fault tolerant" is the app (could you loose data from the last hour?) If you could loose some data and you don't intend keeping the whole data set - I'd look at using RAM as your storage - it's t

[GENERAL] scaling postgres

2015-08-02 Thread Chris Withers
Hi All, I'm curious if there are recommendations for scaling postgres to what, for me, seems like "a lot" of data... The app in question currently writes around 1.5 billion rows into a table before rolling them up into tables that have a few million roll up rows each. That 1.5 billion row ta

Re: [GENERAL] Scaling PostgreSQL-9

2010-09-28 Thread Igor Neyman
> -Original Message- > From: sandeep prakash dhumale [mailto:sandy9...@rediffmail.com] > Sent: Tuesday, September 28, 2010 6:32 AM > To: pgsql-general@postgresql.org > Subject: Scaling PostgreSQL-9 > > Hello All, > > Need some help in scaling PostgreSQL: > > I have a table with 400M

Re: [GENERAL] Scaling PostgreSQL-9

2010-09-28 Thread Vick Khera
On Tue, Sep 28, 2010 at 8:56 AM, Sandy wrote: > Table has mobile number,status and expiry date. I can not partition on > expiry date as all SELECT's are on mobile number. > Then partition on the mobile number. If your updates and queries are all tied to that, then it is the ideal candidate. Yo

[GENERAL] Re: Re: [GENERAL] Scaling PostgreSQL-9

2010-09-28 Thread Sandy
On Tue, 28 Sep 2010 17:45:16 +0530 wrote >On Tue, Sep 28, 2010 at 6:31 AM, sandeep prakash dhumale wrote: I have a table with 400M records with 5 int columns having index only on 1 column. How is your data used?  Is the update done by the primary key?  Are the queries segmented in some way

Re: [GENERAL] Scaling PostgreSQL-9

2010-09-28 Thread Vick Khera
On Tue, Sep 28, 2010 at 6:31 AM, sandeep prakash dhumale < sandy9...@rediffmail.com> wrote: > I have a table with 400M records with 5 int columns having index only on 1 > column. > How is your data used? Is the update done by the primary key? Are the queries segmented in some way that may divid

[GENERAL] Scaling PostgreSQL-9

2010-09-28 Thread sandeep prakash dhumale
Hello All, Need some help in scaling PostgreSQL: I have a table with 400M records with 5 int columns having index only on 1 column. Rows are updated by a perl script which takes 10k numbers in one transactions and fires single single update in a loop on database keeping track of the result re

Re: [GENERAL] scaling

2000-01-26 Thread Ed Loehr
"Scott V. McGuire" wrote: > > > See http://www.deja.com/getdoc.xp?AN=563958392 > > > > I suspect vacuum also would have a dramatic impact on performance in > > your test case. > > Ok, well there's at least two things I should have said. I am running > with -F and I did a vacuum analyze last nig

Re: [GENERAL] scaling

2000-01-26 Thread Scott V. McGuire
On Wed, Jan 26, 2000 at 04:51:26PM -0600, Ed Loehr wrote: > See http://www.deja.com/getdoc.xp?AN=563958392 > > I suspect vacuum also would have a dramatic impact on performance in > your test case. > > Cheers, > Ed Loehr > Ok, well there's at least two things I should have said. I am running

Re: [GENERAL] scaling

2000-01-26 Thread Ed Loehr
"Scott V. McGuire" wrote: > > I'm surprised about how long certain things are taking... > > What I'm surprised about is that the 10,000 row copies take < 1 > minute while the 5 row copies take > 10 minutes. See http://www.deja.com/getdoc.xp?AN=563958392 I suspect vacuum also would have a d

[GENERAL] scaling

2000-01-26 Thread Scott V. McGuire
Hi, I'm surprised about how long certain things are taking. Here's what I'm doing: 1. create a temp_table which is 5-7 columns by 1e4 - 1e5 rows. This part is pretty quick. 2. when its done, copy it into the big table which is 7 columns by about 5e7 rows. What I'm surprised about is that the