Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-08-16 Thread Amitabh Kant
On Tue, Aug 16, 2016 at 1:08 PM, Achilleas Mantzios < ach...@matrix.gatewaynet.com> wrote: > On 29/07/2016 21:06, Larry Rosenman wrote: > >> On 2016-07-29 12:59, Bruce Momjian wrote: >> >>> On Fri, Jul 29, 2016 at 07:49:36PM +0200, Maeldron T. wrote: >>> And yes, I hate upgrading PostgreSQL e

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-08-16 Thread Achilleas Mantzios
On 29/07/2016 21:06, Larry Rosenman wrote: On 2016-07-29 12:59, Bruce Momjian wrote: On Fri, Jul 29, 2016 at 07:49:36PM +0200, Maeldron T. wrote: And yes, I hate upgrading PostgreSQL especially on FreeBSD where pg_upgrade isn’t really an option. Is that because it is hard to install the old a

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-08-05 Thread Tatsuo Ishii
> Yes, the VACUUM truncation is still an issue. But statements are > retryable, just like deadlocks. > > Unfo the truncation logic always kicks in or small tables of less than > 16 blocks. It's more forgiving on bigger tables. Oh, I didn't know that. Thanks for the info. > Maybe we could defer t

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-08-05 Thread Simon Riggs
On 5 August 2016 at 14:06, Tatsuo Ishii wrote: > On primary: > test=# vacuum verbose t1; > INFO: vacuuming "public.t1" > INFO: "t1": removed 3 row versions in 1 pages > INFO: "t1": found 3 removable, 0 nonremovable row versions in 1 out of 1 > pages > DETAIL: 0 dead row versions cannot be re

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-08-05 Thread Tatsuo Ishii
>>> https://www.postgresql.org/docs/current/static/runtime-config-replication.html#GUC-HOT-STANDBY-FEEDBACK >> >> I wonder if their problem could be fixed by using >> hot_standby_feedback. I have encountered similar problem but it seems >> hot_standby_feedback was not any help in this case: >> >>

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-08-05 Thread Simon Riggs
On 5 August 2016 at 09:16, Tatsuo Ishii wrote: >> On 27/07/16 18:54, Chris Travers wrote: >>> Another one I think they obliquely referred to (in the subtle problems >>> section) was the fact that if you have longer-running queries on the >>> replica with a lot of updates, you can get funny auto-va

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-08-05 Thread Tatsuo Ishii
> On 27/07/16 18:54, Chris Travers wrote: >> Another one I think they obliquely referred to (in the subtle problems >> section) was the fact that if you have longer-running queries on the >> replica with a lot of updates, you can get funny auto-vacuum-induced >> errors (writes from autovacuum on th

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-08-01 Thread Bruce Momjian
On Mon, Aug 1, 2016 at 11:54:00AM -0700, Jeff Janes wrote: > > Uh, that is only true if the slowness was in _dumping_ many objects. > > Most of the fixes have been for _restoring_ many objects, and that is > > done in the new cluster, so they should be OK. > > There have been improvements on both

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-08-01 Thread Jeff Janes
On Thu, Jul 28, 2016 at 3:16 PM, Bruce Momjian wrote: > On Thu, Jul 28, 2016 at 12:35:23AM -0700, Jeff Janes wrote: >> On Wed, Jul 27, 2016 at 9:48 PM, John R Pierce wrote: >> > On 7/27/2016 9:39 PM, Jeff Janes wrote: >> >> >> >> That depends on how how many objects there are consuming that 1 TB.

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-30 Thread Jason Dusek
On Thu, 28 Jul 2016 at 01:18 Gavin Flower wrote: > On 28/07/16 17:52, Jason Dusek wrote: > > With regards to write amplification, it makes me think about about > > OIDs. Used to be, every row had an OID and that OID persisted across > > row versions. > > > > > https://www.postgresql.org/docs/9.5/

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Kevin Grittner
On Fri, Jul 29, 2016 at 3:18 PM, Condor wrote: > On 29-07-2016 20:33, Jerry Sievers wrote: >> I've done several ~7TB pg_upgrades and with the hard link option and a >> framework that parallelizes the post-analyzer phase... >> >> ...45 minutes till completion. > GL to you Luck has nothing to do

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Condor
On 29-07-2016 20:33, Jerry Sievers wrote: Condor writes: On 26-07-2016 21:04, Dorian Hoxha wrote: Many comments: https://news.ycombinator.com/item?id=12166585 https://www.reddit.com/r/programming/comments/4uph84/why_uber_engineering_switched_from_postgres_to/ On Tue, Jul 26, 2016 at 7:39 PM

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Bruce Momjian
On Fri, Jul 29, 2016 at 02:50:32PM -0400, Stephen Frost wrote: > > Er, using a not yet invented pg_downgrade:-) > > The short answer is 'no'. Consider a case like the GIN page changes- as > soon as you execute DML on a column that has a GIN index on it, we're > going to rewrite that page using a

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Bruce Momjian
On Fri, Jul 29, 2016 at 03:03:46PM -0400, D'Arcy J.M. Cain wrote: > As does NetBSD. The problem is that unlike Python (which BSD allows > multiple versions) there is only one executable to deal with. It's not > an insurmountable problem but it could get messy. > > The answer is either chroot or

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread D'Arcy J.M. Cain
On Fri, 29 Jul 2016 13:06:04 -0500 Larry Rosenman wrote: > > Is that because it is hard to install the old and new clusters on > > the same server on FreeBSD? > > > The current FreeBSD Ports collection ports only allow ONE version to > be installed at a time. As does NetBSD. The problem is that

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Stephen Frost
Jerry, * Jerry Sievers (gsiever...@comcast.net) wrote: > Bruce Momjian writes: > > I agree, but I am not sure how to improve it. The big complaint I have > > heard is that once you upgrade and open up writes on the upgraded > > server, you can't re-apply those writes to the old server if you nee

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Jerry Sievers
Bruce Momjian writes: > On Wed, Jul 27, 2016 at 10:22:27AM -0400, Scott Mead wrote: > >> That being said, it doesn't really provide a back-out plan.  The beauty of >> replication is that you can halt the upgrade at any point if need be and cut >> your (hopefully small) losses. If you use -k, you

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Larry Rosenman
On 2016-07-29 12:59, Bruce Momjian wrote: On Fri, Jul 29, 2016 at 07:49:36PM +0200, Maeldron T. wrote: And yes, I hate upgrading PostgreSQL especially on FreeBSD where pg_upgrade isn’t really an option. Is that because it is hard to install the old and new clusters on the same server on FreeB

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Bruce Momjian
On Fri, Jul 29, 2016 at 07:49:36PM +0200, Maeldron T. wrote: > And yes, I hate upgrading PostgreSQL especially on FreeBSD where pg_upgrade > isn’t really an option. Is that because it is hard to install the old and new clusters on the same server on FreeBSD? -- Bruce Momjian http://mo

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Maeldron T.
On 26/07/16 19:39, Guyren Howe wrote: Honestly, I've never heard of anyone doing that. But it sounds like they had good reasons. https://eng.uber.com/mysql-migration/ Thoughts? 1. Open the page 2. Press Cmd-F on Mac, Ctrl-F on Linux/Windows. (Find on page) 3. Type "transaction" in the sear

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Jerry Sievers
Condor writes: > On 26-07-2016 21:04, Dorian Hoxha wrote: > >> Many comments: https://news.ycombinator.com/item?id=12166585 >> https://www.reddit.com/r/programming/comments/4uph84/why_uber_engineering_switched_from_postgres_to/ >> >> On Tue, Jul 26, 2016 at 7:39 PM, Guyren Howe wrote: >> >>> Hon

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-29 Thread Bruce Momjian
On Wed, Jul 27, 2016 at 01:02:40PM -0400, Bruce Momjian wrote: > Watching the video was helpful: > > https://vimeo.com/145842299 > > You can see the failover happened because of various user errors. That > doesn't excuse our bug, but I am not sure exactly how much they > understood of Post

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Achilleas Mantzios
On 28/07/2016 21:09, Edson Richter wrote: Em 28/07/2016 13:07, Chris Travers escreveu: On Thu, Jul 28, 2016 at 3:38 PM, Scott Marlowe mailto:scott.marl...@gmail.com>> wrote: On Wed, Jul 27, 2016 at 9:51 AM, Geoff Winkless mailto:pgsqlad...@geoff.dj>> wrote: > On 27 July 2016 at 15:22,

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Chris Travers
On Thu, Jul 28, 2016 at 8:09 PM, Edson Richter wrote: > Em 28/07/2016 13:07, Chris Travers escreveu: > > > > On Thu, Jul 28, 2016 at 3:38 PM, Scott Marlowe > wrote: > >> On Wed, Jul 27, 2016 at 9:51 AM, Geoff Winkless >> wrote: >> > On 27 July 2016 at 15:22, Scott Mead wrote: >> >> >> >> "The

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Joshua D. Drake
On 07/28/2016 04:58 PM, Joe Conway wrote: On 07/28/2016 03:16 PM, Bruce Momjian wrote: Not really true. I ran into two separate cases where on older (pre 9.3 I believe) Postgres if you had hundreds of thousands of tables (in the case I remember well, it was about 500k tables) the schema dump fro

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Joe Conway
On 07/28/2016 03:16 PM, Bruce Momjian wrote: > On Thu, Jul 28, 2016 at 12:35:23AM -0700, Jeff Janes wrote: >> On Wed, Jul 27, 2016 at 9:48 PM, John R Pierce wrote: >>> On 7/27/2016 9:39 PM, Jeff Janes wrote: That depends on how how many objects there are consuming that 1 TB. With mi

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Bruce Momjian
On Thu, Jul 28, 2016 at 03:26:17PM -0700, John R Pierce wrote: > Uh, that is only true if the slowness was in _dumping_ many objects. > Most of the fixes have been for _restoring_ many objects, and that is > done in the new cluster, so they should be OK. > > > I thought we were talkin

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread John R Pierce
On 7/28/2016 3:16 PM, Bruce Momjian wrote: On Thu, Jul 28, 2016 at 12:35:23AM -0700, Jeff Janes wrote: >On Wed, Jul 27, 2016 at 9:48 PM, John R Pierce wrote: > >On 7/27/2016 9:39 PM, Jeff Janes wrote: > >> > >>That depends on how how many objects there are consuming that 1 TB. > >>With millio

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Bruce Momjian
On Thu, Jul 28, 2016 at 12:35:23AM -0700, Jeff Janes wrote: > On Wed, Jul 27, 2016 at 9:48 PM, John R Pierce wrote: > > On 7/27/2016 9:39 PM, Jeff Janes wrote: > >> > >> That depends on how how many objects there are consuming that 1 TB. > >> With millions of small objects, you will have problems.

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread D'Arcy J.M. Cain
On Thu, 28 Jul 2016 19:32:39 +0300 Alex Ignatov wrote: > On 28.07.2016 18:41, Igor Neyman wrote: > > "Compactable" or not, in "Oracle world" no DBA in their right mind > > will attempt major upgrade without having complete (and tested) > > backup that could be used in case of upgrade going wrong.

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Edson Richter
Em 28/07/2016 13:07, Chris Travers escreveu: On Thu, Jul 28, 2016 at 3:38 PM, Scott Marlowe mailto:scott.marl...@gmail.com>> wrote: On Wed, Jul 27, 2016 at 9:51 AM, Geoff Winkless mailto:pgsqlad...@geoff.dj>> wrote: > On 27 July 2016 at 15:22, Scott Mead mailto:sco...@openscg.com

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Joshua D. Drake
On 07/28/2016 10:43 AM, Scott Marlowe wrote: On Thu, Jul 28, 2016 at 11:23 AM, Alex Ignatov wrote: And? Oracle and MySql doesnt have it but can downgrade right out the box. Quick and easy. So you can swap between oracle 11 and 12 back and forth in a live environment with no downtime? Please

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Scott Marlowe
On Thu, Jul 28, 2016 at 11:23 AM, Alex Ignatov wrote: > > On 28.07.2016 19:43, Scott Marlowe wrote: >> >> On Thu, Jul 28, 2016 at 10:32 AM, Alex Ignatov >> wrote: >>> >>> Oh, so in contrast to "Oracle world" "Postgres world" DBA in their >>> right >>> to do major upgrade without complete and t

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Rakesh Kumar
> And? Oracle and MySql doesnt have it but can downgrade right out the box. > Quick and easy. No it is not for mysql. http://dev.mysql.com/doc/refman/5.7/en/downgrading.html "In-place Downgrade: Involves shutting down the new MySQL version, replacing the new MySQL binaries or packages with the o

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Alex Ignatov
On 28.07.2016 19:43, Scott Marlowe wrote: On Thu, Jul 28, 2016 at 10:32 AM, Alex Ignatov wrote: Oh, so in contrast to "Oracle world" "Postgres world" DBA in their right to do major upgrade without complete and tested backup? Ok, I understand you. In Postgres world there always sky is blue

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Scott Marlowe
On Thu, Jul 28, 2016 at 10:32 AM, Alex Ignatov wrote: > > Oh, so in contrast to "Oracle world" "Postgres world" DBA in their right > to do major upgrade without complete and tested backup? > Ok, I understand you. In Postgres world there always sky is blue and sun is > shining. Of course we hav

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Chris Travers
r < >> rakeshkumar46...@gmail.com> >> Cc: PostgreSQL General >> Subject: Re: [GENERAL] Uber migrated from Postgres to MySQL >> >> >> On 28.07.2016 18:09, Igor Neyman wrote: >> >>> -Original Message- >>> From: pgsql-general-ow..

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Alex Ignatov
On 28.07.2016 18:41, Igor Neyman wrote: -Original Message- From: Alex Ignatov [mailto:a.igna...@postgrespro.ru] Sent: Thursday, July 28, 2016 11:26 AM To: Igor Neyman ; Rakesh Kumar Cc: PostgreSQL General Subject: Re: [GENERAL] Uber migrated from Postgres to MySQL On 28.07.2016 18

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Chris Travers
On Thu, Jul 28, 2016 at 3:38 PM, Scott Marlowe wrote: > On Wed, Jul 27, 2016 at 9:51 AM, Geoff Winkless > wrote: > > On 27 July 2016 at 15:22, Scott Mead wrote: > >> > >> "The bug we ran into only affected certain releases of Postgres 9.2 and > >> has been fixed for a long time now. However, w

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Igor Neyman
-Original Message- From: Alex Ignatov [mailto:a.igna...@postgrespro.ru] Sent: Thursday, July 28, 2016 11:26 AM To: Igor Neyman ; Rakesh Kumar Cc: PostgreSQL General Subject: Re: [GENERAL] Uber migrated from Postgres to MySQL On 28.07.2016 18:09, Igor Neyman wrote: > -Origi

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Geoff Winkless
On 28 July 2016 at 16:34, Igor Neyman wrote: > Which means that you can make use of some new feature, but definitely not > all. > That makes "downgrade" feature very, very limited, if useful at all. > ​ Sufficient to allow you to run the upgrade, find that there's a catastrophic bug in the new v

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Igor Neyman
-Original Message- From: Alex Ignatov [mailto:a.igna...@postgrespro.ru] Sent: Thursday, July 28, 2016 11:26 AM To: Igor Neyman ; Rakesh Kumar Cc: PostgreSQL General Subject: Re: [GENERAL] Uber migrated from Postgres to MySQL On 28.07.2016 18:09, Igor Neyman wrote: > -Origi

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Alex Ignatov
migrated from Postgres to MySQL On 28.07.2016 17:50, Rakesh Kumar wrote: On Thu, Jul 28, 2016 at 10:38 AM, Alex Ignatov wrote: Sorry, what? You can rollback to previous version of software what you had. https://docs.oracle.com/cd/E11882_01/server.112/e23633/downgrade.htm# UPGRD007 Not so fast. This

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Igor Neyman
-Original Message- From: pgsql-general-ow...@postgresql.org [mailto:pgsql-general-ow...@postgresql.org] On Behalf Of Alex Ignatov Sent: Thursday, July 28, 2016 10:59 AM To: Rakesh Kumar Cc: PostgreSQL General Subject: Re: [GENERAL] Uber migrated from Postgres to MySQL On 28.07.2016 17

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Alex Ignatov
On 28.07.2016 17:50, Rakesh Kumar wrote: On Thu, Jul 28, 2016 at 10:38 AM, Alex Ignatov wrote: Sorry, what? You can rollback to previous version of software what you had. https://docs.oracle.com/cd/E11882_01/server.112/e23633/downgrade.htm#UPGRD007 Not so fast. This requires the db to be se

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Rakesh Kumar
On Thu, Jul 28, 2016 at 10:38 AM, Alex Ignatov wrote: > Sorry, what? You can rollback to previous version of software what you had. > > https://docs.oracle.com/cd/E11882_01/server.112/e23633/downgrade.htm#UPGRD007 Not so fast. This requires the db to be set in compatibility mode of the lower ver

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Alex Ignatov
On 27.07.2016 19:51, Rakesh Kumar wrote: On Wed, Jul 27, 2016 at 12:47 PM, Bruce Momjian wrote: Yes. I was saying I don't know how to improve pg_upgrade to address it. This problem is there even in oracle/db2/sqlserver. None of them allow rollback to the lower version unless it is a minor v

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread James Keener
If we're talking about favourite bug https://bugs.mysql.com/bug.php?id=21153 is mine Join with many tables hangs mysql (and taking 100% cpu) > Description: > the following query hangs the mysql server taking 100% cpu. also an > "explain" of the query hangs the server! It's "not a bug" because yo

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Scott Marlowe
On Wed, Jul 27, 2016 at 9:51 AM, Geoff Winkless wrote: > On 27 July 2016 at 15:22, Scott Mead wrote: >> >> "The bug we ran into only affected certain releases of Postgres 9.2 and >> has been fixed for a long time now. However, we still find it worrisome that >> this class of bug can happen at al

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Gavin Flower
On 28/07/16 17:52, Jason Dusek wrote: With regards to write amplification, it makes me think about about OIDs. Used to be, every row had an OID and that OID persisted across row versions. https://www.postgresql.org/docs/9.5/static/runtime-config-compatible.html#GUC-DEFAULT-WITH-OIDS Would rei

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-28 Thread Jeff Janes
On Wed, Jul 27, 2016 at 9:48 PM, John R Pierce wrote: > On 7/27/2016 9:39 PM, Jeff Janes wrote: >> >> That depends on how how many objects there are consuming that 1 TB. >> With millions of small objects, you will have problems. Not as many >> in 9.5 as there were in 9.1, but still it does not sc

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Jason Dusek
With regards to write amplification, it makes me think about about OIDs. Used to be, every row had an OID and that OID persisted across row versions. https://www.postgresql.org/docs/9.5/static/runtime-config-compatible.html#GUC-DEFAULT-WITH-OIDS Would reintroducing such a feature address some of

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread James Keener
So, millions is a lot, but it's not difficult to get to a place where you have thousands or tables. Image a case in which census data and the associated geometries. https://github.com/censusreporter/census-postgres has 22 surveys, each with 230+ tables. That's 5000+ tables right there. Now, the T

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread John R Pierce
On 7/27/2016 9:39 PM, Jeff Janes wrote: That depends on how how many objects there are consuming that 1 TB. With millions of small objects, you will have problems. Not as many in 9.5 as there were in 9.1, but still it does not scale linearly in the number of objects. If you only have thousands

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Jeff Janes
On Wed, Jul 27, 2016 at 7:23 PM, Greg Sabino Mullane wrote: > Marc wrote: >> I donât have a 1TB database to try it on, mind you, so your >> âwait couple of daysâ might be *with* the âlink option? > > I think you mean *without*, but yeah, there is no way the --link > option is going to take that l

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Bruce Momjian
On Thu, Jul 28, 2016 at 02:23:18AM -, Greg Sabino Mullane wrote: > > Marc wrote: > > I donât have a 1TB database to try it on, mind you, so your > > âwait couple of daysâ might be *with* the âlink option? > > I think you mean *without*, but yeah, there is no way the --link > option is going

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Greg Sabino Mullane
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Marc wrote: > I donât have a 1TB database to try it on, mind you, so your > âwait couple of daysâ might be *with* the âlink option? I think you mean *without*, but yeah, there is no way the --link option is going to take that long. Hard links

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Bruce Momjian
On Wed, Jul 27, 2016 at 07:02:52PM -0400, Alvaro Herrera wrote: > Patrick B wrote: > > > > > > I think it's safe to say that that has absolutely nothing to do > > > with the size being 3TB. They symptoms you report are a little > > > thin to diagnose the actual cause. > > > > might be... we're us

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Alvaro Herrera
Patrick B wrote: > > > > I think it's safe to say that that has absolutely nothing to do > > with the size being 3TB. They symptoms you report are a little > > thin to diagnose the actual cause. > > might be... we're using SATA disks... and that's a big problem. But still.. > the size of the DB i

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Patrick B
> > I think it's safe to say that that has absolutely nothing to do > with the size being 3TB. They symptoms you report are a little > thin to diagnose the actual cause. might be... we're using SATA disks... and that's a big problem. But still.. the size of the DB is indeed a problem.

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Kevin Grittner
On Wed, Jul 27, 2016 at 4:34 PM, Patrick B wrote: > We can't use the pg_upgrade in our 3TB database just does not work.. > that's the main reason we're still using 9.2. I think it's safe to say that that has absolutely nothing to do with the size being 3TB. They symptoms you report are a lit

pg_upgrade and not working (was Re: [GENERAL] Uber migrated from Postgres to MySQL)

2016-07-27 Thread Andrew Sullivan
I bet there are some people around here who could help you troubleshoot. Just sayin' (I'm not among them.) A On Thu, Jul 28, 2016 at 09:34:05AM +1200, Patrick B wrote: > We can't use the pg_upgrade in our 3TB database just does not work.. > that's the main reason we're still using 9.2. --

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Patrick B
We can't use the pg_upgrade in our 3TB database just does not work.. that's the main reason we're still using 9.2.

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Andrew Sullivan
On Wed, Jul 27, 2016 at 02:33:54PM -0500, Kevin Grittner wrote: > Until you get to the end of the upgrade and *start the cluster > under the new version* you can fall back to the old version. Yeah, but to be fair a lot of well-funded businesses (note what started this discussion) are pico-managed

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Kevin Grittner
On Wed, Jul 27, 2016 at 9:22 AM, Scott Mead wrote: > On Wed, Jul 27, 2016 at 3:34 AM, Achilleas Mantzios > wrote: >> Our last 1TB upgrade from 9.0 -> 9.3 went like a charm in something like >> seconds. (with the -k option) >> However, be warned that the planing and testing took one full week.

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Kevin Grittner
On Wed, Jul 27, 2016 at 2:15 AM, Condor wrote: > They are right for upgrades. > It's a hard to shutdown 1 TB database and wait couple of days pg_upgrade to > finish upgrade and meanwhile database is offline. What? I had a cluster over 3TB and it was offline for only 10 minutes doing a major ver

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Andrew Sullivan
On Wed, Jul 27, 2016 at 01:58:25PM -0400, Rakesh Kumar wrote: > > I am surprised PG does not even allow minor version rollback. It almost never happens that a minor version (N.M.x, x is minor) requires an upgrade at all. Change your binaries and you're done. Catalogue incompatibility historicall

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Rakesh Kumar
On Wed, Jul 27, 2016 at 2:07 PM, Andrew Sullivan wrote: > It almost never happens that a minor version (N.M.x, x is minor) > requires an upgrade at all. Change your binaries and you're done. > Catalogue incompatibility historically was the basis for something > becoming a major version upgrade.

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Bruce Momjian
On Wed, Jul 27, 2016 at 01:58:25PM -0400, Rakesh Kumar wrote: > On Wed, Jul 27, 2016 at 1:54 PM, Marc Fournier > wrote: > > > Stupid question here, but do we provide any less then what MySQL does? I’m > > reading: > > mysql provides same functionality for rollback like oracle/db2 > provides. T

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Bruce Momjian
On Wed, Jul 27, 2016 at 10:54:25AM -0700, Marc Fournier wrote: > http://dev.mysql.com/doc/refman/5.7/en/downgrading.html#downgrade-paths > > == > >Unless otherwise documented, the following downgrade paths are > supported: > > • Downgrading from a release series version to an older rel

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Rakesh Kumar
On Wed, Jul 27, 2016 at 1:54 PM, Marc Fournier wrote: > Stupid question here, but do we provide any less then what MySQL does? I’m > reading: mysql provides same functionality for rollback like oracle/db2 provides. That is, rollback on a minor version upgrade possible, but not on major version

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Marc Fournier
> On Jul 27, 2016, at 09:59, Bruce Momjian wrote: > > I think long-term we are looking at pg_logical for zero-downtime > upgrades and _downgrades_, and pg_upgrade for less overhead (I don't > want to make a second copy of my data) upgrades (but not downgrades). > > I think this is probably the

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Vik Fearing
On 27/07/16 18:54, Chris Travers wrote: > Another one I think they obliquely referred to (in the subtle problems > section) was the fact that if you have longer-running queries on the > replica with a lot of updates, you can get funny auto-vacuum-induced > errors (writes from autovacuum on the mast

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Bruce Momjian
On Wed, Jul 27, 2016 at 12:59:59PM -0400, Bruce Momjian wrote: > On Wed, Jul 27, 2016 at 12:47:24PM -0400, Bruce Momjian wrote: > > On Wed, Jul 27, 2016 at 12:33:27PM -0400, Rakesh Kumar wrote: > > > On Wed, Jul 27, 2016 at 11:45 AM, Bruce Momjian wrote: > > > > > > > I agree, but I am not sure h

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Bruce Momjian
On Wed, Jul 27, 2016 at 09:17:58AM +0200, Chris Travers wrote: > The replication section made me wonder though if they were using the right > replication solution for the job.  If you don't want an on-disk copy, don't > use > physical replication.  This being said there is one serious issue here t

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Bruce Momjian
On Wed, Jul 27, 2016 at 12:47:24PM -0400, Bruce Momjian wrote: > On Wed, Jul 27, 2016 at 12:33:27PM -0400, Rakesh Kumar wrote: > > On Wed, Jul 27, 2016 at 11:45 AM, Bruce Momjian wrote: > > > > > I agree, but I am not sure how to improve it. The big complaint I have > > > heard is that once you

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Chris Travers
On Wed, Jul 27, 2016 at 4:22 PM, Scott Mead wrote: > On Wed, Jul 27, 2016 at 3:34 AM, Achilleas Mantzios < > ach...@matrix.gatewaynet.com> wrote: > >> On 27/07/2016 10:15, Condor wrote: >> >>> On 26-07-2016 21:04, Dorian Hoxha wrote: >>> Many comments: https://news.ycombinator.com/item?id=12

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Bruce Momjian
On Wed, Jul 27, 2016 at 12:51:40PM -0400, Rakesh Kumar wrote: > On Wed, Jul 27, 2016 at 12:47 PM, Bruce Momjian wrote: > > > Yes. I was saying I don't know how to improve pg_upgrade to address it. > > This problem is there even in oracle/db2/sqlserver. None of them allow > rollback to the lower

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Rakesh Kumar
On Wed, Jul 27, 2016 at 12:47 PM, Bruce Momjian wrote: > Yes. I was saying I don't know how to improve pg_upgrade to address it. This problem is there even in oracle/db2/sqlserver. None of them allow rollback to the lower version unless it is a minor version upgrade. Major version upgrade almos

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Bruce Momjian
On Wed, Jul 27, 2016 at 12:33:27PM -0400, Rakesh Kumar wrote: > On Wed, Jul 27, 2016 at 11:45 AM, Bruce Momjian wrote: > > > I agree, but I am not sure how to improve it. The big complaint I have > > heard is that once you upgrade and open up writes on the upgraded > > server, you can't re-apply

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Geoff Winkless
On 27 July 2016 at 17:11, Andrew Sullivan wrote: > Given > the discussion in the post in question, the decision to use MySQL > appears to have been well-justified: > ​Well yes, but that's pretty-much the point of back-justification, isn't it? ​[snip a whole bunch of good points] > > For what

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Rakesh Kumar
On Wed, Jul 27, 2016 at 11:45 AM, Bruce Momjian wrote: > I agree, but I am not sure how to improve it. The big complaint I have > heard is that once you upgrade and open up writes on the upgraded > server, you can't re-apply those writes to the old server if you need to > fall back to the old se

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Andrew Sullivan
On Wed, Jul 27, 2016 at 04:51:42PM +0100, Geoff Winkless wrote: > technical reasons. Most developers will harp on at their boss about how > terrible their current database is and how performs > much better. Eventually one of two things happens: either a) those > developers end up in a position whe

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Geoff Winkless
On 27 July 2016 at 15:22, Scott Mead wrote: > "The bug we ran into only affected certain releases of Postgres 9.2 and > has been fixed for a long time now. However, we still find it worrisome > that this class of bug can happen at all. A new version of Postgres could > be released at any time th

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Bruce Momjian
On Wed, Jul 27, 2016 at 10:22:27AM -0400, Scott Mead wrote: > That being said, it doesn't really provide a back-out plan.  The beauty of > replication is that you can halt the upgrade at any point if need be and cut > your (hopefully small) losses. If you use -k, you are all in.  Sure, you could >

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Scott Mead
On Wed, Jul 27, 2016 at 3:34 AM, Achilleas Mantzios < ach...@matrix.gatewaynet.com> wrote: > On 27/07/2016 10:15, Condor wrote: > >> On 26-07-2016 21:04, Dorian Hoxha wrote: >> >>> Many comments: https://news.ycombinator.com/item?id=12166585 >>> >>> https://www.reddit.com/r/programming/comments/4u

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Achilleas Mantzios
On 27/07/2016 10:15, Condor wrote: On 26-07-2016 21:04, Dorian Hoxha wrote: Many comments: https://news.ycombinator.com/item?id=12166585 https://www.reddit.com/r/programming/comments/4uph84/why_uber_engineering_switched_from_postgres_to/ On Tue, Jul 26, 2016 at 7:39 PM, Guyren Howe wrote: Ho

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Marc Fournier
> On Jul 27, 2016, at 00:15, Condor wrote: > > On 26-07-2016 21:04, Dorian Hoxha wrote: >> Many comments: https://news.ycombinator.com/item?id=12166585 >> https://www.reddit.com/r/programming/comments/4uph84/why_uber_engineering_switched_from_postgres_to/ >> On Tue, Jul 26, 2016 at 7:39 PM, Guyr

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Chris Travers
Just a few points on reading this. First, the timeline bugs regarding replication (particularly iirc in the 9.1 days). I remember accidentally corrupting a (fortunately only demonstration!) database cluster in the process of demonstrating promotion at least once. Iirc last time I tried to reprod

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-27 Thread Condor
On 26-07-2016 21:04, Dorian Hoxha wrote: Many comments: https://news.ycombinator.com/item?id=12166585 https://www.reddit.com/r/programming/comments/4uph84/why_uber_engineering_switched_from_postgres_to/ On Tue, Jul 26, 2016 at 7:39 PM, Guyren Howe wrote: Honestly, I've never heard of anyone d

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-26 Thread Rakesh Kumar
-- Sent from mobile. On Jul 26, 2016, at 5:56 PM, Joshua D. Drake wrote: > On 07/26/2016 02:49 PM, Rakesh Kumar wrote: > This is an old news. They are using mysql as a nosql to store schemaless. > Basically one giant blob col. And thats where the role of mysql ends. The > bulk of the proces

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-26 Thread Joshua D. Drake
On 07/26/2016 02:49 PM, Rakesh Kumar wrote: This is an old news. They are using mysql as a nosql to store schemaless. Basically one giant blob col. And thats where the role of mysql ends. The bulk of the processing will be in nosql. That doesn't mean they didn't bring up some very good points

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-26 Thread Rakesh Kumar
This is an old news. They are using mysql as a nosql to store schemaless. Basically one giant blob col. And thats where the role of mysql ends. The bulk of the processing will be in nosql. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscript

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-26 Thread Dorian Hoxha
Many comments: https://news.ycombinator.com/item?id=12166585 https://www.reddit.com/r/programming/comments/4uph84/why_uber_engineering_switched_from_postgres_to/ On Tue, Jul 26, 2016 at 7:39 PM, Guyren Howe wrote: > Honestly, I've never heard of anyone doing that. But it sounds like they > had

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-26 Thread Joshua D. Drake
On 07/26/2016 10:39 AM, Guyren Howe wrote: Honestly, I've never heard of anyone doing that. But it sounds like they had good reasons. https://eng.uber.com/mysql-migration/ Thoughts? Hello, I started a thread about hackers on this. Sincerely, JD -- Command Prompt, Inc. h