[GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-23 Thread Yan Chunlu
I was doing postgresql streaming replication, which was fine when two machine in the same datecenter. but recently I was planning to deploy new slave at a different datecent, the latency between the master and slave is 20ms; below is the related configurateion: Both master and slave have below conf

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-23 Thread Yan Chunlu
e: > On Jul 23, 2011, at 6:50 AM, Yan Chunlu wrote: > >> what does invalid record length and invalid magic number  normally >> means? xlog  corrupted? >> Thanks for any further help! > > It means your build settings for pg are not compatible across the 2 machines. > Fo

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-23 Thread Yan Chunlu
mas Vondra wrote: >> On 23 Červenec 2011, 18:14, Yan Chunlu wrote: >>> thanks for the help! >>> are there any other possible reasons? >>> >>> both system are using Debian amd64, as uname -a shows: >>> Linux washington 2.6.26-2-amd64 #1 SMP Tue Mar 9

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-23 Thread Yan Chunlu
less postgres didn't showing anything... cause it's binary, I tried with -a less postgres |grep -a ELF less postgres | grep -a Class nothing related to (ELF32 => 32bit, ELF64 => 64bit). On Sun, Jul 24, 2011 at 1:55 AM, Tomas Vondra wrote: > On 23 Červenec 2011, 18:1

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-23 Thread Yan Chunlu
the system date formats? looks the same: master: #date Sat Jul 23 21:53:34 CDT 2011 slave: #date Sat Jul 23 21:52:56 CDT 2011 On Sun, Jul 24, 2011 at 2:44 AM, Scott Marlowe wrote: > On Sat, Jul 23, 2011 at 11:55 AM, Tomas Vondra wrote: >> On 23 Červenec 2011, 18:14, Yan Chu

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-23 Thread Yan Chunlu
o disk, so when postgresql started it was seeing corrupted files? On Sun, Jul 24, 2011 at 11:23 AM, Scott Ribe wrote: > On Jul 23, 2011, at 8:43 PM, Yan Chunlu wrote: > >> I used apt-get to install postgresql, running pg_config showing they >> are exactly the same... > >

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-24 Thread Yan Chunlu
at 8:43 PM, Yan Chunlu wrote: > >> I used apt-get to install postgresql, running pg_config showing they >> are exactly the same... > > BTW, forgot to mention this in my first message: I run streaming replication > across the country with latency well over 100ms and no pr

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-24 Thread Yan Chunlu
checkpoint_segments = 64 wal_keep_segments = 128 On Sun, Jul 24, 2011 at 8:25 PM, Tomas Vondra wrote: > On 24 Červenec 2011, 6:09, Yan Chunlu wrote: >> thanks for all the help! >> >> @Adrian:  yes, only one instance on each machine >> >> not the slave fin

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-24 Thread Yan Chunlu
-24 10:56:11 CDT 2011-07-24 10:56:11 CDT @ LOG: invalid magic number in log file 87, segment 107, offset 4849664 On Sun, Jul 24, 2011 at 8:46 PM, Yan Chunlu wrote: > checkpoint_segments = 64 > wal_keep_segments = 128 > > On Sun, Jul 24, 2011 at 8:25 PM, Tomas Vondra wro

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-24 Thread Yan Chunlu
fter I complete sync data and start slave. If I stop slave later and restart, yes it could show xlog not found and can not catch master. but why the "invalid" things in the first place? On Mon, Jul 25, 2011 at 4:28 AM, Tomas Vondra wrote: > Dne 24.7.2011 14:46, Yan Chunlu napsa

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-25 Thread Yan Chunlu
9.0.4-1+b1 additional facilities for PostgreSQL On Mon, Jul 25, 2011 at 3:38 PM, Fujii Masao wrote: > On Mon, Jul 25, 2011 at 10:56 AM, Yan Chunlu wrote: >> I think the problem is still "invalid record length" and "invalid >> magic number", it start show

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-25 Thread Yan Chunlu
sorry for the typo, "debian and apt-get" On Mon, Jul 25, 2011 at 5:39 PM, Yan Chunlu wrote: > I am using debian and apt-get to install postgresql, dpkg list shows > they are the same?  is there anyway to tell what's version it is > compiled from? thanks! > > M

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-25 Thread Yan Chunlu
gcc compiler on my machine is 4.6.1, but I didn't compile it myself. just installed the binary from apt-get. I will try to compile it by myself to see what will happen On Mon, Jul 25, 2011 at 8:18 PM, Tomas Vondra wrote: > On 25 Červenec 2011, 11:39, Yan Chunlu wrote: >> I am us

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-25 Thread Yan Chunlu
hot-stanby server? seems master is okay. On Mon, Jul 25, 2011 at 10:06 PM, Tom Lane wrote: > "Tomas Vondra" writes: >> On 25 Červenec 2011, 11:39, Yan Chunlu wrote: >>> I am using debian ant apt-get to install postgresql, dpkg list shows >>> they are the

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-25 Thread Yan Chunlu
is okay? then I could compile my own version with 4.6.1 but how about the data? from the bug information, the data file seems not compatible I need to do pg_dump and pg_restore? what a nightmare. On Mon, Jul 25, 2011 at 10:47 PM, Tom Lane wrote: > Yan Chunlu writes: >>

Re: [GENERAL] streaming replication does not work across datacenter with 20ms latency?

2011-07-25 Thread Yan Chunlu
how about 4.3.2? I have gcc 4.3.2 compiled postgresql 9.0.4 as slave, is that okay that I turn the slave into master? so the switch will be a lot more smooth. On Tue, Jul 26, 2011 at 12:08 AM, Tom Lane wrote: > Yan Chunlu writes: >> oh god...thanks a lot for the tip. I did actually

[GENERAL] wiki.postgresql.org down?

2011-07-27 Thread Yan Chunlu
I tried several times to visit the site recently but the domain always can not be resolved. is there something wrong? -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] wiki.postgresql.org down?

2011-07-27 Thread Yan Chunlu
thanks! working now On Wed, Jul 27, 2011 at 3:54 PM, John R Pierce wrote: > On 07/27/11 12:48 AM, John R Pierce wrote: >> >> ...hopefully its reserved... > > errr.  i meant to type RESOLVED.  hah. > > -- > john r pierce                            N 37, W 122 > santa cruz ca                      

[GENERAL] how to adjust auto increment id offset?

2011-11-15 Thread Yan Chunlu
I would like to implement two master db with even-odd id sharding. in mysql it is fairly easy by using the configuration: auto_increment_offset = 1 auto_increment_increment = 2 but I have searched a lot didn't find anything related to this, some users doing this via trigger like "rubyrep". is

Re: [GENERAL] how to adjust auto increment id offset?

2011-11-15 Thread Yan Chunlu
b way but it seems they only choice for me, the delay within several minutes is acceptable. On Tue, Nov 15, 2011 at 5:12 PM, John R Pierce wrote: > On 11/15/11 12:56 AM, Bèrto ëd Sèra wrote: > >> Hi >> >> >> On 15 November 2011 11:44, Yan Chunlu > springri...

[GENERAL] how could duplicate pkey exist in psql?

2011-11-16 Thread Yan Chunlu
recently I have found several tables has exactly the same pkey, here is the definition: "diggcontent_data_account_pkey" PRIMARY KEY, btree (thing_id, key) the data is like this: 159292 | funnypics_link_point | 41 | num 159292 | funnypics_link_point | 40 | num

Re: [GENERAL] how could duplicate pkey exist in psql?

2011-11-17 Thread Yan Chunlu
I am using pgpool's replication feature, it does copy pg_xlog from one server to another, was that possible cause of the problem? thanks for the help! On Thu, Nov 17, 2011 at 5:38 PM, Edson Richter wrote: > > Em 17-11-2011 03:19, Yan Chunlu escreveu: > > recently I have fou

Re: [GENERAL] how could duplicate pkey exist in psql?

2011-11-17 Thread Yan Chunlu
seems they are identical: 159292 | |funnypicscn_link_karma| 159292 | |funnypicscn_link_karma| On Thu, Nov 17, 2011 at 4:07 PM, Szymon Guz wrote: > > > On 17 November 2011 06:19, Yan Chunlu wrote: > >> recently I have found several tables has exactly the same pke

Re: [GENERAL] how could duplicate pkey exist in psql?

2011-11-21 Thread Yan Chunlu
wrote: > Em 17-11-2011 09:21, Yan Chunlu escreveu: > > I am using pgpool's replication feature, it does copy pg_xlog from one > server to another, was that possible cause of the problem? > > > I did not mean that this IS your problem, I just gave you a tip regarding >

Re: [GENERAL] how could duplicate pkey exist in psql?

2011-11-21 Thread Yan Chunlu
nique constraint "account_pkey" does that means one duplicate record will prevent all other records to import? On Mon, Nov 21, 2011 at 10:55 PM, Adrian Klaver wrote: > On Monday, November 21, 2011 6:39:55 am Yan Chunlu wrote: > > got it. thank you very much for you help. I fo

[GENERAL] pg_dump and pg_restore make data dir doubled size?

2011-12-18 Thread Yan Chunlu
I just tried several times to make sure I didn't do anything wrong. I use pg_dump -U postgres -E UTF8 -F c -b -v -a dbcontent -f data.backup then pg_restore -a -d dbcontent data.backup to restore a db, and the data dir's size on the source machine is 56GB, after I restored data on the target

Re: [GENERAL] pg_dump and pg_restore make data dir doubled size?

2011-12-18 Thread Yan Chunlu
never mind. just found I changed the wal_keep_segments to 5000, it's the size of pg_xlog dir who used a lot of space. On Sun, Dec 18, 2011 at 10:18 PM, Yan Chunlu wrote: > I just tried several times to make sure I didn't do anything wrong. > > I use > pg_dump -U postgre

[GENERAL] ignore duplicate key while using COPY?

2011-12-18 Thread Yan Chunlu
I am using COPY public.table_name FROM STDIN to import data. it is very efficient, but if there's any duplicate key exists, the whole procedure has been stopped. is there anyway to around this? why does not postgresql just give a warning and continue the copy? -- Sent via pgsql-general mailin

Re: [GENERAL] ignore duplicate key while using COPY?

2011-12-21 Thread Yan Chunlu
thanks a lot! On Mon, Dec 19, 2011 at 10:19 PM, Adrian Klaver wrote: > On Sunday, December 18, 2011 10:54:21 pm Yan Chunlu wrote: > > I am using COPY public.table_name FROM STDIN to import data. it is > > very efficient, but if there's any duplicate key exists, the whole &g

[GENERAL] select statment going slow and slow while using IN (xx,xx)

2012-01-12 Thread Yan Chunlu
I am transforming a db with millions records to anther schema. for some reason I need to select the records using IN (xx,xx). the ids in the IN was about 1000 recored every time, but I found the query was getting slow while the selection moving on. the shared buffer is 2048M. and the cpu and i

Re: [GENERAL] select statment going slow and slow while using IN (xx,xx)

2012-01-12 Thread Yan Chunlu
nd: (thing_id = ANY ('{164438,112478,102941,112377,164442,181764,104028}'::bigint[])) On Thu, Jan 12, 2012 at 4:47 PM, Yan Chunlu wrote: > I am transforming a db with millions records to anther schema.  for > some reason I need to select the records using IN (xx,xx). > > the i

Re: [GENERAL] select statment going slow and slow while using IN (xx,xx)

2012-01-13 Thread Yan Chunlu
seems similar to this problem: http://stackoverflow.com/questions/5198380/improving-postgres-psycopg2-query-performance-for-python-to-the-same-level-of-ja but no solution yet. On Thu, Jan 12, 2012 at 5:00 PM, Yan Chunlu wrote: > I also tried explain but found nothing special: > > expla