Re: [HACKERS] tweaking MemSet() performance - 7.4.5
On Sat, 2004-09-25 at 23:23 +0200, Manfred Spraul wrote: > [EMAIL PROTECTED] wrote: > > >>If the memset > >>bypasses the cache then the following access will cause a cache line > >>miss, which can be so slow that using the faster memset can result in a > >>net performance loss. > >> > >> > >> > > > >Could you suggest some structs to test? If I get your meaning, I would make a loop > >that sets then reads from the structure. > > > > > > > Read the sources and the cpu specs. Benchmarking such problems is > virtually impossible. > I don't have OS-X, thus I checked the Linux-kernel sources: It seems > that the power architecture doesn't have the same problem as x86. > There is a special clear cacheline instruction for large memsets and the > rest is done through carefully optimized store byte/halfword/word/double > word sequences. > > Thus I'd check what happens if you memset not perfectly aligned buffers. > That's another point where over-optimized functions sometimes break > down. If there is no slowdown, then I'd replace the postgres function > with the OS provided function. > > I'd add some __builtin_constant_p() optimizations, but I guess Tom won't > like gcc hacks ;-) I think it cannot be problem if you write it to some .h file (in port directory?) as macro with "#ifdef GCC". The other thing is real advantage of hacks like this in practical PG usage :-) Karel -- Karel Zak http://home.zf.jcu.cz/~zakkr ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] tweaking MemSet() performance - 7.4.5
Karel Zak wrote: > On Sat, 2004-09-25 at 23:23 +0200, Manfred Spraul wrote: > > [EMAIL PROTECTED] wrote: > > > > >>If the memset > > >>bypasses the cache then the following access will cause a cache line > > >>miss, which can be so slow that using the faster memset can result in a > > >>net performance loss. > > >> > > >> > > >> > > > > > >Could you suggest some structs to test? If I get your meaning, I would make a > > >loop that sets then reads from the structure. > > > > > > > > > > > Read the sources and the cpu specs. Benchmarking such problems is > > virtually impossible. > > I don't have OS-X, thus I checked the Linux-kernel sources: It seems > > that the power architecture doesn't have the same problem as x86. > > There is a special clear cacheline instruction for large memsets and the > > rest is done through carefully optimized store byte/halfword/word/double > > word sequences. > > > > Thus I'd check what happens if you memset not perfectly aligned buffers. > > That's another point where over-optimized functions sometimes break > > down. If there is no slowdown, then I'd replace the postgres function > > with the OS provided function. > > > > I'd add some __builtin_constant_p() optimizations, but I guess Tom won't > > like gcc hacks ;-) > > I think it cannot be problem if you write it to some .h file (in port > directory?) as macro with "#ifdef GCC". The other thing is real > advantage of hacks like this in practical PG usage :-) The reason MemSet is a win is not that the C code is great but because it eliminates a function call. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] shared memory release following failed lock acquirement.
Tgl wrote: > > As I see it, this means the user-locks (and perhaps all > > locks...?) eat around ~ 6k bytes memory each. > > They're allocated in groups of 32, which would work out to close to 6k; > maybe you were measuring the incremental cost of allocating the first one? I got my 6k figure by dividing 1 into 64M, 1 being the value that crashed the server. That's reasonable because doubling shared buffers slightly more than doubled the crash value. I was wondering how ~ 10k locks ran me out of shared memory when each lock takes ~ 260b (half that, as you say) and I am running 8k buffers = 64M. 260 * 100 backends * 64 maxlocks = 1.7 M. Sure, the hash table and other stuff adds some...but this is no where near what it should take to run me out. Am I just totally misunderstanding how to estimate locks memory consumption? Merlin ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] shared memory release following failed lock acquirement.
"Merlin Moncure" <[EMAIL PROTECTED]> writes: > I was wondering how ~ 10k locks ran me out of shared memory when each > lock takes ~ 260b (half that, as you say) and I am running 8k buffers = > 64M. The number of buffers you have doesn't have anything to do with this. The question is how much shared memory space is there for the lock table, above and beyond what's used for everything else (such as buffers). I just went through and corrected some minor errors in the calculation of shared memory block size (mostly stuff where the estimation code had gotten out of sync with the actual work over time). I now find that with all-default configuration parameters I can create 7808 locks before running out of shared memory, rather than the promised 6400. (YMMV due to platform-specific differences in MAXALIGN, sizeof(pointer), etc.) This is coming from two places: LockShmemSize deliberately adds on 10% slop factor to its calculation of the lock table size, and then CreateSharedMemoryAndSemaphores adds on 100KB for safety margin. Both of those numbers are kinda pulled from the air, but I don't see a strong reason to change them. The other space calculations seem to be pretty nearly dead-on. regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] shared memory release following failed lock acquirement.
>Tom Lane > "Merlin Moncure" <[EMAIL PROTECTED]> writes: > > According to postgresql.conf, using these settings the lock table eats > > 64*260*100 bytes = < 2M. Well, if it's running my server out of shared > > memory, it's eating much, much more shmem than previously thought. > > Hmm, the 260 is out of date I think. I was seeing about 184 bytes/lock > in my tests just now. > > > Also, I was able to acquire around 10k locks before the server borked. > > This is obviously a lot more than than 64*100. > > Sure, because there's about 100K of deliberate slop in the shared memory > size allocation, and you are probably also testing a scenario where the > buffer and FSM hash tables haven't ramped to full size yet, so the lock > table is able to eat more than the nominal amount of space. > > > As I see it, this means the user-locks (and perhaps all > > locks...?) eat around ~ 6k bytes memory each. > > They're allocated in groups of 32, which would work out to close to 6k; > maybe you were measuring the incremental cost of allocating the first one? > > I did some digging, and as far as I can see the only shared memory > allocations that occur after postmaster startup are for the four shmem > hash tables: buffers, FSM relations, locks, and proclocks. Of these, > the buffer and FSM hashtables have predetermined maximum sizes. So > arranging for the space in those tables to be fully preallocated should > prevent any continuing problems from lock table overflow. I've > committed a fix that does this. I verified that after running the thing > out of shared memory via creating a lot of user locks and then releasing > same, I could run the regression tests. > Few questions: Is that fix in 8.0? Does this mean that the parameter max_locks_per_transaction isn't honoured at all, it is just used to size the lock table - which itself can expand beyond that max limit in various circumstances? (Though with the bug fix, not THAT much more than the max limit) Should we rename and redocument the parameter? If that is so, the current name is so far away from its real meaning as to constitute a bug in itself Best Regards, Simon Riggs ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] shared memory release following failed lock acquirement.
"Simon Riggs" <[EMAIL PROTECTED]> writes: > Does this mean that the parameter max_locks_per_transaction isn't honoured > at all, it is just used to size the lock table Yes, and that's how it's documented. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] tweaking MemSet() performance - 7.4.5
On Wed, 2004-09-29 at 21:37, Bruce Momjian wrote: > The reason MemSet is a win is not that the C code is great but because > it eliminates a function call. A reasonable compiler ought to be able to implement memset() as a compiler intrinsic where it makes sense to do so. MSVC++ can certainly do this; per the GCC 3.4 docs, it seems GCC can/does as well: The ISO C90 functions abort, abs, acos, asin, atan2, atan, calloc, ceil, cosh, cos, exit, exp, fabs, floor, fmod, fprintf, fputs, frexp, fscanf, labs, ldexp, log10, log, malloc, memcmp, memcpy, memset, modf, pow, printf, putchar, puts, scanf, sinh, sin, snprintf, sprintf, sqrt, sscanf, strcat, strchr, strcmp, strcpy, strcspn, strlen, strncat, strncmp, strncpy, strpbrk, strrchr, strspn, strstr, tanh, tan, vfprintf, vprintf and vsprintf are all recognized as built-in functions unless -fno-builtin is specified (or -fno-builtin-function is specified for an individual function). All of these functions have corresponding versions prefixed with __builtin_. (http://gcc.gnu.org/onlinedocs/gcc-3.4.2/gcc/Other-Builtins.html#Other-Builtins) -Neil ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] tweaking MemSet() performance - 7.4.5
Neil Conway wrote: > On Wed, 2004-09-29 at 21:37, Bruce Momjian wrote: > > The reason MemSet is a win is not that the C code is great but because > > it eliminates a function call. > > A reasonable compiler ought to be able to implement memset() as a > compiler intrinsic where it makes sense to do so. MSVC++ can certainly > do this; per the GCC 3.4 docs, it seems GCC can/does as well: > > The ISO C90 functions abort, abs, acos, asin, atan2, atan, calloc, ceil, > cosh, cos, exit, exp, fabs, floor, fmod, fprintf, fputs, frexp, fscanf, > labs, ldexp, log10, log, malloc, memcmp, memcpy, memset, modf, pow, > printf, putchar, puts, scanf, sinh, sin, snprintf, sprintf, sqrt, > sscanf, strcat, strchr, strcmp, strcpy, strcspn, strlen, strncat, > strncmp, strncpy, strpbrk, strrchr, strspn, strstr, tanh, tan, vfprintf, > vprintf and vsprintf are all recognized as built-in functions unless > -fno-builtin is specified (or -fno-builtin-function is specified for an > individual function). All of these functions have corresponding versions > prefixed with __builtin_. > > (http://gcc.gnu.org/onlinedocs/gcc-3.4.2/gcc/Other-Builtins.html#Other-Builtins) MemSet was written when gcc 2.X wasn't even stable yet. Have you run any tests on 3.4 to see if MemSet is still a win with that compiler? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
[HACKERS] Fwd: error: unicode characters greater than or equal to 0x10000
good day everyone!
i am using postgresql-8.0beta2-dev3 windows port, with postgis-0.9
add-on. anyway, while trying to insert a table using shp2pgsql, i get
this error:
psql:temp.sql:38: ERROR: Unicode characters greater than or equal to
0x1 are not supported
this is line 38 of temp.sql:
INSERT INTO "esiadminbounds"
(gid,"area","town","barangay","dens","pop90","popd90","pop95","popd95","pop97","popd97",the_geom)
VALUES
('37','1.424549','Bauan','Pitogo','0.00','0','4.03','0','0.00','0','0.',GeometryFromText('MULTIPOLYGON(((120.955684569578
13.7725006999301 ,120.9554735709 13.7724626988676 ,120.955403570202
13.7724506990452 ,120.955229571123 13.7722566997561 ,120.95522957124
13.7722516993503 ,120.955229570082 13.7722466989339 ,120.955070571067
13.7722456995021 ,120.954931571036 13.7722436992655 ,120.954931571145
13.7722496995042 ,120.954897570853 13.7722496999685 ,120.954776571639
13.7722426992188 ,120.954677571648 13.7722366990558 ,120.954480571351
13.772390698913 ,120.954469571209 13.7723906997389 ,120.954469571997
13.7723996994764 ,120.954418571133 13.7724396989397 ,120.954417571232
13.7724396993554 ,120.954417571463 13.7724406991889 ,120.954378572154
13.7724706990999 ,120.95426457253 13.7723926996639 ,120.954264571623
13.7723886990838 ,120.954258572245 13.7723886990816 ,120.954236572153
13.7723726996713 ,120.954121572059 13.7724486995595 ,120.954069571652
13.7724736993948 ,120.954011571976 13.7725216990704 ,120.95388257267
13.7726066993588 ,120.953660572868 13.7726926990537 ,120.953463573
13.7727506992206 ,120.9533925736 13.7726866993123 ,120.953150573447
13.7724657000595 ,120.951933574673 13.77245579 ,120.95057757599
13.7724446997878 ,120.9498125778 13.772822700065 ,120.949654577942
13.7728726991687 ,120.949389577744 13.7731016997272 ,120.949284578733
13.7731526999402 ,120.949072578313 13.7733046998845 ,120.948914579293
13.773354699701 ,120.948548579143 13.773526699911 ,120.94790157952
13.7737966994859 ,120.94790157952 13.7737966994859 ,120.947780579619
13.7738826991356 ,120.947781579913 13.7738846996532 ,120.947685579841
13.7739317000246 ,120.947491580821 13.7740436995634 ,120.947881580104
13.774638699372 ,120.947875580117 13.7752566999521 ,120.947691580716
13.7752806995878 ,120.947028581302 13.7758416993301 ,120.947027580884
13.7759436995362 ,120.946894581169 13.7760456991315 ,120.946156582435
13.7761686995377 ,120.9459695827 13.7765017001618 ,120.945968582131
13.7765786996464 ,120.946045582334 13.7768107000939 ,120.946124582967
13.7768376991713 ,120.946307582683 13.7769417002886 ,120.946251582585
13.7772756997884 ,120.945954583113 13.7782256996654 ,120.945768582824
13.7783786995522 ,120.945081583341 13.7786307001322 ,120.944682584804
13.7790646996173 ,120.944339584713 13.7792416999451 ,120.944046584884
13.7795226992718 ,120.94365058579 13.7796737001305 ,120.943646585685
13.7801367003654 ,120.942773586314 13.7806187000421 ,120.942145588119
13.7807477000423 ,120.94214758713 13.7807577000478 ,120.941949587394
13.7808966994756 ,120.941898587357 13.7808967003478 ,120.941898587357
13.7808967003478 ,120.941898588216 13.7809166993276 ,120.941882588124
13.7809167000871 ,120.941665588478 13.7811476998331 ,120.941711588532
13.7812886996558 ,120.941543588505 13.7814507000611 ,120.941326589029
13.7815886995907 ,120.941324588756 13.7818456998425 ,120.941222588855
13.7820557003543 ,120.941221588706 13.7820927004117 ,120.941221588706
13.7820927004117 ,120.941221589376 13.7821427003189 ,120.941375589047
13.782143700156 ,120.941373588147 13.7823437004827 ,120.941372588722
13.7824537001356 ,120.941366588069 13.782546699571 ,120.941532588916
13.7826416996957 ,120.942150587739 13.7830436993852 ,120.942317587143
13.7830216993328 ,120.942844587087 13.7829096996479 ,120.943083586784
13.78288770041 ,120.943081586605 13.7830987002521 ,120.943151586748
13.7833087003247 ,120.943076586407 13.7837057002219 ,120.942361587216
13.783442700435 ,120.942284587 13.7840957004391 ,120.94285558738
13.7843576993204 ,120.943071585955 13.7843127001598 ,120.943521586262
13.7847366993097 ,120.943521586245 13.7847146994284 ,120.943810585371
13.7847166997569 ,120.943810585371 13.7847166997569 ,120.943903585345
13.7847176992901 ,120.944144584882 13.7846126993342 ,120.944337584852
13.7844027002044 ,120.944554584503 13.7842646993978 ,120.944866584234
13.7841736999533 ,120.945274584138 13.7841296999024 ,120.945684583269
13.7838056997859 ,120.945901582998 13.7837366993142 ,120.946166582781
13.7835746992964 ,120.947193581247 13.7838646992182 ,120.947139581061
13.7846376991343 ,120.947401581608 13.784756700147 ,120.947952579977
13.7848316993526 ,120.948478580273 13.7849526990372 ,120.949122578952
13.7852156999324 ,120.949362578372 13.7852176990199 ,120.949746577914
13.785174699536 ,120.950011577631 13.7849886993708 ,120.950347577
13.7849216992745 ,120.950632576422 13.7852286994772 ,120.950726576421
13.785463699583 ,120.951227576775 13.7856786999332 ,120.951200575881
13.7859826996745 ,120.951866575863 13.7865966990361 ,120.95199957530
Re: [PATCHES] [HACKERS] Win32 Version numbering patch
> > I was hoping this would be in for beta 3, but alas - can someone > > *please* commit the win32 version patch at: > > http://candle.pha.pa.us/mhonarc/patches/msg0.html > > Personally I was holding off in hopes of seeing a cleaner solution. > A patch that requires every executable-building Makefile to > have a platform-specific wart isn't going to be very > maintainable. You already missed pg_config, plus whichever > of the contrib modules build executables... IIRC, when this patch was posted, pg_config was a shellscript. Definitly when the original version of the patch was posted. But sure, I should probably have updated it when that changed. Do you have any ideas on *how* to do this? Or specifically which parts you don't like (or is it the whole concept)? I had no idea you were holding off based on that. I've asked repeatedly for comments about what is bad, and this is the first time I heard something about the new patch (other than the location of the files and the inclusion of the icon which Peter E commented on sevearal times), if I'm not mistaken. Part of the versioninfo struct is file-specific, part is product specific. I see no way around it other than having some parts of it in every Makefile. Sure, we could have some kind of central mapping file somewhere, but I would consider that *less* maintainable. I specifically did not include contrib modules. That can be done once we have a final verdict on exactly how it should be. //Magnus ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
