Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-06 Thread KONDO Mitsumasa
(2013/08/06 20:19), Florian Weimer wrote: The first file name resolution is slow, but subsequent resolutions typically happen from the dentry cache. (The cache is not populated when the directory is opened.) I see. I understand why ext file system is slow when we put large number of files. Tha

Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-06 Thread Florian Weimer
On 08/06/2013 12:28 PM, KONDO Mitsumasa wrote: (2013/08/05 20:38), Florian Weimer wrote: On 08/05/2013 10:42 AM, John R Pierce wrote: On 8/5/2013 1:01 AM, KONDO Mitsumasa wrote: When we open file, ext3 or ext4 file system seems to sequential search inode for opening file in file directory. n

Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-06 Thread KONDO Mitsumasa
(2013/08/05 20:38), Florian Weimer wrote: On 08/05/2013 10:42 AM, John R Pierce wrote: On 8/5/2013 1:01 AM, KONDO Mitsumasa wrote: When we open file, ext3 or ext4 file system seems to sequential search inode for opening file in file directory. no, ext3/4 uses H-tree structures to search direc

Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-06 Thread KONDO Mitsumasa
(2013/08/05 19:28), Andres Freund wrote: On 2013-08-05 18:40:10 +0900, KONDO Mitsumasa wrote: (2013/08/05 17:14), Amit Langote wrote: So, within the limits of max_files_per_process, the routines of file.c should not become a bottleneck? It may not become bottleneck. 1 FD consumes 160 byte in 6

Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-05 Thread Tom Lane
Andres Freund writes: > ... Also, there are global > limits to the amount of filehandles that can simultaneously opened on a > system. Yeah. Raising max_files_per_process puts you at serious risk that everything else on the box will start falling over for lack of available FD slots. (PG itself

Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-05 Thread Florian Weimer
On 08/05/2013 10:42 AM, John R Pierce wrote: On 8/5/2013 1:01 AM, KONDO Mitsumasa wrote: When we open file, ext3 or ext4 file system seems to sequential search inode for opening file in file directory. no, ext3/4 uses H-tree structures to search directories over 1 block long quite efficiently.

Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-05 Thread Andres Freund
On 2013-08-05 18:40:10 +0900, KONDO Mitsumasa wrote: > (2013/08/05 17:14), Amit Langote wrote: > >So, within the limits of max_files_per_process, the routines of file.c > >should not become a bottleneck? > It may not become bottleneck. > 1 FD consumes 160 byte in 64bit system. See linux manual at "

Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-05 Thread KONDO Mitsumasa
(2013/08/05 17:14), Amit Langote wrote: So, within the limits of max_files_per_process, the routines of file.c should not become a bottleneck? It may not become bottleneck. 1 FD consumes 160 byte in 64bit system. See linux manual at "epoll". Regards, -- Mitsumasa KONDO NTT Open Source Software

Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-05 Thread John R Pierce
On 8/5/2013 1:01 AM, KONDO Mitsumasa wrote: When we open file, ext3 or ext4 file system seems to sequential search inode for opening file in file directory. no, ext3/4 uses H-tree structures to search directories over 1 block long quite efficiently. -- john r pierce

Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-05 Thread Amit Langote
On Mon, Aug 5, 2013 at 5:01 PM, KONDO Mitsumasa wrote: > Hi Amit, > > > (2013/08/05 15:23), Amit Langote wrote: >> >> May the routines in fd.c become bottleneck with a large number of >> concurrent connections to above database, say something like "pgbench >> -j 8 -c 128"? Is there any other place

Re: [GENERAL] Bottlenecks with large number of relation segment files

2013-08-05 Thread KONDO Mitsumasa
Hi Amit, (2013/08/05 15:23), Amit Langote wrote: May the routines in fd.c become bottleneck with a large number of concurrent connections to above database, say something like "pgbench -j 8 -c 128"? Is there any other place I should be paying attention to? What kind of file system did you use?